Tracked down the crash, which was argument-evaluation-order UB
in the wrapping indexStandardLibrary().
Sorry for the churn!
This reverts commit 77533ea443.
This reverts commit ccdb56ac10.
Still seeing windows failures on GN bots: http://45.33.8.238/win/58316/step_9.txt
Unfortunately I can't debug these at all - it's a bare unsymbolized
stacktrace, and I can't reproduce the failure.
This provides a nice "warm start" with all headers indexed, not just
those included so far.
The standard library is indexed after a preamble is parsed, using that
file's configuration. The result is pushed into the dynamic index.
If we later see a higher language version, we reindex it.
It's configurable as Index.StandardLibrary, off by default for now.
Based on D105177 by @kuhnel
Fixes https://github.com/clangd/clangd/issues/618
Differential Revision: https://reviews.llvm.org/D115232
It's deprecated; migrate to FileEntryRef::getName where it doesn't matter.
Also change one subtle case of implicit FileEntry::getName to be explicit.
After this patch, all the remaining FileEntry::getName calls are subtle
cases where we may be relying on exactly which filename variant is returned
(for indexing, IWYU directive handling, etc).
New values:
- Split Dynamic into Open/Preamble
- Add Background (previously was just Unknown)
- Soon: stdlib index
This requires extending to 16 bits, which fits within the padding of Symbol.
Unfortunately we're also *serializing* SymbolOrigin as a fixed 8 bits.
Stop serializing SymbolOrigin:
- conceptually, the source is whoever indexes or *deserializes* a symbol
- deserialization takes SymbolOrigin as a parameter and stamps it on each sym
- this is a breaking format change
Differential Revision: https://reviews.llvm.org/D115243
Main use of these is in the standard library, where they generally clutter up
the index.
Certain macros are also common, we don't touch indexing of macros in this patch.
Differential Revision: https://reviews.llvm.org/D115301
Also give CanonicalIncludes a less powerful interface (canonicalizes
symbols vs headers separately) so we can cache its results better.
Prior to this:
- path->uri conversions were not consistently cached, this is
particularly cheap when we start from a FileEntry* (which we often can)
- only a small fraction of header-to-include calculation was cached
This is a significant speedup at least for dynamic indexing of preambles.
On my machine, opening XRefs.cpp:
```
PreambleCallback 1.208 -> 1.019 (-15.7%)
BuildPreamble 5.538 -> 5.214 (-5.8%)
```
Differential Revision: https://reviews.llvm.org/D98371
Without this patch the file list of the preamble index contains URIs, but other indexes file lists contain file paths.
This makes `indexedFiles()` always returns `IndexContents::None` for the preamble index, because current implementation expects file paths inside the file list of the index.
This patch fixes this problem and also helps to avoid a lot of URI to path conversions during indexes merge.
Reviewed By: kadircet
Differential Revision: https://reviews.llvm.org/D97535
Current indexes merge logic skip data from the static index if the file is in the dynamic index, but sometimes the dynamic index does not contain references (e.g. preamble (dynamic) index vs background (static) index).
This problem is masked with the fact, that the preamble index file list consists of file URI's and other indexes file lists consist of file paths.
This patch introduces the index contents (symbols, references, etc.), which makes indexes merge more flexible and makes it able to use URI's for the index file list.
Reviewed By: sammccall
Differential Revision: https://reviews.llvm.org/D94952
Previously we did not record local class declarations. Now with features like
findImplementation and typeHierarchy, we have a need to index such local
classes to accurately report subclasses and implementations of methods.
Performance testing results:
- No changes in indexing timing.
- No significant change in memory usage.
- **1%** increase in #relations.
- **0.17%** increase in #refs.
- **0.22%** increase #symbols.
**New index stats**
Time to index: **4:13 min**
memory usage **543MB**
number of symbols: **521.5K**
number of refs: **8679K**
number of relations: **49K**
**Base Index stats**
Time to index: **4:15 min**
memory usage **542MB**
number of symbols: **520K**
number of refs: **8664K**
number of relations: **48.5K**
Fixes: https://github.com/clangd/clangd/issues/644
Differential Revision: https://reviews.llvm.org/D94785
This patch fixes the following problem:
- open a file with references to the symbol `Foo`
- remove all references to `Foo` (from the dynamic index).
- `MergedIndex::refs()` result will contain positions of removed references (from the static index).
The idea of this patch is to keep a set of files which were used during index build inside the index.
Thus at processing the static index references we can check if the file of processing reference is a part of the dynamic index or not.
Reviewed By: sammccall
Differential Revision: https://reviews.llvm.org/D93393
This was causing duplicate `symbols` components on the path as both the
edge from an index to filesymbols and filesymbols to symbolslabs were named
symbols.
Differential Revision: https://reviews.llvm.org/D89685
Summary: This will be needed to support call hierarchy
Reviewers: kadircet
Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, usaxena95, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D83536
Summary:
FileIndex was built out of threadsafe components, so update() didn't have data
races, but wasn't actually correct.
Reviewers: kadircet
Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, usaxena95, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D82891
Summary:
This makes FileShardedIndex more robust and gets rid of the need for a
URIToFileCache, as it is done by the callers now and it is only once per file,
rather than once per symbol.
Reviewers: sammccall
Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, usaxena95, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D79079
Summary:
This reduces memory usage by dynamic index from more than 400MB to 32MB
when all files in clang-tools-extra/clangd/*.cpp are active in clangd.
Reviewers: sammccall
Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, usaxena95, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D77732
Summary:
This ties to an LSP feature (diagnostic versioning) but really a lot
of the value is in being able to log what's happening with file versions
and queues more descriptively and clearly.
As such it's fairly invasive, for a logging patch :-\
Key decisions:
- at the LSP layer, we don't reqire the client to provide versions (LSP
makes it mandatory but we never enforced it). If not provided,
versions start at 0 and increment. DraftStore handles this.
- don't propagate magically using contexts, but rather manually:
addDocument -> ParseInputs -> (ParsedAST, Preamble, various callbacks)
Context-propagation would hide the versions from ClangdServer, which
would make producing good log messages hard
- within ClangdServer, treat versions as opaque and unordered.
std::string is a convenient type for this, and allows richer versions
for embedders. They're "mandatory" but "null" is a reasonable default.
Subscribers: ilya-biryukov, javed.absar, MaskRay, jkorous, arphaman, kadircet, usaxena95, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D75582
This is how it should've been and brings it more in line with
std::string_view. There should be no functional change here.
This is mostly mechanical from a custom clang-tidy check, with a lot of
manual fixups. It uncovers a lot of minor inefficiencies.
This doesn't actually modify StringRef yet, I'll do that in a follow-up.
Summary:
Adds macro references to the dynamic index.
Tests added.
Also exposed a new API to convert path to URI in URI.h
Reviewers: hokein
Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, kadircet, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D71406
Builtins are rarely if ever accessed via the Preprocessor. They are
typically found on the ASTContext, so there should be no performance
penalty to using a pointer indirection to store the builtin context.
This much better reflects what is (now) in this header.
Maybe a rename to ParsedTU would be an improvement, but that's a much
more invasive change and life is too short.
ClangdUnit is dead, long live ClangdUnitTests!
llvm-svn: 370862
Now that we've moved to C++14, we no longer need the llvm::make_unique
implementation from STLExtras.h. This patch is a mechanical replacement
of (hopefully) all the llvm::make_unique instances across the monorepo.
Differential revision: https://reviews.llvm.org/D66259
llvm-svn: 368944
Summary:
Previously, we randomly pick one main file symbol in dynamic index, we
may loose the ideal symbol (with definition location) in the index.
It fixes the issue where sometimes we fail to go to the symbol definition, see:
1. call go-to-decl on Foo in Foo.cpp
2. jump to Foo.h, call go-to-def on Foo in Foo.h
we can't go back to Foo.cpp -- because we open Foo.cpp, Foo.h in clangd, both
files have Foo symbol (one with def&decl, one with decl only), we randomely
choose one.
Reviewers: kadircet
Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D63425
llvm-svn: 363568
Summary:
This builds on the relations support added in D59407, D62459,
and D62471 to expose relations in SymbolIndex and its
implementations.
Reviewers: kadircet
Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D62839
llvm-svn: 363481
Summary:
After rL360344, BackgroundIndex expects symbols with zero refcounts.
Therefore existing index files are no longer valid.
Assertion regarding finding target of a reference was wrong, since
background-index might still be going on or we might've loaded only some part of
the shards and might be missing the declaring shards for the symbol.
Reviewers: ilya-biryukov
Subscribers: MaskRay, jkorous, arphaman, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D61734
llvm-svn: 360349
Summary:
For counting number of references clangd was relying on merging every
duplication of a symbol. Unfortunately this does not apply to FileIndex(and one
of its users' BackgroundIndex), since we get rid of duplication by simply
dropping symbols coming from non-canonical locations. So only one or two(coming
from canonical declaration header and defined source file, if exists)
replications of the same symbol reaches merging step.
This patch changes reference counting logic to rather count number of different
RefSlabs a given SymbolID exists.
Reviewers: ilya-biryukov
Subscribers: ioeric, MaskRay, jkorous, mgrang, arphaman, jdoerfert, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D59481
llvm-svn: 360344
Summary:
This is a further optimization of r350803, we drop docs in static index for
symbols not being indexed for completion, while keeping the docs in dynamic
index (we rely on dynamic index to get docs for class members).
Reviewers: ilya-biryukov
Reviewed By: ilya-biryukov
Subscribers: ioeric, MaskRay, jkorous, arphaman, kadircet, cfe-commits
Differential Revision: https://reviews.llvm.org/D56539
llvm-svn: 354792
Summary:
This enables include insertion by adding canonical includes into
preambledata.
Reviewers: ioeric, ilya-biryukov
Subscribers: javed.absar, MaskRay, jkorous, arphaman, cfe-commits
Differential Revision: https://reviews.llvm.org/D57508
llvm-svn: 353054
Sam McCall