Combined with 'da98651 - Revert "DR2064:
decltype(E) is only a dependent', this change (5a391d3) caused verifier
errors when building Chromium. See https://crbug.com/1168494#c1 for a
reproducer.
Additionally it reverts changes that were dependent on this one, see
below.
> Following up on PR48517, fix handling of template arguments that refer
> to dependent declarations.
>
> Treat an id-expression that names a local variable in a templated
> function as being instantiation-dependent.
>
> This addresses a language defect whereby a reference to a dependent
> declaration can be formed without any construct being value-dependent.
> Fixing that through value-dependence turns out to be problematic, so
> instead this patch takes the approach (proposed on the core reflector)
> of allowing the use of pointers or references to (but not values of)
> dependent declarations inside value-dependent expressions, and instead
> treating template arguments as dependent if they evaluate to a constant
> involving such dependent declarations.
>
> This ends up affecting a bunch of OpenMP tests, due to OpenMP
> imprecisely handling instantiation-dependent constructs, bailing out
> early instead of processing dependent constructs to the extent possible
> when handling the template.
>
> Previously committed as 8c1f2d15b8, and
> reverted because a dependency commit was reverted.
This reverts commit 5a391d38ac.
It also restores clang/test/SemaCXX/coroutines.cpp to its state before
da986511fb.
Revert "[c++20] P1907R1: Support for generalized non-type template arguments of scalar type."
> Previously committed as 9e08e51a20, and
> reverted because a dependency commit was reverted. This incorporates the
> following follow-on commits that were also reverted:
>
> 7e84aa1b81 by Simon Pilgrim
> ed13d8c667 by me
> 95c7b6cadb by Sam McCall
> 430d5d8429 by Dave Zarzycki
This reverts commit 4b574008ae.
Revert "[msabi] Mangle a template argument referring to array-to-pointer decay"
> [msabi] Mangle a template argument referring to array-to-pointer decay
> applied to an array the same as the array itself.
>
> This follows MS ABI, and corrects a regression from the implementation
> of generalized non-type template parameters, where we "forgot" how to
> mangle this case.
This reverts commit 18e093faf7.
Previously we did not record local class declarations. Now with features like
findImplementation and typeHierarchy, we have a need to index such local
classes to accurately report subclasses and implementations of methods.
Performance testing results:
- No changes in indexing timing.
- No significant change in memory usage.
- **1%** increase in #relations.
- **0.17%** increase in #refs.
- **0.22%** increase #symbols.
**New index stats**
Time to index: **4:13 min**
memory usage **543MB**
number of symbols: **521.5K**
number of refs: **8679K**
number of relations: **49K**
**Base Index stats**
Time to index: **4:15 min**
memory usage **542MB**
number of symbols: **520K**
number of refs: **8664K**
number of relations: **48.5K**
Fixes: https://github.com/clangd/clangd/issues/644
Differential Revision: https://reviews.llvm.org/D94785
Previously committed as 9e08e51a20, and
reverted because a dependency commit was reverted. This incorporates the
following follow-on commits that were also reverted:
7e84aa1b81 by Simon Pilgrim
ed13d8c667 by me
95c7b6cadb by Sam McCall
430d5d8429 by Dave Zarzycki
This patch is a try to fix `WorkspaceSymbols.Macros` test after D93796.
If a macro definition is in the preamble section, then it appears to be in the preamble (static) index and not in the main-file (dynamic) index.
Thus, a such macro could not be found at a symbol search according to the logic that we skip symbols from the static index if the location of these symbols is inside the dynamic index files.
To fix this behavior this patch adds main file macros into the main-file (dynamic) index.
Reviewed By: sammccall
Differential Revision: https://reviews.llvm.org/D94477
This used to implicitly never happen due to only discovering each CDB
once.
We may want to carefully support reindexing one day, but we need to do
it carefully (tricky tradeoffs) and it would need further support in
background indexer.
Making this explicit here rather than just turning off rebroadcast in
background index for a few reasons:
- allows *new* files in the same CDB to be indexed
- relying on bugs-at-a-distance cancelling each other out is bound to bite us
- gets us closer to actually supporting reindexing, which requires similar tracking
Differential Revision: https://reviews.llvm.org/D94503
This is follow up to D93393.
Without this patch `MergedIndex::fuzzyFind()` returns stale symbols from the static index even if these symbols were removed.
Reviewed By: sammccall
Differential Revision: https://reviews.llvm.org/D93796
This is follow up to D93393.
Without this patch clangd takes the symbol definition from the static index if this definition was removed from the dynamic index.
Reviewed By: sammccall
Differential Revision: https://reviews.llvm.org/D93683
This patch fixes the following problem:
- open a file with references to the symbol `Foo`
- remove all references to `Foo` (from the dynamic index).
- `MergedIndex::refs()` result will contain positions of removed references (from the static index).
The idea of this patch is to keep a set of files which were used during index build inside the index.
Thus at processing the static index references we can check if the file of processing reference is a part of the dynamic index or not.
Reviewed By: sammccall
Differential Revision: https://reviews.llvm.org/D93393
SmallVector<T> with default size is now the recommended version (D92522).
Reviewed By: sammccall
Differential Revision: https://reviews.llvm.org/D92788
Figuring out whether the server is responding and debugging issues with remote
index setup is no easy task: add verbose logging for client side RPC requests
to relieve some pain.
Reviewed By: sammccall
Differential Revision: https://reviews.llvm.org/D92181
Makes it easier to diagnose remote index issues with --debug-origins flag.
Reviewed By: sammccall
Differential Revision: https://reviews.llvm.org/D92202
Put project-aware-index between command-line specified static index and
ClangdServer indexes.
This also moves remote-index dependency from clangDaemon to ClangdMain
in an attempt to prevent cyclic dependency between clangDaemon and
remote-index-marshalling.
Differential Revision: https://reviews.llvm.org/D91860
An index implementation that can dispatch to a variety of indexes
depending on the file path. Enables clangd to work with multiple indexes in the
same instance, configured via config files.
Depends on D90749, D90746
Differential Revision: https://reviews.llvm.org/D90750
- Add verbose logging of payloads
- Add public logging of request summaries
- fix non-logging of messages in request scopes (oops!)
- add test for public/non-public logging, extending pipeline_helper a bit.
We've accumulated quite a lot of duplication in the request handlers by now.
I should factor that out, but not in this patch...
Differential Revision: https://reviews.llvm.org/D90654
Previously a corrupted index shard could cause us to resize arrays to an
arbitrary int32. This tends to be a huge number, and can render the
system unresponsive.
Instead, cap this at the amount of data that might reasonably be read
(e.g. the #bytes in the file). If the specified length is more than that,
assume the data is corrupt.
Differential Revision: https://reviews.llvm.org/D91258
https://reviews.llvm.org/D89670 changed the Ref structure, we need to
bump the version to invalidate all stored stale data, otherwise we will
get ` Error while reading shard: malformed or truncated refs` when
building the background index.
Differential Revision: https://reviews.llvm.org/D91131
* Even though remote index is still somewhat experimental, it can now be
used withing clangd itself: this should be the primary way of trying
it out
* Remove `protobuf-compiler` from list of needed Debian packages as it
`protobuf-compiler-grpc` already depends on it
* Bump recommended gRPC version to 1.32.3
There is not reason to check `std::make_unique<...>(..)` return value,
but `clangd::clang::loadIndex()` returns `nullptr` if an index file could not be loaded (e.g. incorrect version).
Reviewed By: kadircet
Differential Revision: https://reviews.llvm.org/D91049
RemoteIndexClient implementations only depends on clangdSupport for
logging functionality and has no dependence on clangDeamon itself. This clears
out that link time dependency and enables depending on it in clangDeamon itself,
so that we can have other index implementations that makes use of the
RemoteIndex.
Differential Revision: https://reviews.llvm.org/D90746
The index server has access to potentially-sensitive information, e.g. a
sequence of fuzzyFind requests reveals details about code completions in the
editor which in turn reveals details about the code being edited.
This information is necessary to provide the service, and our intention[1] is it
should never be retained beyond the scope of the request (e.g. not logged).
At the same time, some log messages should be exposed:
- server startup/index reloads etc that don't pertain to a user request
- basic request logs (method, latency, #results, error code) for monitoring
- errors while handling requests, without request-specific data
The -log=public design accommodates these by allowing three types of logs:
- those not associated with any user RPC request (via context-propagation)
- those explicitly tagged as [public] in the log line
- logging of format strings only, with no interpolated data (error level only)
[1] Specifically: Google is likely to run public instances of this server
for LLVM and potentially other projects, they will run in this configuration.
The details agreed in a Google-internal privacy review.
As clangd developers, we'd encourage others to use this configuration for public
instances too.
Differential Revision: https://reviews.llvm.org/D90526
We were default initializing SymbolIDs before, which would leave
indeterminate values in underlying std::array.
This patch updates the underlying data initalization to be value-init and adds a
way to check for validness of a SymbolID.
Differential Revision: https://reviews.llvm.org/D90397
Previous attempt (15f6bad6d7) introduced
add_dependencies but unfortunately it does not actually add a dependency
between RemoteIndexProto and RemoteIndexServiceProto. This is likely due
to some requirements of it that clang_add_library violates.
As a workaround, we will link RemoteIndexProto library to
RemoteIndexServiceProto which is logical because the library can not be
without linking to RemoteIndexProto anyway.
We only need to version these messages if they actually diverge.
Unlike the service, the namespace name isn't part of the wire format.
clangd-index-server was broken by 81e5f298c4
as the namespace names weren't updated there, this fixes it (by adding
them for the service, and not requiring them elsewhere).
This allows it to have a separate namespace (grpc versioned service) without
putting versioning info on all of the other protos (before we need it).
clang-index-server is still broken (from 81e5f298c4).
Differential Revision: https://reviews.llvm.org/D90031
`llvm::sys::path` is used a lot in the remote index marshalling code. We can save space by avoiding spelling it out explicitly for most functions and times.
Reviewed By: kadircet
Differential Revision: https://reviews.llvm.org/D90016
This allows us to check whether enum field is actually sent over the wire or missing.
Reviewed By: sammccall
Differential Revision: https://reviews.llvm.org/D89882
And also introduce Protobuf package versioning, it will help to deal
with breaking changes. Inroducing package version itself is a breaking
change, clients and servers need to be updated.
Reviewed By: sammccall
Differential Revision: https://reviews.llvm.org/D89862
Without this patch 6 marshalling tests fail on Windows.
This patch contains the following changes:
- Allow paths with Windows slashes (convert to the POSIX style instead of assertion)
- Add support for URI with Windows path.
- Change the value of the second parameter of several `llvm::sys::path::convert_to_slash()` calls: we should use `windows` instead of `posix` to ensure UNIX slashes in the path.
- Port `RemoteMarshallingTest::IncludeHeaderURI` test to Windows.
Reviewed By: kbobyrev
Differential Revision: https://reviews.llvm.org/D89529
This was causing duplicate `symbols` components on the path as both the
edge from an index to filesymbols and filesymbols to symbolslabs were named
symbols.
Differential Revision: https://reviews.llvm.org/D89685
Up until now, we relied on matching the filename.
This depends on unstable details of libstdc++ and doesn't work well on other
stdlibs. Also we'd like to remove it (see D88204).
Differential Revision: https://reviews.llvm.org/D88885
Relative paths received from the server are always in posix style. So
we need to ensure they are relative using that style, and not the native one.
Differential Revision: https://reviews.llvm.org/D88507
Since we have 2 scoring functions (heuristics and decision forest),
renaming the existing evaluate() function to be more descriptive of the
Heuristics being evaluated in it.
Differential Revision: https://reviews.llvm.org/D88431
Translating between JSON objects and C++ strutctures is common.
From experience in clangd, fromJSON/ObjectMapper work well and save a lot of
code, but aren't adopted elsewhere at least partly due to total lack of error
reporting beyond "ok"/"bad".
The recently-added error model should be rich enough for most applications.
It requires tracking the path within the root object and reporting local
errors at appropriate places.
To do this, we exploit the fact that the call graph of recursive
parse functions mirror the structure of the JSON itself.
The current path is represented as a linked list of segments, each of which is
on the stack as a parameter. Concretely, fromJSON now looks like:
bool fromJSON(const Value&, T&, Path);
Beyond the signature change, this is reasonably unobtrusive: building
the path segments is mostly handled by ObjectMapper and the vector<T> fromJSON.
However the root caller of fromJSON must now create a Root object to
store the errors, which is a little clunky.
I've added high-level parse<T>(StringRef) -> Expected<T>, but it's not
general enough to be the primary interface I think (at least, not usable in
clangd).
All existing users (mostly just clangd) are updated in this patch,
making this change backwards-compatible is a bit hairy.
Differential Revision: https://reviews.llvm.org/D88103
This patch adds a mechanism to load new versions of index into
clangd-index-server using SwapIndex and FileStatus information about last
modification time without downtime.
Reviewed By: kadircet
Differential Revision: https://reviews.llvm.org/D87450
Without this patch `clangd` crashes at try to load compressed string table when `zlib` is not available.
Example:
- Build `clangd` with MinGW (`zlib` found)
- Build index
- Build `clangd` with Visual Studio compiler (`zlib` not found)
- Try to load index
Reviewed By: sammccall, adamcz
Differential Revision: https://reviews.llvm.org/D87673
Summary:
This is considerably terser than the makeStringError and friends, and
avoids verbosity cliffs that discourage adding log information.
It follows the syntax used in log/elog/vlog/dlog that have been successful.
The main caveats are:
- it's strictly out-of-place in logger.h, though kind of fits thematically and
in implementation
- it claims the "error" identifier, which seems a bit too opinionated
to put higher up in llvm
I've updated some users of StringError mostly at random - there are lots
more mechanical changes but I'd like to get this reviewed before making
them all.
Reviewers: kbobyrev, hokein
Subscribers: mgorny, ilya-biryukov, javed.absar, MaskRay, jkorous, arphaman, kadircet, usaxena95, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D83419
Summary: This will be needed to support call hierarchy
Reviewers: kadircet
Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, usaxena95, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D83536
I've dropped the background context parameter, since we in practice just pass the
current context there, and we now have a different way to specify context too.
While here, clean up a couple of comments.
Reviewed By: kadircet
Differential Revision: https://reviews.llvm.org/D83157
When running dexp in remote mode without --project-root it shuts down
with an assertion. This is not the desired behaviour: instruct user on
how to run it properly when the configuration is incorrect.
This is a refactoring: errors should be logged only on the highest level.
Switch from Optional to Expected in the serialization code.
Reviewed By: kadircet
Differential Revision: https://reviews.llvm.org/D84939
This is the last missing bit in the core remote index implementation. The only
remaining bits are some API refactorings (replacing Optional with Expected and
being better at reporting errors).
Reviewed By: kadircet
Differential Revision: https://reviews.llvm.org/D84894
Function void run() on line 286 overrides a virtual function on line 92 of
clang-tools-extra/clangd/index/dex/dexp/Dexp.cpp. Not marking it override will
cause a build failure when we use -Werror (every warning is treated as an error).
Reviewed By: kbobyrev (Kirill Bobyrev)
Differential Revision: https://reviews.llvm.org/D84794
Summary:
Without this patch clangd does not collect references for main-file symbols if there is no public declaration in preamble.
Example:
`test1.c`
```
void f1() {}
```
`test2.c`
```
extern void f1();
void f2() {
f^1();
}
```
`Find all references` does not show definition of f1() in the result, but GTD works OK.
Reviewers: sammccall, kadircet
Reviewed By: kadircet
Subscribers: ilya-golovenko, ilya-biryukov, MaskRay, jkorous, arphaman, kadircet, usaxena95, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D84513
Summary:
Only FuzzyFindRequest is implemented via Marshaller even though other requests
also follow a similar pattern. Unify them under the marshalling umbrella and
make the server requests even more uniform to complement D84499.
Reviewers: kadircet
Reviewed By: kadircet
Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, usaxena95, cfe-commits, sammccall
Tags: #clang
Differential Revision: https://reviews.llvm.org/D84525
Summary:
This was resulting in macros coming from preambles vanishing when user
have opened the source header. For example:
```
// test.h:
#define X
```
and
```
// test.cc
#include "test.h
^
```
If user only opens test.cc, we'll get `X` as a completion candidate,
since it is indexed as part of the preamble. But if the user opens
test.h afterwards we would index it as part of the main file and lose
the symbol (as new index shard for test.h will override the existing one
in dynamic index).
Also we were not setting origins for macros correctly, this patch also
fixes it.
Fixes https://github.com/clangd/clangd/issues/461
Reviewers: hokein
Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, usaxena95, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D84297
Summary:
Remote server should not send messages that are invalid and will cause problems
on the client side. The client should not be affected by server's failures
whenever possible.
Also add more error messages and logs.
Reviewers: sammccall
Reviewed By: sammccall
Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, kadircet, usaxena95, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D83826
Summary:
We only support Build/Skip for now, but with 'Load' or similar as an
option for future (load existing shards but don't build new ones).
This requires creating the config for each TU on startup. In LLVM, this
is 4000 occurrences for a total of 800ms on my machine.
But together with caching from D83755 it is only 25ms.
Reviewers: kadircet
Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, usaxena95, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D83768
Summary:
.clangd/index was well-intentioned in 2754942cba, but `.clangd` is the best
filename for the clangd config file (matching .clang-format and .clang-tidy).
And of course we can't have both .clangd/index and .clangd...
There are a few overlapping goals to satisfy:
- it should be clear from the directory name that this is transient
data that is safe to delete at the cost of recomputation, i.e. a cache
- it should be easy and self-documenting to blacklist these files in .gitignore
- we should have some consistency between filenames in-tree and
corresponding files in user storage (e.g. under XDG's ~/.cache/)
- we should be consistent across platforms (including windows, which
doesn't have distinct cache vs config directories)
So the plan is:
$PROJECT/.clangd (project config)
$PROJECT/.cache/clangd/index/ (project index)
$PROJECT/.cache/clangd/modules/ (maybe in future)
$XDG_CONFIG_HOME/clangd/config.yaml (user config)
$XDG_CACHE_HOME/clangd/index/ (index of non-project files)
$XDG_CACHE_HOME/clangd/modules/ (maybe in future)
This is sensible if XDG_{CONFIG,CACHE}_HOME coincide, and has a simple
.gitignore rule going forward: `.cache/`.
The monorepo gitignore is updated to reflect the backwards-compatible practice:
ignore .clangd/ (with trailing slash) matching index files from clangd 9/10
ignore .cache matching index from clangd 11+, and potentially other tools.
The entries from llvm-project/llvm gitignore are removed (obsolete).
Reviewers: kadircet, hokein
Subscribers: ilya-biryukov, MaskRay, jkorous, omtcyfz, arphaman, usaxena95, cfe-commits, llvm-commits
Tags: #clang, #llvm
Differential Revision: https://reviews.llvm.org/D83099
Summary:
ClangdServer owns the question of exactly which config to create, but
TUScheduler/BackgroundIndex control threads and so decide at which point
to inject it.
Reviewers: kadircet
Subscribers: ilya-biryukov, javed.absar, MaskRay, jkorous, arphaman, usaxena95, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D83095
Summary:
FileIndex was built out of threadsafe components, so update() didn't have data
races, but wasn't actually correct.
Reviewers: kadircet
Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, usaxena95, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D82891
Summary:
This confusion was inadvertently introduced in a change to the
heavyweight_hardware_concurrency API: 8404aeb56a
- don't indirect through the rebuilder policy when building the thread pool
- document that rebuilder thresholds are exposed for testing only
- don't use 0 as a sentinel value for "all threads", as we use it as a
sentinel value for "synchronous" (though unsupported for BackgroundIndex)
- rather than pick some new sentinel value, just always use 4 threads for tests
Reviewers: kadircet, aganea
Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, usaxena95, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D82352
Summary:
Clangd uses FSProvider to get threadsafe views into file systems. This
patch changes naming to make that more explicit.
Depends on D81920
Reviewers: sammccall
Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, usaxena95, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D81998
Summary:
We've faced a couple of problems when the returned FS didn't have the
proper working directory. New signature makes the API safer against such
problems.
Reviewers: sammccall
Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, usaxena95, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D81920
Summary:
We already skip function bodies from these files while parsing, and drop symbols
found in them. However, traversing their ASTs still takes a substantial amount
of time.
Non-scientific benchmark on my machine:
background-indexing llvm-project (llvm+clang+clang-tools-extra), wall time
before: 7:46
after: 5:13
change: -33%
Indexer.cpp libclang should be updated too, I'm less familiar with that code,
and it's doing tricky things with the ShouldSkipFunctionBody callback, so it
needs to be done separately.
Reviewers: kadircet
Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, usaxena95, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D80296
Summary:
YAML serialization was used in the Proof of Concept for simplicity.
This patch replaces implements Protobuf (de) serialization of almost all
types that need to be transferred over the protocol.
Reviewers: sammccall
Reviewed By: sammccall
Subscribers: mgorny, ilya-biryukov, MaskRay, jkorous, arphaman, kadircet, usaxena95, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D79862
Summary: This is worth another 10% or so on InedxBenchmark.DexBuild.
Reviewers: kbobyrev
Subscribers: ilya-biryukov, MaskRay, jkorous, mgrang, arphaman, kadircet, usaxena95, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D79950
Summary:
The Token objects are relatively expensive and we were spending a lot of
CPU creating them for each trigram emitted. Instead, use a tiny trigram
structure until we're ready to finalize the index.
This improves the new BuildDex benchmark by 20%. This code is hot and on
the critical path in clangd: it runs after a new preamble is built.
Reviewers: kbobyrev
Subscribers: ilya-biryukov, MaskRay, jkorous, mgrang, arphaman, kadircet, usaxena95, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D79918
Summary:
Ubuntu 18.04 and older versions do not provide latest gRCP packages and the
ones that are in the repository use deprecated headers. Use these headers to
make builds possible.
https://packages.ubuntu.com/bionic/amd64/libgrpc++-dev/filelist
Reviewers: sammccall
Reviewed By: sammccall
Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, kadircet, usaxena95, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D79487
Summary:
This makes FileShardedIndex more robust and gets rid of the need for a
URIToFileCache, as it is done by the callers now and it is only once per file,
rather than once per symbol.
Reviewers: sammccall
Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, usaxena95, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D79079
Summary:
Dependencies ensure that Protobufs are generated before all libraries
depending on the headers are **built**, not linked.
Reviewers: sammccall
Reviewed By: sammccall
Subscribers: mgorny, ilya-biryukov, MaskRay, jkorous, arphaman, kadircet, usaxena95, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D79085
It's not clear whether clangd should really consider #import to mark a
header as self-contained or not, but this change preserves the old (and
unit-tested) behavior.
This reverts commit ad38f4b371.
As it broke building the unittests:
.../sources/llvm-project/llvm/unittests/Support/Path.cpp:334:5: error: use of undeclared identifier 'set'
set(Value);
^
1 error generated.
Summary:
This patch adds a function that is similar to `llvm::sys::path::home_directory`, but provides access to the system cache directory.
For Windows, that is %LOCALAPPDATA%, and applications should put their files under %LOCALAPPDATA%\Organization\Product\.
For *nixes, it adheres to the XDG Base Directory Specification, so it first looks at the XDG_CACHE_HOME environment variable and falls back to ~/.cache/.
Subsequently, the Clangd Index storage leverages this new API to put index files somewhere else than the users home directory.
Fixes https://github.com/clangd/clangd/issues/341
Reviewers: sammccall, chandlerc, Bigcheese
Reviewed By: sammccall
Subscribers: hiraditya, ilya-biryukov, MaskRay, jkorous, dexonsmith, arphaman, kadircet, ormris, usaxena95, cfe-commits, llvm-commits
Tags: #clang-tools-extra, #clang, #llvm
Differential Revision: https://reviews.llvm.org/D78501
Summary:
Generated Protobuf library has to be in CLANG_EXPORTS and should also be
installed appropriately. The easiest way to do that is via CMake's
add_clang_library. That unfortunately applies "one directory - one
clang_(library|tool)" policy so .proto files should be in a separate directory
and complicates the layout.
This setup works both in shared and static libs mode.
Resolves: https://github.com/clangd/clangd/issues/351
Reviewers: sammccall
Reviewed By: sammccall
Subscribers: mgorny, ilya-biryukov, MaskRay, jkorous, arphaman, kadircet, usaxena95, cfe-commits, llvm-commits
Tags: #clang, #llvm
Differential Revision: https://reviews.llvm.org/D78885
Summary: Add a command to dexp that exports index data in chosen format (e.g. YAML).
Reviewers: sammccall
Subscribers: kbobyrev, mgorny, ilya-biryukov, MaskRay, jkorous, arphaman, kadircet, usaxena95, cfe-commits
Tags: #clang-tools-extra, #clang
Differential Revision: https://reviews.llvm.org/D77385
Summary:
This patch allows using installed gRPC to build two simple tools which
currently provide the functionality of looking up the symbol by name.
remote-index-client is a simplified version of dexp which connects to
remote-index-server passes lookup requests.
I also significantly reduced the scope of this patch to prevent large changelist
and more bugs. The next steps would be:
* Extending Protocol for deep copies of Symbol and inherit RemoteIndex from
Index to unify the interfaces
* Make remote-index-server more generic and merge the remote index client with
dexp
* Modify Clangd to allow using remote index instead of the local one for all
global index requests
Reviewers: sammccall
Reviewed By: sammccall
Subscribers: mgorny, ilya-biryukov, MaskRay, jkorous, arphaman, kadircet, usaxena95, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D77794
Summary:
This reduces memory usage by dynamic index from more than 400MB to 32MB
when all files in clang-tools-extra/clangd/*.cpp are active in clangd.
Reviewers: sammccall
Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, usaxena95, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D77732
Summary:
This ties to an LSP feature (diagnostic versioning) but really a lot
of the value is in being able to log what's happening with file versions
and queues more descriptively and clearly.
As such it's fairly invasive, for a logging patch :-\
Key decisions:
- at the LSP layer, we don't reqire the client to provide versions (LSP
makes it mandatory but we never enforced it). If not provided,
versions start at 0 and increment. DraftStore handles this.
- don't propagate magically using contexts, but rather manually:
addDocument -> ParseInputs -> (ParsedAST, Preamble, various callbacks)
Context-propagation would hide the versions from ClangdServer, which
would make producing good log messages hard
- within ClangdServer, treat versions as opaque and unordered.
std::string is a convenient type for this, and allows richer versions
for embedders. They're "mandatory" but "null" is a reasonable default.
Subscribers: ilya-biryukov, javed.absar, MaskRay, jkorous, arphaman, kadircet, usaxena95, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D75582
Summary:
Though we don't have new changes to the index format, we have changes to
symbol collector, e.g. collect marcos, spelled references. Bump the
version to force background-index to rebuild.
Reviewers: kadircet
Reviewed By: kadircet
Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, usaxena95, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D74127
The goal of this patch is to maximize CPU utilization on multi-socket or high core count systems, so that parallel computations such as LLD/ThinLTO can use all hardware threads in the system. Before this patch, on Windows, a maximum of 64 hardware threads could be used at most, in some cases dispatched only on one CPU socket.
== Background ==
Windows doesn't have a flat cpu_set_t like Linux. Instead, it projects hardware CPUs (or NUMA nodes) to applications through a concept of "processor groups". A "processor" is the smallest unit of execution on a CPU, that is, an hyper-thread if SMT is active; a core otherwise. There's a limit of 32-bit processors on older 32-bit versions of Windows, which later was raised to 64-processors with 64-bit versions of Windows. This limit comes from the affinity mask, which historically is represented by the sizeof(void*). Consequently, the concept of "processor groups" was introduced for dealing with systems with more than 64 hyper-threads.
By default, the Windows OS assigns only one "processor group" to each starting application, in a round-robin manner. If the application wants to use more processors, it needs to programmatically enable it, by assigning threads to other "processor groups". This also means that affinity cannot cross "processor group" boundaries; one can only specify a "preferred" group on start-up, but the application is free to allocate more groups if it wants to.
This creates a peculiar situation, where newer CPUs like the AMD EPYC 7702P (64-cores, 128-hyperthreads) are projected by the OS as two (2) "processor groups". This means that by default, an application can only use half of the cores. This situation could only get worse in the years to come, as dies with more cores will appear on the market.
== The problem ==
The heavyweight_hardware_concurrency() API was introduced so that only *one hardware thread per core* was used. Once that API returns, that original intention is lost, only the number of threads is retained. Consider a situation, on Windows, where the system has 2 CPU sockets, 18 cores each, each core having 2 hyper-threads, for a total of 72 hyper-threads. Both heavyweight_hardware_concurrency() and hardware_concurrency() currently return 36, because on Windows they are simply wrappers over std:🧵:hardware_concurrency() -- which can only return processors from the current "processor group".
== The changes in this patch ==
To solve this situation, we capture (and retain) the initial intention until the point of usage, through a new ThreadPoolStrategy class. The number of threads to use is deferred as late as possible, until the moment where the std::threads are created (ThreadPool in the case of ThinLTO).
When using hardware_concurrency(), setting ThreadCount to 0 now means to use all the possible hardware CPU (SMT) threads. Providing a ThreadCount above to the maximum number of threads will have no effect, the maximum will be used instead.
The heavyweight_hardware_concurrency() is similar to hardware_concurrency(), except that only one thread per hardware *core* will be used.
When LLVM_ENABLE_THREADS is OFF, the threading APIs will always return 1, to ensure any caller loops will be exercised at least once.
Differential Revision: https://reviews.llvm.org/D71775
Summary:
DeclarationName for cxx constructor is special, it is not an identifier.
thus the "Spelled" flag are not set for all ctor references, this patch
fixes it.
Reviewers: kbobyrev
Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, kadircet, usaxena95, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D74125
This patch allows the index does to provide a way to distinguish
implicit references (e.g. coming from macro expansions) from the spelled
ones. The corresponding flag was added to RefKind and symbols that are
referenced without spelling their name explicitly are now marked
implicit. This allows fixing incorrect behavior when renaming a symbol
that was referenced in macro expansions would try to rename macro
invocations.
Differential Revision: D72746
Reviewed by: hokein
This is how it should've been and brings it more in line with
std::string_view. There should be no functional change here.
This is mostly mechanical from a custom clang-tidy check, with a lot of
manual fixups. It uncovers a lot of minor inefficiencies.
This doesn't actually modify StringRef yet, I'll do that in a follow-up.
Summary:
It simply shows the completed/total items on the background queue, e.g.
indexing: 233/1000
The denominator is reset to zero every time the queue goes idle.
The protocol is fairly complicated here (requires creating a remote "progress"
resource before sending updates). We implement the full protocol, but I've added
an extension allowing it to be skipped to reduce the burden on clients - in
particular the lit test takes this shortcut.
The addition of background index progress to DiagnosticConsumer seems ridiculous
at first glance, but I believe that interface is trending in the direction of
"ClangdServer callbacks" anyway. It's due for a rename, but otherwise actually
fits.
Reviewers: kadircet, usaxena95
Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, jfb, cfe-commits, llvm-commits
Tags: #clang, #llvm
Differential Revision: https://reviews.llvm.org/D73218
Summary:
Adds macro references to the dynamic index.
Tests added.
Also exposed a new API to convert path to URI in URI.h
Reviewers: hokein
Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, kadircet, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D71406
Haojian Wu