Some command-line codegen arguments are likely to differ between identical modules discovered from different translation units. This patch removes them to make builds deterministic and/or reduce the number of built modules.
Reviewed By: Bigcheese
Differential Revision: https://reviews.llvm.org/D112923
The filesystem used during dependency scanning does two things: it caches file entries and minimizes source file contents. We use the term "ignored file" in a couple of places, but it's not clear what exactly that means. This commit clears up the semantics, explicitly spelling out this relates to minimization.
During explicit modules build, when all modules are provided via `-fmodule-file=<path>` and implicit modules and implicit module maps are disabled (`-fno-implicit-modules`, `-fno-implicit-module-maps`), we don't need to load the original module map files at all. This patch stops emitting the `-fmodule-map-file=` arguments we don't need, saving some compilation time due to avoiding parsing such module maps and making the command line shorter.
Reviewed By: bnbarham
Differential Revision: https://reviews.llvm.org/D113473
The dependency scanner works with multiple instances of `Compiler{Instance,Invocation}`. From names of the variables/members, their purpose is not obvious.
This patch gives descriptive name to the generated `CompilerInvocation` that can be used to derive the command-line to build a modular dependency.
Depends on D111725.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D111728
The dependency scanner works with multiple instances of `Compiler{Instance,Invocation}`. From names of the variables/members, their purpose is not obvious.
This patch gives a distinct name to the `CompilerInstance` that's used to run the implicit build during dependency scan.
Depends on D111724.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D111725
The `ModuleDepCollectorPP` class holds a reference to `ModuleDepCollector` as well as `ModuleDepCollector`'s `CompilerInstance`. The fact that these refer to the same object is non-obvious.
This patch removes the `CompilerInvocation` reference from `ModuleDepCollectorPP` and accesses it through `ModuleDepCollector` instead.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D111724
One of main goals of the dependency scanner is to be strict about module compatibility. This is achieved through strict context hash. This patch ensures that strict context hash is enabled not only during the scan itself (and its minimized implicit build), but also when actually reporting the dependency.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D111720
Adds `selectBound`, a `Stencil` combinator that allows the user to supply multiple alternative cases, discriminated by bound node IDs.
Differential Revision: https://reviews.llvm.org/D111708
Previously it only used Windows command lines for MSVC triples, but this
was causing issues for windows-gnu. In fact, everything 'native' Windows
(ie, not Cygwin) should use Windows command line parsing.
Reviewed By: mstorsjo
Differential Revision: https://reviews.llvm.org/D111195
During explicit modular build, PCM files are typically specified via the `-fmodule-file=<path>` command-line option. Early during the compilation, Clang uses the `ASTReader` to read their contents and caches the result so that the module isn't loaded implicitly later on. A listener is attached to the `ASTReader` to collect names of the modules read from the PCM files. However, if the PCM has already been loaded previously via PCH:
1. the `ASTReader` doesn't do anything for the second time,
2. the listener is not invoked at all,
3. the module load result is not cached,
4. the compilation fails when attempting to load the module implicitly later on.
This patch solves this problem by attaching the listener to the `ASTReader` for PCH reading as well.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D111560
To reduce the number of explicit builds of a single module, we can try to squash multiple occurrences of the module with different command-lines (and context hashes) by removing benign command-line options. The greatest contributors to benign differences between command-lines are the header search paths.
In this patch, the lookup cache in `HeaderSearch` is used to identify paths that were actually used when implicitly building the module during scanning. This information is serialized into the unhashed control block of the implicitly-built PCM. The dependency scanner then loads this and may use it to prune the header search paths before computing the context hash of the module and generating the command-line.
We could also prune the header search paths when serializing `HeaderSearchOptions` into the PCM. That way, we could do it only once instead of every load of the PCM file by dependency scanner. However, that would result in a PCM file whose contents don't produce the same context hash as the original build, which is probably highly surprising.
There is an alternative approach to storing extra information into the PCM: wire up preprocessor callbacks to capture the used header search paths on-the-fly during preprocessing of modularized headers (similar to what we currently do for the main source file and textual headers). Right now, that's not compatible with the fact that we do an actual implicit build producing PCM files during dependency scanning. The second run of dependency scanner loads the PCM from the first run, skipping the preprocessing altogether, which would result in different results between runs. We can revisit this approach when we stop building implicitly during dependency scanning.
Depends on D102923.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D102488
As described on D111049, we're trying to remove the <string> dependency from error handling and replace uses of report_fatal_error(const std::string&) with the Twine() variant which can be forward declared.
Rename methods to clearly signal when they only deal with ASCII,
simplify the parsing of identifier, and use start/continue instead of
head/body for consistency with Unicode terminology.
During dependency scanning, we generally want to suppress -Werror. Apply the same logic to the DiagnosticOptions instance used for command-line parsing.
This fixes a test failure on the PS4 bot, where the system header directory could not be found, which was reported due to -Werror being on the command line and not being sanitized.
In this patch the dependency scanner starts using proper `DiagnosticOptions` parsed from the actual TU command-line in order to mimic what the actual compiler would do. The actual functionality will be enabled and tested in follow-up patches. (This split is necessary to avoid temporary regression.)
Depends on D108976.
Reviewed By: dexonsmith, arphaman
Differential Revision: https://reviews.llvm.org/D108982
This patch allows the clients of `ToolInvocation` to provide custom diagnostic options to be used during driver -> cc1 command-line transformation and parsing.
Tests covering this functionality are in a follow-up commit. To make this testable, the `DiagnosticsEngine` needs to be properly initialized via `CompilerInstance::createDiagnostics`.
Reviewed By: dexonsmith, arphaman
Differential Revision: https://reviews.llvm.org/D108976
The dependency scanner currently uses `ClangTool` to invoke the dependency scanning action.
However, `ClangTool` seems to be the wrong level of abstraction. It's intended to be run over a collection of compile commands, which we actively avoid via `SingleCommandCompilationDatabase`. It automatically injects `-fsyntax-only` and other flags, which we avoid by calling `clearArgumentsAdjusters()`. It deduces the resource directory based on the current executable path, which we'd like to change to deducing from `argv[0]`.
Internally, `ClangTool` uses `ToolInvocation` which seems to be more in line with what the dependency scanner tries to achieve. This patch switches to directly using `ToolInvocation` instead. NFC.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D108979
This patch changes how the dependency scanner creates the fake input file when scanning dependencies of a single module (introduced in D109485). The scanner now has its own `InMemoryFilesystem` which sits under the minimizing FS (when that's requested). This makes it possible to drop the duplicate work in `DependencyScanningActions::runInvocation` that sets up the main file ID. Besides that, this patch makes it possible to land D108979, where we drop `ClangTool` entirely.
Depends on D109485.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D109498
module lookup by name alone
This removes the need to create a fake source file that imports a
module.
rdar://64538073
Differential Revision: https://reviews.llvm.org/D109485
The way we parse `DiagnosticOptions` is a bit involved.
`DiagnosticOptions` are parsed as part of the cc1-parsing function `CompilerInvocation::CreateFromArgs` which takes `DiagnosticsEngine` as an argument to be able to report errors in command-line arguments. But to create `DiagnosticsEngine`, `DiagnosticOptions` are needed. This is solved by exposing the `ParseDiagnosticArgs` to clients and making its `DiagnosticsEngine` argument optional, essentially breaking the dependency cycle.
The `ParseDiagnosticArgs` function takes `llvm::opt::ArgList &`, which each client needs to create from the command-line (typically represented as `std::vector<const char *>`). Creating this data structure in this context is somewhat particular. This code pattern is copy-pasted in some places across the upstream code base and also in downstream repos. To make things a bit more uniform, this patch extracts the code into a new reusable function: `CreateAndPopulateDiagOpts`.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D108918
There are a number of language and preprocessor options that are reset in the `CompilerInvocation` that describes the build of an implicit module. This patch uses the logic for explicit modules as well.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D108710
Translation units with multiple direct modular dependencies trigger a non-deterministic ordering in `clang-scan-deps`. This boils down to usage of `std::unordered_map`, which gets replaced by `std::map` in this patch.
Depends on D103526.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D103807
The `ASTReader` populates `Module::PresumedModuleMapFile` only for top-level modules, not submodules. To avoid generating empty `-fmodule-map-file=` arguments, make discovered modules depend on top-level precompiled modules. The granularity of submodules is not important here.
The documentation of `Module::PresumedModuleMapFile` says this field is non-empty only when building from preprocessed source. This means there can still be cases where the dependency scanner generates empty `-fmodule-map-file=` arguments. That's being addressed in separate patch: D108544.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D108647
In this patch, the dependency scanner starts collecting precompiled dependencies from all encountered submodules, not only from top-level modules.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D108540
Some files still contained the old University of Illinois Open Source
Licence header. This patch replaces that with the Apache 2 with LLVM
Exception licence.
Differential Revision: https://reviews.llvm.org/D107528
When `-fno-integrated-as` is passed to the Clang driver (or set by default by a specific toolchain), it will construct an assembler job in addition to the cc1 job. Similarly, the `-fembed-bitcode` driver flag will create additional cc1 job that reads LLVM IR file.
The Clang tooling library only cares about the job that reads a source file. Instead of relying on the fact that the client injected `-fsyntax-only` to the driver invocation to get a single `-cc1` invocation that reads the source file, this patch filters out such jobs from `Compilation` automatically and ignores the rest.
This fixes a test failure in `ClangScanDeps/headerwithname.cpp` and `ClangScanDeps/headerwithnamefollowedbyinclude.cpp` on AIX reported here: https://reviews.llvm.org/D103461#2841918 and `clang-scan-deps` failures with `-fembed-bitcode`.
Depends on D106788.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D105695
Whenever -fmodule-name=top_level_module name is parsed, and clang actually tries to
import top_level_module, the headers are imported textually and the module isn't actually
built. However, the dependency scanner could still record it as a potential dependency
if the module was reimported and thus recorded by the preprocessor callbacks.
This change avoids collecting this kind of module as a dependency by verifying that we don't
collect top level modules without actual PCM files.
Differential Revision: https://reviews.llvm.org/D106100
This patch avoid minimizing input files that contributed to a PCH or its modules. This prevents the implicit modular build to fail on unexpected file size. Depends on D106146.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D104536
This patch separates the local and global caches of `DependencyScanningFilesystem` into two buckets: minimized files and original files. This is necessary to deal with precompiled modules/headers.
Consider a single worker with its instance of filesystem:
1. Build system uses the worker to scan dependencies of module A => filesystem cache gets populated with minimized input files.
2. Build system uses the results to explicitly build module A => explicitly built module captures the state of the real filesystem (containing non-minimized input files).
3. Build system uses the prebuilt module A as an explicit precompiled dependency for another compile job B.
4. Build system uses the same worker to scan dependencies for job B => worker uses implicit modular build to discover dependencies, which validates the filesystem state embedded in the prebuilt module (non-minimized files) to the current view of the filesystem (minimized files), resulting in validation failures.
This problem can be avoided in step 4 by collecting input files from the precompiled module and marking them as "ignored" in the minimizing filesystem. This way, the validation should succeed, since we should be always dealing with the original (non-minized) input files. However, the filesystem already minimized the input files in step 1 and put it in the cache, which gets used in step 4 as well even though it's marked ignored (do not minimize). This patch essentially fixes this oversight by making the `"file is minimized"` part of the cache key (from high level).
Depends on D106064.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D106146
This patch normalizes filenames in `DependencyScanningWorkerFilesystem` so that lookup of ignored files works correctly on Windows (where `/` and `\` are equivalent).
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D106064
Compilation database might have empty string as a command line argument.
But ExpandResponseFilesDatabase::expand doesn't expect this and assumes
that string.front() can be used for any argument. It is undefined behaviour if
string is empty. With debug build mode it causes crash in clangd.
Test Plan: check-clang
Differential Revision: https://reviews.llvm.org/D105120
This is mostly a mechanical change, but a testcase that contains
parts of the StringRef class (clang/test/Analysis/llvm-conventions.cpp)
isn't touched.
Currently, `access` doesn't recognize a dereferenced smart pointer. So,
`access(e, "field")` where `e = *x`, yields:
* `x->field`, for normal-pointer x,
* `(*x).field`, for smart-pointer x.
This patch normalizes handling of smart pointer to match normal pointer, when
the smart pointer type supports `->`.
Differential Revision: https://reviews.llvm.org/D104390
Currently, the implementation combines OOP and overloads, using a template to
tie the two together. In practice, this has proven confusing with no
benefits. This patch simplifies the code to use standard OOP design (a
collection of classes deriving from an interface).
Differential Revision: https://reviews.llvm.org/D104317
The dependency scanning worker uses `std::move` to "reset" `DependencyOutputOptions` in the `CompilerInstance` that performs the implicit build. It's probably preferable to replace the object with value-initialized instance, rather than depending on the behavior of a moved-from object.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D104106
There's no need to pass `DependencyOutputOptions` to each call of `handleFileDependency`, since the options don't ever change.
This patch adds new `handleDependencyOutputOpts` method to the `DependencyConsumer` interface and the dependency scanner uses it to report the options only once.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D104104
One of the goals of the dependency scanner is to generate command-lines that can be used to explicitly build modular dependencies of a translation unit. The only modifications to these command-lines should be for the purposes of explicit modular build.
However, the current version of dependency scanner leaks its implementation details into the command-lines.
The first problem is that the `clang-scan-deps` tool adjusts the original textual command-line (adding `-Eonly -M -MT <target> -sys-header-deps -Wno-error -o /dev/null `, removing `--serialize-diagnostics`) in order to set up the `DependencyScanning` library. This has been addressed in D103461, D104012, D104030, D104031, D104033. With these patches, the `DependencyScanning` library receives the unmodified `CompilerInvocation`, sets it up and uses it for the implicit modular build.
Finally, to prevent leaking the implementation details to the resulting command-lines, this patch generates them from the **original** unmodified `CompilerInvocation` rather than from the one that drives the implicit build.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D104036
This patch moves enabling system header deps from `clang-scan-deps` into the `DependencyScanning` library. This will make it easier to preserve semantics of the original TU command-line for modular dependencies (see D104036).
Reviewed By: arphaman
Differential Revision: https://reviews.llvm.org/D104033
This moves another piece of logic specific to `clang-scan-deps` into the `DependencyScanning` library. This makes it easier to check how the original command-line looked like in the library and will enable the library to stop inventing `-Wno-error` for modular dependencies (see D104036).
Reviewed By: arphaman
Differential Revision: https://reviews.llvm.org/D104031
The `clang-scan-deps` tool has some logic that parses and modifies the original Clang command-line. The goal is to setup `DependencyOutputOptions` by injecting `-M -MT <target>` and prevent the creation of output files.
This patch moves the logic into the `DependencyScanning` library, and uses the parsed `CompilerInvocation` instead of the raw command-line. The code simpler and can be used from the C++ API as well.
The `-o /dev/null` arguments are not necessary, since the `DependencyScanning` library only runs a preprocessing action, so there's no way it'll produce an actual object file.
Related: The `-M` argument implies `-w`, which would appear on the command-line of modular dependencies even though it was not on the original TU command line (see D104036).
Some related tests were updated.
Reviewed By: arphaman
Differential Revision: https://reviews.llvm.org/D104030
To prevent the creation of diagnostics file, `clang-scan-deps` strips the corresponding command-line argument. This behavior is useful even when using the C++ `DependencyScanner` library.
This patch transforms stripping of command-line in `clang-scan-deps` into stripping of `CompilerInvocation` in `DependencyScanning`.
AFAIK, the `clang-cl` driver doesn't even accept `--serialize-diagnostics`, so I've removed the test. (It would fail with an unknown command-line argument otherwise.)
Note: Since we're generating command-lines for modular dependencies from `CompilerInvocation`, the `--serialize-diagnostics` will be dropped. This was already happening in `clang-scan-deps` before this patch, but it will now happen also when using `DependencyScanning` library directly. This is resolved in D104036.
Reviewed By: dexonsmith, arphaman
Differential Revision: https://reviews.llvm.org/D104012
When a translation unit uses a PCH and imports the same modules as the PCH, we'd prefer to resolve to those modules instead of inventing new modules and reporting them as modular dependencies. Since the PCH modules have already been built nudge the compiler to reuse them when deciding whether to build a new module and don't report them as regular modular dependencies.
Depends on D103524 & D103802.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D103526
The `PreprocessOnlyAction` doesn't support loading the AST file of a precompiled header. This is problematic for dependency scanning, since the `#include` manufactured for the PCH is treated as textual. This means the PCH contents get scanned with each TU, which is redundant. Moreover, dependencies of the PCH end up being considered dependency of the TU.
To handle AST file of PCH properly, this patch creates new `FrontendAction` that behaves the same way `PreprocessorOnlyAction` does, but treats the manufactured PCH `#include` as a normal compilation would (by not claiming it only uses a preprocessor and creating the default AST consumer).
The AST file is now reported as a file dependency of the TU.
Depends on D103519.
Reviewed By: Bigcheese
Differential Revision: https://reviews.llvm.org/D103524
When a project uses PCH with explicit modules, the build will look like this:
1. scan PCH dependencies
2. explicitly build PCH
3. scan TU dependencies
4. explicitly build TU
Step 2 produces an object file for the PCH, which the dependency scanner needs to read in step 3. This patch adds support for this.
The `clang-scan-deps` invocation in the attached test would fail without this change.
Depends on D103516.
Reviewed By: Bigcheese
Differential Revision: https://reviews.llvm.org/D103519
This reapplies commit 95033eb3 that reverted commit 1d9e8e13.
The tests were failing on Windows due to spaces and backslashes in paths not being handled carefully.
This patch adds support for inferred modules to the dependency scanner.
Effectively a cherry-pick of https://github.com/apple/llvm-project/pull/699 authored by @Bigcheese with libclang and other changes omitted.
Contains following changes:
1. [Clang][ScanDeps] Ignore __inferred_module.map dependency.
* This shows up with inferred modules, but it doesn't exist on disk, so don't report it as a dependency.
2. [Clang][ScanDeps] Use the module map a module was inferred from for inferred modules.
Also includes a smoke test that uses clang-scan-deps output to perform an explicit build. There's no intention to duplicate whatever `test/Modules` contains, just to verify the produced command-line does "work" (with very loose definition of work).
Split from D100934.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D102495
This patch eagerly constructs and modifies CompilerInvocation of modular dependencies in order to report the correct context hash instead of the hash of the original translation unit.
No functionality change here, since we currently don't modify CompilerInvocation in a way that affects the context hash.
Depends on D102473.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D102482
The context hash of modular dependencies can be different from the context hash of the original translation unit if we modify their `CompilerInvocation`s.
Stop assuming the TU's context hash everywhere.
No functionality change here, since we're still currently using the unmodified TU CompilerInvocation to compute the context hash.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D102473
Stop using the compatibility spellings of `OF_{None,Text,Append}`
left behind by 1f67a3cba9. A follow-up
will remove them.
Differential Revision: https://reviews.llvm.org/D101650
Add option to `clang-scan-deps` to enable/disable generation of command-line arguments with absolute paths. This is essentially a revert of D100533, but with improved naming and added test.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D101051
Avoid string allocation in particular, but also avoid attempting to
impose any particular ordering based on formatted results.
Differential Revision: https://reviews.llvm.org/D101054
CommandLine.h is indirectly included in ~50% of TUs when building
clang, and VirtualFileSystem.h is large.
(Already remarked by jhenderson on D70769.)
No behavior change.
Differential Revision: https://reviews.llvm.org/D100957
This patch removes the `FullDependencies::AdditionalNonPathCommandLine` member, as it's value-initialized and never mutated.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D100536
This patch uses the new `CompilerInvocation::generateCC1CommandLine` to generate the full canonical command line for modular dependencies, instead of only appending additional arguments.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D100534
This patch removes the `-full-command-line` option from `clang-scan-deps`. It's only used with `-format=experimental-full`, where omitting the command lines doesn't make much sense. There are no tests without `-full-command-line`.
Depends on D100531.
Reviewed By: Bigcheese
Differential Revision: https://reviews.llvm.org/D100533
This patch simplifies (and renames) the `appendCommonModuleArguments` function.
It no longer tries to construct the command line for explicitly building modules. Instead, it only performs the DFS traversal of modular dependencies and queries the callbacks to collect paths to `.pcm` and `.modulemap` files.
This makes it more flexible and usable in two contexts:
* Generating additional command line arguments for the main TU in modular build. The `std::vector<std::string>` output parameters can be used to manually generate appropriate command line flags.
* Generate full command line for a module. The output parameters can be the corresponding parts of `CompilerInvocation`. (In a follow-up patch.)
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D100531
Saves running the generate inc script in the, somewhat common, case where the json file doesn't need changing.
Reviewed By: steveire
Differential Revision: https://reviews.llvm.org/D100719
Extend the matchers gathering API for types to record template
parameters. The TypeLoc type hierarchy has some types which are
templates used in CRTP such as PointerLikeTypeLoc. Record the inherited
template and template arguments of types inheriting those CRTP types in
the ClassInheritance map. Because the name inherited from is now
computed, the value type in that map changes from StringRef to
std::string. This also causes the toJSON override signature used to
serialize that map to change.
Remove the logic for skipping over empty ClassData instances. Several
classes such as TypeOfExprTypeLoc inherit a CRTP class which provides
interesting locations though the derived class does not. Record it as a
class to make the locations it inherits available.
Record the typeSourceInfo accessors too as they provide access to
TypeLocs in many classes.
The existing unit tests use UnorderedElementsAre to compare the
introspection result with the expected result. Our current
implementation of google mock (in gmock-generated-matchers.h) is limited
to support for comparing a container of 10 elements. As we are now
returning more than 10 results for one of the introspection tests,
change it to instead compare against an ordered vector of pairs.
Because a macro is used to generate API strings and API calls, disable
clang-format in blocks of expected results. Otherwise clang-format
would insert whitespaces which would then be compared against the
introspected strings and fail the test.
Introduce a recursion guard in the generated code. The TypeLoc class
has IgnoreParens() API which by default returns itself, so it would
otherwise recurse infinitely.
Differential Revision: https://reviews.llvm.org/D100516
This class initially had args to be generic to future needs. In
particular, I thought that source location introspection should show the
getBeginLoc of CallExpr args and the getArgLoc of
TemplateSpecializationLocInfo etc. However, that is probably best left
out of source location introspection because it involves node traversal.
If something like this is needed in the future, it can be added in the
future.
Differential Revision: https://reviews.llvm.org/D100688
This could probably be made into a compile time constant, but that would involve generating a second inc file.
Reviewed By: steveire
Differential Revision: https://reviews.llvm.org/D100530
Add a print method that takes a raw_ostream.
Change LocationCallFormatterCpp::format to call that method.
Reviewed By: steveire
Differential Revision: https://reviews.llvm.org/D100423
We can use the Preprocessor to remap this file, cleaning up the cmake code.
Reviewed By: steveire
Differential Revision: https://reviews.llvm.org/D100343
Fix the logic of detecting pseudo-virtual getBeginLoc etc on Stmt and
Decl subclasses.
Adjust the test infrastructure to filter out invalid source locations.
This makes the tests more clear about which nodes have which locations.
Differential Revision: https://reviews.llvm.org/D99231
clang Tooling, and more specifically Refactoring/Rename, have support
code to extract source locations given a Unified Symbol Resolution set.
This support code is used by clang-rename and other tools that might not
be in the tree.
Currently field designated initializer are not supported.
So, renaming S::a to S::b in this code:
S s = { .a = 10 };
will not extract the field designated initializer for a (the 'a' after the
dot).
This patch adds support for field designated initialized to
RecursiveSymbolVisitor and RenameLocFinder that is used in
createRenameAtomicChanges.
Differential Revision: https://reviews.llvm.org/D100310
Fixes bug http://bugs.llvm.org/show_bug.cgi?id=49000.
This patch allows Clang-Tidy checks to do
diag(X->getLocation(), "text") << Y->getSourceRange();
and get the highlight of `Y` as expected:
warning: text [blah-blah]
xxx(something)
^ ~~~~~~~~~
Reviewed-By: aaron.ballman, njames93
Differential Revision: http://reviews.llvm.org/D98635
Remove the find_package(Python3 ...) call from Tooling/CMakeLists.txt as
it would override the python 3 version determined in llvm/CMakeLists.txt.
This call did not respect the LLVM_MINIMUM_PYTHON_VERSION.
This fixes the check-all target when building LLVM on a system where the
default python version is not the minimum required version for running tests.
Reviewed By: serge-sans-paille
Differential Revision: https://reviews.llvm.org/D99715
In D84673, we started using `DiagnosticsEngine` during command-line parsing in more contexts.
When using `ToolInvocation`, a custom `DiagnosticsConsumer` can be specified and it might expect `SourceManager` to be present on the emitted diagnostics.
This patch ensures the `SourceManager` is set up in such scenarios.
Test authored by Jordan Rupprecht.
Reviewed By: rupprecht
Differential Revision: https://reviews.llvm.org/D99414
`expandedTokens(SourceRange)` used to do a binary search to get the
expanded tokens belonging to a source range. Each binary search uses
`isBeforeInTranslationUnit` to order two source locations. This is
inherently very slow.
By profiling clangd we found out that users like clangd::SelectionTree
spend 95% of time in `isBeforeInTranslationUnit`. Also it is worth
noting that users of `expandedTokens(SourceRange)` majorly use ranges
provided by AST to query this funciton. The ranges provided by AST are
token ranges (starting at the beginning of a token and ending at the
beginning of another token).
Therefore we can avoid the binary search in majority of the cases by
maintaining an index of ExpandedToken by their SourceLocations. We still
do binary search for ranges which are not token ranges but such
instances are quite low.
Performance:
`~/build/bin/clangd --check=clang/lib/Serialization/ASTReader.cpp`
Before: Took 2:10s to complete.
Now: Took 1:13s to complete.
Differential Revision: https://reviews.llvm.org/D99086
In future patches I will be setting the IsText parameter frequently so I will refactor the args to be in the following order. I have removed the FileSize parameter because it is never used.
```
static ErrorOr<std::unique_ptr<MemoryBuffer>>
getFile(const Twine &Filename, bool IsText = false,
bool RequiresNullTerminator = true, bool IsVolatile = false);
static ErrorOr<std::unique_ptr<MemoryBuffer>>
getFileOrSTDIN(const Twine &Filename, bool IsText = false,
bool RequiresNullTerminator = true);
static ErrorOr<std::unique_ptr<MB>>
getFileAux(const Twine &Filename, uint64_t MapSize, uint64_t Offset,
bool IsText, bool RequiresNullTerminator, bool IsVolatile);
static ErrorOr<std::unique_ptr<WritableMemoryBuffer>>
getFile(const Twine &Filename, bool IsVolatile = false);
```
Reviewed By: jhenderson
Differential Revision: https://reviews.llvm.org/D99182
As of CMake commit https://gitlab.kitware.com/cmake/cmake/-/commit/d993ebd4,
which first appeared in CMake 3.19.x series, in the compile commands for
clang-cl, CMake puts `--` before the input file. When operating on such a
database, the `InterpolatingCompilationDatabase` - specifically, the
`TransferableCommand` constructor - does not recognize that pattern and so, does
not strip the input, or the double dash when 'transferring' the compile command.
This results in a incorrect compile command - with the double dash and old input
file left in, and the language options and new input file appended after them,
where they're all treated as inputs, including the language version option.
Test files for some tests have names similar enough to be matched to commands
from the database, e.g.:
`.../path-mappings.test.tmp/server/bar.cpp`
can be matched to:
`.../Driver/ToolChains/BareMetal.cpp`
etc. When that happens, the tool being tested tries to use the matched, and
incorrectly 'transferred' compile command, and fails, reporting errors similar
to:
`error: no such file or directory: '/std:c++14'; did you mean '/std:c++14'? [clang-diagnostic-error]`
This happens in at least 4 tests:
Clang Tools :: clang-tidy/checkers/performance-trivially-destructible.cpp
Clangd :: check-fail.test
Clangd :: check.test
Clangd :: path-mappings.test
The fix for `TransferableCommand` removes the `--` and everything after it when
determining the arguments that apply to the new file. `--` is inserted in the
'transferred' command if the new file name starts with `-` and when operating in
clang-cl mode, also `/`. Additionally, other places in the code known to do
argument adjustment without accounting for the `--` and causing the tests to
fail are fixed as well.
Differential Revision: https://reviews.llvm.org/D98824
This patch documents how `ModuleDepCollector{,PP}` work and what their members store. Also renames somewhat vague `MainDeps` to `FileDeps` and `Deps` to `ModularDeps`.
Depends on D98943.
Reviewed By: Bigcheese
Differential Revision: https://reviews.llvm.org/D98950
This patch extracts the `ModuleName` and `ContextHash` members of `ClangModuleDep`, `FullDependencies` and `ModuleDeps` into a single struct `ModuleID`. This makes it easier to understand how the full dependency graph works.
Reviewed By: Bigcheese, dexonsmith
Differential Revision: https://reviews.llvm.org/D98943
Generate a json file containing descriptions of AST classes and their
public accessors which return SourceLocation or SourceRange.
Use the JSON file to generate a C++ API and implementation for accessing
the source locations and method names for accessing them for a given AST
node.
This new API can be used to implement 'srcloc' output in clang-query:
http://ce.steveire.com/z/m_kTIo
The JSON file can also be used to generate bindings for other languages,
such as Python and Javascript:
https://steveire.wordpress.com/2019/04/30/the-future-of-ast-matching
In this first version of this feature, only the accessors for Stmt
classes are generated, not Decls, TypeLocs etc. Those can be added
after this change is reviewed, as this change is mostly about
infrastructure of these code generators.
Also in this version, the platforms/cmake configurations are excluded as
much as possible so that support can be added iteratively. Currently a
break on any platform causes a revert of the entire feature. This way,
the `OR WIN32` can be removed in a future commit and if it breaks the
buildbots, only that commit gets reverted, making the entire process
easier to manage.
Differential Revision: https://reviews.llvm.org/D93164
Generate a json file containing descriptions of AST classes and their
public accessors which return SourceLocation or SourceRange.
Use the JSON file to generate a C++ API and implementation for accessing
the source locations and method names for accessing them for a given AST
node.
This new API can be used to implement 'srcloc' output in clang-query:
http://ce.steveire.com/z/m_kTIo
The JSON file can also be used to generate bindings for other languages,
such as Python and Javascript:
https://steveire.wordpress.com/2019/04/30/the-future-of-ast-matching
In this first version of this feature, only the accessors for Stmt
classes are generated, not Decls, TypeLocs etc. Those can be added
after this change is reviewed, as this change is mostly about
infrastructure of these code generators.
Also in this version, the platforms/cmake configurations are excluded as
much as possible so that support can be added iteratively. Currently a
break on any platform causes a revert of the entire feature. This way,
the `OR WIN32` can be removed in a future commit and if it breaks the
buildbots, only that commit gets reverted, making the entire process
easier to manage.
Differential Revision: https://reviews.llvm.org/D93164
Generate a json file containing descriptions of AST classes and their
public accessors which return SourceLocation or SourceRange.
Use the JSON file to generate a C++ API and implementation for accessing
the source locations and method names for accessing them for a given AST
node.
This new API can be used to implement 'srcloc' output in clang-query:
http://ce.steveire.com/z/m_kTIo
The JSON file can also be used to generate bindings for other languages,
such as Python and Javascript:
https://steveire.wordpress.com/2019/04/30/the-future-of-ast-matching
In this first version of this feature, only the accessors for Stmt
classes are generated, not Decls, TypeLocs etc. Those can be added
after this change is reviewed, as this change is mostly about
infrastructure of these code generators.
Also in this version, the platforms/cmake configurations are excluded as
much as possible so that support can be added iteratively. Currently a
break on any platform causes a revert of the entire feature. This way,
the `OR WIN32` can be removed in a future commit and if it breaks the
buildbots, only that commit gets reverted, making the entire process
easier to manage.
Differential Revision: https://reviews.llvm.org/D93164
Generate a json file containing descriptions of AST classes and their
public accessors which return SourceLocation or SourceRange.
Use the JSON file to generate a C++ API and implementation for accessing
the source locations and method names for accessing them for a given AST
node.
This new API can be used to implement 'srcloc' output in clang-query:
http://ce.steveire.com/z/m_kTIo
In this first version of this feature, only the accessors for Stmt
classes are generated, not Decls, TypeLocs etc. Those can be added
after this change is reviewed, as this change is mostly about
infrastructure of these code generators.
Differential Revision: https://reviews.llvm.org/D93164
Generate a json file containing descriptions of AST classes and their
public accessors which return SourceLocation or SourceRange.
Use the JSON file to generate a C++ API and implementation for accessing
the source locations and method names for accessing them for a given AST
node.
This new API can be used to implement 'srcloc' output in clang-query:
http://ce.steveire.com/z/m_kTIo
In this first version of this feature, only the accessors for Stmt
classes are generated, not Decls, TypeLocs etc. Those can be added
after this change is reviewed, as this change is mostly about
infrastructure of these code generators.
Differential Revision: https://reviews.llvm.org/D93164
Currently our strategy for getting header compile flags is something like:
A) look for flags for the header in compile_commands.json
This basically never works, build systems don't generate this info.
B) try to match to an impl file in compile_commands.json and use its flags
This only (mostly) works if the headers are in the same project.
C) give up and use fallback flags
This kind of works for stdlib in the default configuration, and
otherwise doesn't.
Obviously there are big gaps here.
This patch inserts a new attempt between A and B: if the header is
transitively included by any open file (whether same project or not),
then we use its compile command.
This doesn't make any attempt to solve some related problems:
- parsing non-self-contained header files in context (importing PP state)
- using the compile flags of non-opened candidate files found in the index
Fixes https://github.com/clangd/clangd/issues/123
Fixes https://github.com/clangd/clangd/issues/695
See https://github.com/clangd/clangd/issues/519
Differential Revision: https://reviews.llvm.org/D97351
OpaqueValueExpr doesn't correspond to the concrete syntax, it has
invalid source location, ignore them.
Reviewed By: kbobyrev
Differential Revision: https://reviews.llvm.org/D96112
The EndLoc of a type loc can be invalid for broken code.
Also extend the existing test to support error code with `error-ok`
annotation.
Differential Revision: https://reviews.llvm.org/D96261
Fail gracefully instead. Prevent further misuse by enforcing the factory builder
instead of the constructor.
Differential Revision: https://reviews.llvm.org/D94420
The class `SymbolOccurrences` can store either a single `SourceRange`
in-place or multiple `SourceRanges` on the heap. In the latter case
the number of source ranges is stored in the internal representation
of the beginning `SourceLocation` of the in-place `SourceRange`
object.
This change gets rid of such hack by placing `SourceRange` in a union
which holds either a valid `SourceRange` or an `unsigned int` (a number
of ranges).
The change also adds `static_assert`s that check that `SourceRange` and
`SourceLocation` are trivially destructible (this is required for the
current patch and for D94237 which has already been committed).
Reviewed By: MarkMurrayARM, simon_tatham
Differential Revision: https://reviews.llvm.org/D94599
This patch changes the default range used to anchor the include insertion to use
an expansion loc. This ensures that the location is valid, when the user relies
on the default range.
Driveby: extend a FIXME for a problem that was emphasized by this change; fix some spellings.
Differential Revision: https://reviews.llvm.org/D93703
Stencils `maybeDeref` and `maybeAddressOf` are designed to handle nodes that may
be pointers. Currently, they only handle native pointers. This patch extends the
support to recognize smart pointers and handle them as well.
Differential Revision: https://reviews.llvm.org/D93637
Allow a `std::unique_ptr` to be moved into the an `IntrusiveRefCntPtr`,
and remove a couple of now-unnecessary `release()` calls.
Differential Revision: https://reviews.llvm.org/D92888
Previously, loading one from a file meant allowing the library to do the IO.
Clangd would prefer to do such IO itself (e.g. to allow caching).
Differential Revision: https://reviews.llvm.org/D92640
This function doesn't seem to be used in-tree outside tests.
However clangd wants to use it soon, and having the CDB be self-contained seems
reasonable.
Differential Revision: https://reviews.llvm.org/D92646
A number of declarations were leftover after the move from `clang::tooling` to
`clang::transformer`. This patch removes those declarations and upgrades the
handful of references to the deprecated declarations.
Differential Revision: https://reviews.llvm.org/D92340
Currently, `node` only includes the semicolon for (some) statements. However,
declarations have the same issue of (potentially) trailing semicolons, so `node`
should behave the same for them.
Differential Revision: https://reviews.llvm.org/D91872
Non-mechanical changes:
- Added FIXME to StringLiteral to cover multi-token string literals.
- LiteralExpression::getLiteralToken() is gone. (It was never called)
This is because we don't codegen methods in Alternatives
It's conceptually suspect if we consider multi-token string literals, though.
Differential Revision: https://reviews.llvm.org/D91277
Jan Svoboda