`expandedTokens(SourceRange)` used to do a binary search to get the
expanded tokens belonging to a source range. Each binary search uses
`isBeforeInTranslationUnit` to order two source locations. This is
inherently very slow.
By profiling clangd we found out that users like clangd::SelectionTree
spend 95% of time in `isBeforeInTranslationUnit`. Also it is worth
noting that users of `expandedTokens(SourceRange)` majorly use ranges
provided by AST to query this funciton. The ranges provided by AST are
token ranges (starting at the beginning of a token and ending at the
beginning of another token).
Therefore we can avoid the binary search in majority of the cases by
maintaining an index of ExpandedToken by their SourceLocations. We still
do binary search for ranges which are not token ranges but such
instances are quite low.
Performance:
`~/build/bin/clangd --check=clang/lib/Serialization/ASTReader.cpp`
Before: Took 2:10s to complete.
Now: Took 1:13s to complete.
Differential Revision: https://reviews.llvm.org/D99086
OpaqueValueExpr doesn't correspond to the concrete syntax, it has
invalid source location, ignore them.
Reviewed By: kbobyrev
Differential Revision: https://reviews.llvm.org/D96112
The EndLoc of a type loc can be invalid for broken code.
Also extend the existing test to support error code with `error-ok`
annotation.
Differential Revision: https://reviews.llvm.org/D96261
Non-mechanical changes:
- Added FIXME to StringLiteral to cover multi-token string literals.
- LiteralExpression::getLiteralToken() is gone. (It was never called)
This is because we don't codegen methods in Alternatives
It's conceptually suspect if we consider multi-token string literals, though.
Differential Revision: https://reviews.llvm.org/D91277
Similar to the previous patch, this doesn't convert *all* the classes that
could be converted. It also doesn't enforce any new invariants etc.
It *does* include some data we don't use yet: specific token types that are
allowed and optional/required status of sequence items. (Similar to Dmitri's
prototype). I think these are easier to add as we go than later, and serve
a useful documentation purpose.
Differential Revision: https://reviews.llvm.org/D90659
So far, only used to generate Kind and implement classof().
My plan is to have this general-purpose Nodes.inc in the style of AST
DeclNodes.inc etc, and additionally a special-purpose backend generating
the actual class definitions. But baby steps...
Differential Revision: https://reviews.llvm.org/D90540
Rationale:
Children of a syntax tree had forward links only, because there was no
need for reverse links.
This need appeared when we started mutating the syntax tree.
On a forward list, to remove a target node in O(1) we need a pointer to the node before the target. If we don't have this "before" pointer, we have to find it, and that requires O(n).
So in order to remove a syntax node from a tree, we would similarly need to find the node before to then remove. This is both not ergonomic nor does it have a good complexity.
Differential Revision: https://reviews.llvm.org/D90240
This gives us slightly nicer syntax (foreach) for idioms currently expressed
as a loop, and the option to use range algorithms where it makes sense
(e.g. llvm::all_of et al encapsulate the needed flow control in a useful way).
It's also a building block for iteration over filtered views (e.g. iterate over
all Stmt children, with the right type):
for (const Statement &S : filter<Statement>(N.children()))
...
I realize the recent direction has been mostly towards strongly-typed
node-specific facilities, but I think it's important we have convenient
generic facilities too.
Differential Revision: https://reviews.llvm.org/D90023
This is an initial attempt to start using Syntax Trees in clangd while improving state of folding ranges feature and experimenting with Syntax Tree capabilities.
Reviewed By: sammccall
Differential Revision: https://reviews.llvm.org/D88553
The patch adjusts the existing `llvm::DenseMap<unsigned, T>` and
`llvm::DenseSet<unsigned>` objects that store source locations, so
that they use `SourceLocation` directly instead of `unsigned`.
This patch relies on the `DenseMapInfo` trait added in D89719.
It also replaces the construction of `SourceLocation` objects from
the constants -1 and -2 with calls to the trait's methods `getEmptyKey`
and `getTombstoneKey` where appropriate.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D69840
After this change all nodes that have a delimited-list are using the
`List` API.
Implementation details:
Let's look at a declaration with multiple declarators:
`int a, b;`
To generate a declarator list node we need to have the range of
declarators: `a, b`:
However, the `ClangAST` actually stores them as separate declarations:
`int a ;`
`int b;`
We solve that by appropriately marking the declarators on each separate
declaration in the `ClangAST` and then for the final declarator `int
b`, shrinking its range to fit to the already marked declarators.
Differential Revision: https://reviews.llvm.org/D88403
There can be Macros that are tagged with `modifiable`. Thus verifying
`canModifyAllDescendants` is not sufficient to avoid macros when deep
copying.
We think the `TokenBuffer` could inform us whether a `Token` comes from
a macro. We'll look into that when we can surface this information
easily, for instance in unit tests for `ComputeReplacements`.
Differential Revision: https://reviews.llvm.org/D88034
* Introduce `TreeTest.cpp` to unit test `Tree.h`
* Add `generateAllTreesWithShape` to generating test cases
* Add tests for `findFirstLeaf` and `findLastLeaf`
* Fix implementations of `findFirstLeaf` and `findLastLeaf` that had
been broken when empty `Tree` were present.
Differential Revision: https://reviews.llvm.org/D87779
Prior to this change `createTree` could not create arbitrary syntax
trees. Now it dispatches to the constructor of the concrete syntax tree
according to the `NodeKind` passed as argument. This allows reuse inside
the Synthesis API. # Please enter the commit message for your changes.
Lines starting
Differential Revision: https://reviews.llvm.org/D87820
Building on Mac OS with clang 12:
```
jhemphill@jhemphill-mbp build % clang --version
Apple clang version 12.0.0 (clang-1200.0.26.2)
Target: x86_64-apple-darwin19.6.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin
```
yields one warning:
```
/Users/jhemphill/oss/llvm-project/clang/lib/Tooling/Syntax/BuildTree.cpp:1126:22: warning: loop variable 'Arg' is always a copy because the range of type 'llvm::iterator_range<clang::Stmt::CastIterator<clang::Expr, clang::Expr *, clang::Stmt *> >' does not return a reference [-Wrange-loop-analysis]
for (const auto &Arg : Args) {
^
/Users/jhemphill/oss/llvm-project/clang/lib/Tooling/Syntax/BuildTree.cpp:1126:10: note: use non-reference type 'clang::Expr *'
for (const auto &Arg : Args) {
```
It appears that `Arg` is an `Expr*`, passed by value rather than by const reference.
Reviewed By: eduucaldas, gribozavr2
Differential Revision: https://reviews.llvm.org/D87482
In a future patch
* Implement helper function to generate Trees for tests
* and test Tree methods, namely `findFirstLeaf` and `findLastLeaf`
Differential Revision: https://reviews.llvm.org/D87533
* Do not visit `CXXDefaultArgExpr`
* To build `CallArguments` nodes, just go through non-default arguments
Differential Revision: https://reviews.llvm.org/D87249
Previously a NodeRole would generally be prefixed with the `NodeKind`,
we remove this prefix, as it we redundant and made tests more noisy.
Differential Revision: https://reviews.llvm.org/D86636
* Generate `CallExpression` syntax node for all semantic nodes inheriting from
`CallExpr` with call-expression syntax - except `CUDAKernelCallExpr`.
* Implement all the accessors
* Arguments of `CallExpression` have their own syntax node which is based on
the `List` base API
Differential Revision: https://reviews.llvm.org/D86544
We should see `NodeRole` information in the dump because that exposes how the
accessors will behave.
Functional changes in the dump:
* Surround Leaf tokens with `'`
* Append `Node` dumps with `NodeRole` information, except for unknown roles
* Append marks to `Node` dumps, instead of prepending
Non-functional changes:
* `::dumpTokens(llvm::raw_ostream, ArrayRef<syntax::Token>, const
SourceManager &SM)` always received as parameter a `syntax::Token *`
pointing to `Leaf::token()`. Changed the function to
`dumpLeaf(llvm::raw_ostream, syntax::Leaf *, const SourceManager&)`
* `dumpTree` acted on a Node, rename to `dumpNode`
Differential Revision: https://reviews.llvm.org/D85330
Utkarsh Saxena