This was done as a test for D137302 and it makes sense to push these changes
Reviewed By: shafik
Differential Revision: https://reviews.llvm.org/D137491
TokenManager defines Token interfaces for the clang syntax-tree. This is the level
of abstraction that the syntax-tree should use to operate on Tokens.
It decouples the syntax-tree from a particular token implementation (TokenBuffer
previously). This enables us to use a different underlying token implementation
for the syntax Leaf node -- in clang pseudoparser, we want to produce a
syntax-tree with its own pseudo::Token rather than syntax::Token.
Differential Revision: https://reviews.llvm.org/D128411
This avoids an unnecessary copy required by 'return OS.str()', allowing
instead for NRVO or implicit move. The .str() call (which flushes the
stream) is no longer required since 65b13610a5,
which made raw_string_ostream unbuffered by default.
Differential Revision: https://reviews.llvm.org/D115374
This gives us slightly nicer syntax (foreach) for idioms currently expressed
as a loop, and the option to use range algorithms where it makes sense
(e.g. llvm::all_of et al encapsulate the needed flow control in a useful way).
It's also a building block for iteration over filtered views (e.g. iterate over
all Stmt children, with the right type):
for (const Statement &S : filter<Statement>(N.children()))
...
I realize the recent direction has been mostly towards strongly-typed
node-specific facilities, but I think it's important we have convenient
generic facilities too.
Differential Revision: https://reviews.llvm.org/D90023
There can be Macros that are tagged with `modifiable`. Thus verifying
`canModifyAllDescendants` is not sufficient to avoid macros when deep
copying.
We think the `TokenBuffer` could inform us whether a `Token` comes from
a macro. We'll look into that when we can surface this information
easily, for instance in unit tests for `ComputeReplacements`.
Differential Revision: https://reviews.llvm.org/D88034
* Introduce `TreeTest.cpp` to unit test `Tree.h`
* Add `generateAllTreesWithShape` to generating test cases
* Add tests for `findFirstLeaf` and `findLastLeaf`
* Fix implementations of `findFirstLeaf` and `findLastLeaf` that had
been broken when empty `Tree` were present.
Differential Revision: https://reviews.llvm.org/D87779
We do that because:
* Big tests generated big tree dumps that could hardly serve as documentation.
* In most cases the tests didn't share setup, thus there was not much addition in lines of code.
We split tests for:
* `UserDefinedLiteral`
* `NestedBinaryOperator`
* `UserDefinedBinaryOperator`
* `UserDefinedPrefixOperator`
* `QualifiedId`
Differential Revision: https://reviews.llvm.org/D85819
For an user define `<`, `x < y` would yield the syntax tree:
```
BinaryOperatorExpression
|-IdExpression
| `-UnqualifiedId
| `-x
|-IdExpression
| `-UnqualifiedId
| `-<
`-IdExpression
`-UnqualifiedId
`-y
```
But there is no syntatic difference at call site between call site or
built-in `<`. As such they should generate the same syntax tree, namely:
```
BinaryOperatorExpression
|-IdExpression
| `-UnqualifiedId
| `-x
|-<
`-IdExpression
`-UnqualifiedId
`-y
```
Differential Revision: https://reviews.llvm.org/D85750
Summary:
We want NestedNameSpecifier syntax nodes to be generally supported, not
only for `DeclRefExpr` and `DependentScopedDeclRefExpr`.
To achieve this we:
* Use the `RecursiveASTVisitor`'s API to traverse
`NestedNameSpecifierLoc`s and automatically create its syntax nodes
* Add links from the `NestedNameSpecifierLoc`s to their syntax nodes.
In this way, from any semantic construct that has a `NestedNameSpecifier`,
we implicitly generate its syntax node via RAV and we can easily access
this syntax node via the links we added.
This is our grammar rule for nested-name-specifiers:
globalbal-specifier:
/*empty*/
simple-template-specifier:
template_opt simple-template-id
name-specifier:
global-specifier
decltype-specifier
identifier
simple-template-specifier
nested-name-specifier:
list(name-specifier, ::, non-empty, terminated)
It is a relaxed version of C++ [expr.prim.id] and quite simpler to map to our API.
TODO: refine name specifiers, `simple-template-name-specifier` and
decltype-name-specifier` are token soup for now.
This was the last binary operator that we supported but didn't have any
test coverage. The recent fix in a crash in member pointers allowed us
to add this test.
Differential Revision: https://reviews.llvm.org/D85185
Currently an Arena can only be built while consuming a TokenBuffer,
some users (like clangd) might want to share a TokenBuffer with multiple
compenents. This patch changes Arena's TokenBuffer member to be a reference so
that it can be created with read-only token buffers.
Differential Revision: https://reviews.llvm.org/D84973
Summary:
Given an UserDefinedLiteral `1.2_w`:
Problem: Lexer generates one Token for the literal, but ClangAST
references two source locations
Fix: Ignore the operator and interpret it as the underlying literal.
e.g.: `1.2_w` token generates syntax node IntegerLiteral(1.2_w)
Subscribers: cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D82157
Summary:
Previously, AST Matchers tests were using a custom way to run a test
with a specific C++ standard version. I'm migrating them to a shared
infrastructure to specify a Clang target from libClangTesting. I'm also
changing tests for AST Matchers to run in multiple language standards
versions, and under multiple triples that have different behavior with
regards to templates.
To keep the size of the patch manageable, in this patch I'm only
migrating one file to get the process started and get feedback on this
approach.
One caveat is that increasing the number of test configuration does
significantly increase the runtime of AST Matchers tests. On my machine,
the test runtime increases from 2.0 to 6.0s. I think it is worth the
improved test coverage.
Reviewers: jdoerfert, ymandel
Reviewed By: ymandel
Subscribers: gribozavr2, jfb, sstefan1, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D82179
Summary:
MSVC does not handle raw string literals with embedded double quotes
correctly. I switched the affected test case to use regular string
literals insetad.
Subscribers: cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D82636
Summary:
The macro `INSTANTIATE_TEST_CASE_P` is defined as
```
\# define INSTANTIATE_TEST_CASE_P(prefix, test_case_name, generator, ...) \
...
```
If we build the test case with -werror, we will get an error like
```
error: ISO C++11 requires at least one argument for the "..." in a
variadic macro
testing::ValuesIn(TestClangConfig::allConfigs()));
^
```
This patch fixes that.
Reviewers: gribozavr, hlopko, eduucaldas, gribozavr2
Reviewed By: gribozavr2
Subscribers: cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D81388
Summary:
The syntax tree test uses a helper function that executes all testing
assertions. When an assertion fails, the only line number that gets
printed to the log refers to the helper function. After this change, we
would also get the line number of the EXPECT_TRUE macro invocation
(unfortunately, the line number of the last token of it, not the first
one, but there's not much I can do about it).
Reviewers: hlopko, eduucaldas
Reviewed By: hlopko, eduucaldas
Subscribers: cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D81107
Summary:
Doing so allows us to increase test coverage by removing unnecessary
language restrictions.
Reviewers: hlopko, eduucaldas
Reviewed By: hlopko, eduucaldas
Subscribers: gribozavr2, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D81040
Summary:
I changed `markStmtChild` to ignore implicit expressions the same way as
`markExprChild` does it already. The test that I modified crashes
without this change.
Reviewers: hlopko, eduucaldas
Reviewed By: hlopko, eduucaldas
Subscribers: gribozavr2, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D81019
Summary:
I think we would be better off with tests explicitly specifying the
language mode. Right now Lang_C means C99, but reads as "any C version",
or as "unspecified C version".
I also changed '-std=c++98' to '-std=c++03' because they are aliases (so
there is no difference in practice), because Clang implements C++03
rules in practice, and because 03 makes a nice sortable progression
between 03, 11, 14, 17, 20.
Reviewers: shafik, hlopko
Reviewed By: hlopko
Subscribers: jfb, martong, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D81000
Summary:
* Test if the code sourcing the SyntaxTree compiles
* Output compiler errors and warnings to err
* Fix tests with code that did not compile
Reviewers: gribozavr2
Reviewed By: gribozavr2
Subscribers: cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D80731
Summary:
Same restrictions apply as in the other direction: macro arguments are
not supported yet, only full macro expansions can be mapped.
Taking over from https://reviews.llvm.org/D72581.
Reviewers: gribozavr2, sammccall
Reviewed By: gribozavr2
Subscribers: cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D77209
Summary:
This patch removes delayed folding and replaces it with forward peeking.
Delayed folding was previously used as a solution to the problem that
declaration doesn't have a representation in the AST. For example following
code:
```
int a,b;
```
is expressed in the AST as:
```
TranslationUnitDecl
|-...
|-VarDecl `int a`
`-VarDecl `int b`
```
And in the syntax tree we need:
```
*: TranslationUnit
`-SimpleDeclaration
|-int
|-SimpleDeclarator
| `-a
|-,
|-SimpleDeclarator
| `-b
|-;
```
So in words, we need to create SimpleDeclaration to be a parent of
SimpleDeclarator nodes. Previously we used delayed folding to make sure SimpleDeclarations will be
eventually created. And in case multiple declarators requested declaration
creation, declaration range was extended to cover all declarators.
This design started to be hard to reason about, so we decided to replace it with
forward peeking. The last declarator node in the chain is responsible for creating
SimpleDeclaration for the whole chain. Range of the declaration corresponds to
the source range of the declarator node. Declarator decides whether its the last
one by peeking to the next AST node (see `isResponsibleForCreatingDeclaration`).
This patch does following:
* Removed delayed folding logic
* Tweaks Token.dumpForTests
* Moves getQualifiedNameStart inside BuildTreeVisitor
* Extracts BuildTreeVisitor.ProcessDeclaratorAndDeclaration
* Renames Builder.getDeclRange to Builder.getDeclarationRange and uses the
method in all places.
* Adds a bunch of tests
Reviewers: gribozavr2
Reviewed By: gribozavr2
Subscribers: cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D76922
Summary:
This increases the coverage for things that differ between Linux and Windows, such as -fdelayed-template-parsing. This would have prevented the rollback of https://reviews.llvm.org/D76346.
While at it, update -std=c++11 to c++17 for the test.
Reviewers: gribozavr2
Reviewed By: gribozavr2
Subscribers: cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D76497
Summary:
This increases the coverage for things that differ between Linux and Windows, such as `-fdelayed-template-parsing`. This would have prevented the rollback of https://reviews.llvm.org/D76346.
While at it, update -std=c++11 to c++17 for the test.
Reviewers: gribozavr2
Reviewed By: gribozavr2
Subscribers: cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D76433
Summary:
This patch split Basic test into multple individual tests to allow simpler
filtering and clearer signal into what's broken when it's broken.
Reviewers: gribozavr2
Reviewed By: gribozavr2
Subscribers: cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D76366
Summary:
Rollforward of
https://reviews.llvm.org/rGdd12826808f9079e164b82e64b0697a077379241 after
temporarily adding -fno-delayed-template-parsing to the TreeTest.
Original summary:
> Copy of https://reviews.llvm.org/D72334, submitting with Ilya's permission.
>
> Handles template declaration of all kinds.
>
> Also builds template declaration nodes for specializations and explicit
> instantiations of classes.
>
> Some missing things will be addressed in the follow-up patches:
>
> * specializations of functions and variables,
> * template parameters.
Reviewers: gribozavr2
Subscribers: cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D76418
Nathan James