TokenManager defines Token interfaces for the clang syntax-tree. This is the level
of abstraction that the syntax-tree should use to operate on Tokens.
It decouples the syntax-tree from a particular token implementation (TokenBuffer
previously). This enables us to use a different underlying token implementation
for the syntax Leaf node -- in clang pseudoparser, we want to produce a
syntax-tree with its own pseudo::Token rather than syntax::Token.
Differential Revision: https://reviews.llvm.org/D128411
LLVM Programmer’s Manual strongly discourages the use of `std::vector<bool>` and suggests `llvm::BitVector` as a possible replacement.
This patch replaces `std::vector<bool>` with `llvm::BitVector` in the Syntax library and replaces range-based for loop with regular for loop. This is necessary due to `llvm::BitVector` not having `begin()` and `end()` (D117116).
Reviewed By: dexonsmith, dblaikie
Differential Revision: https://reviews.llvm.org/D118109
This avoids an unnecessary copy required by 'return OS.str()', allowing
instead for NRVO or implicit move. The .str() call (which flushes the
stream) is no longer required since 65b13610a5,
which made raw_string_ostream unbuffered by default.
Differential Revision: https://reviews.llvm.org/D115374
So far, only used to generate Kind and implement classof().
My plan is to have this general-purpose Nodes.inc in the style of AST
DeclNodes.inc etc, and additionally a special-purpose backend generating
the actual class definitions. But baby steps...
Differential Revision: https://reviews.llvm.org/D90540
Rationale:
Children of a syntax tree had forward links only, because there was no
need for reverse links.
This need appeared when we started mutating the syntax tree.
On a forward list, to remove a target node in O(1) we need a pointer to the node before the target. If we don't have this "before" pointer, we have to find it, and that requires O(n).
So in order to remove a syntax node from a tree, we would similarly need to find the node before to then remove. This is both not ergonomic nor does it have a good complexity.
Differential Revision: https://reviews.llvm.org/D90240
This gives us slightly nicer syntax (foreach) for idioms currently expressed
as a loop, and the option to use range algorithms where it makes sense
(e.g. llvm::all_of et al encapsulate the needed flow control in a useful way).
It's also a building block for iteration over filtered views (e.g. iterate over
all Stmt children, with the right type):
for (const Statement &S : filter<Statement>(N.children()))
...
I realize the recent direction has been mostly towards strongly-typed
node-specific facilities, but I think it's important we have convenient
generic facilities too.
Differential Revision: https://reviews.llvm.org/D90023
This is an initial attempt to start using Syntax Trees in clangd while improving state of folding ranges feature and experimenting with Syntax Tree capabilities.
Reviewed By: sammccall
Differential Revision: https://reviews.llvm.org/D88553
* Introduce `TreeTest.cpp` to unit test `Tree.h`
* Add `generateAllTreesWithShape` to generating test cases
* Add tests for `findFirstLeaf` and `findLastLeaf`
* Fix implementations of `findFirstLeaf` and `findLastLeaf` that had
been broken when empty `Tree` were present.
Differential Revision: https://reviews.llvm.org/D87779
Previously a NodeRole would generally be prefixed with the `NodeKind`,
we remove this prefix, as it we redundant and made tests more noisy.
Differential Revision: https://reviews.llvm.org/D86636
We should see `NodeRole` information in the dump because that exposes how the
accessors will behave.
Functional changes in the dump:
* Surround Leaf tokens with `'`
* Append `Node` dumps with `NodeRole` information, except for unknown roles
* Append marks to `Node` dumps, instead of prepending
Non-functional changes:
* `::dumpTokens(llvm::raw_ostream, ArrayRef<syntax::Token>, const
SourceManager &SM)` always received as parameter a `syntax::Token *`
pointing to `Leaf::token()`. Changed the function to
`dumpLeaf(llvm::raw_ostream, syntax::Leaf *, const SourceManager&)`
* `dumpTree` acted on a Node, rename to `dumpNode`
Differential Revision: https://reviews.llvm.org/D85330
We defined a List construct to help with the implementation of list-like
grammar rules. This is a first implementation of this API.
Differential Revision: https://reviews.llvm.org/D85295
Currently an Arena can only be built while consuming a TokenBuffer,
some users (like clangd) might want to share a TokenBuffer with multiple
compenents. This patch changes Arena's TokenBuffer member to be a reference so
that it can be created with read-only token buffers.
Differential Revision: https://reviews.llvm.org/D84973
Summary:
Copy of https://reviews.llvm.org/D72446, submitting with Ilya's permission.
Only used to assign roles to child nodes for now. This is more efficient
than doing range-based queries.
In the future, will be exposed in the public API of syntax trees.
Reviewers: gribozavr2
Reviewed By: gribozavr2
Subscribers: cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D76355
Add checks for some structural invariants when building and mutating
the syntax trees.
Fix a bug failing the invariants after mutations: the parent of nodes
added into the tree was null.
Summary:
This patch adds facilities to mutate the syntax trees and produce
corresponding text replacements.
The public interface of the syntax library now includes facilities to:
1. perform type-safe modifications of syntax trees,
2. compute textual replacements to apply the modifications,
3. create syntax trees not backed by the source code.
For each of the three, we only add a few example transformations in this
patch to illustrate the idea, support for more kinds of nodes and
transformations will be done in follow-up patches.
The high-level mutation operations are implemented on top of operations
that allow to arbitrarily change the trees. They are considered to be
implementation details and are not available to the users of the
library.
Reviewers: sammccall, gribozavr2
Reviewed By: gribozavr2
Subscribers: merge_guards_bot, mgorny, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D64573
Summary:
Most of the statements mirror the ones provided by clang AST.
Major differences are:
- expressions are wrapped into 'ExpressionStatement' instead of being
a subclass of statement,
- semicolons are always consumed by the leaf expressions (return,
expression satement, etc),
- some clang statements are not handled yet, we wrap those into an
UnknownStatement class, which is not present in clang.
We also define an 'Expression' and 'UnknownExpression' classes in order
to produce 'ExpressionStatement' where needed. The actual implementation
of expressions is not yet ready, it will follow later.
Reviewers: sammccall
Reviewed By: sammccall
Subscribers: cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D63835
Summary:
A tooling-focused alternative to the AST. This commit focuses on the
memory-management strategy and the structure of the AST.
More to follow later:
- Operations to mutate the syntax trees and corresponding textual
replacements.
- Mapping between clang AST nodes and syntax tree nodes.
- More node types corresponding to the language constructs.
Reviewers: sammccall
Reviewed By: sammccall
Subscribers: llvm-commits, mgorny, cfe-commits
Tags: #clang, #llvm
Differential Revision: https://reviews.llvm.org/D61637
........
Fixes buildbots which were crashing on SyntaxTests.exe
llvm-svn: 365465
Summary:
A tooling-focused alternative to the AST. This commit focuses on the
memory-management strategy and the structure of the AST.
More to follow later:
- Operations to mutate the syntax trees and corresponding textual
replacements.
- Mapping between clang AST nodes and syntax tree nodes.
- More node types corresponding to the language constructs.
Reviewers: sammccall
Reviewed By: sammccall
Subscribers: llvm-commits, mgorny, cfe-commits
Tags: #clang, #llvm
Differential Revision: https://reviews.llvm.org/D61637
llvm-svn: 365355
Haojian Wu