A followup patch of d489b3807f, but for
member functions, this will eliminate a false parse of member
declaration.
Differential Revision: https://reviews.llvm.org/D131720
We happened to introduce a `member-declaration := ;` rule
when inlining the `member-declaration := decl-specifier-seq_opt
member-declarator-list_opt ;`.
And with the `member-declaration := empty-declaration` rule, we had two parses of `;`.
This patch is to restrict the grammar to eliminate the
`member-declaration := ;` rule.
Differential Revision: https://reviews.llvm.org/D131724
Now that we've updated to C++17 MSVC gives very verbose warnings about not creating classes that inherit from std::iterator - use llvm::iterator_facade_base instead
Fixes#57005
I went over the output of the following mess of a command:
`(ulimit -m 2000000; ulimit -v 2000000; git ls-files -z | parallel --xargs -0 cat | aspell list --mode=none --ignore-case | grep -E '^[A-Za-z][a-z]*$' | sort | uniq -c | sort -n | grep -vE '.{25}' | aspell pipe -W3 | grep : | cut -d' ' -f2 | less)`
and proceeded to spend a few days looking at it to find probable typos
and fixed a few hundred of them in all of the llvm project (note, the
ones I found are not anywhere near all of them, but it seems like a
good start).
Reviewed By: kadircet
Differential Revision: https://reviews.llvm.org/D130826
The solution is to favor the longest possible nest-name-specifier, and
drop other alternatives by using the guard, per per C++ [basic.lookup.qual.general].
Motivated cases:
```
Foo::Foo() {};
// the constructor can be parsed as:
// - Foo ::Foo(); // where the first Foo is return-type, and ::Foo is the function declarator
// + Foo::Foo(); // where Foo::Foo is the function declarator
```
```
void test() {
// a very slow parsing case when there are many qualifers!
X::Y::Z;
// The statement can be parsed as:
// - X ::Y::Z; // ::Y::Z is the declarator
// - X::Y ::Z; // ::Z is the declarator
// + X::Y::Z; // a declaration without declarator (X::Y::Z is decl-specifier-seq)
// + X::Y::Z; // a qualifed-id expression
}
```
Differential Revision: https://reviews.llvm.org/D130511
- Place rules under rule::lhs::rhs__rhs__rhs
- Change mangling of keywords to ALL_CAPS (needed to turn keywords that appear
alone on RHS into valid identifiers)
- Make enums implicitly convertible to underlying type (though still scoped,
using alias tricks)
In principle this lets us exhaustively write a switch over all rules of a NT:
switch ((rule::declarator)N->rule()) {
case rule::declarator::noptr_declarator:
...
}
In practice we don't do this anywhere yet as we're often switching over multiple
nonterminal kinds at once.
Differential Revision: https://reviews.llvm.org/D130414
This allows incomplete code such as `namespace foo {` to be modeled as a
normal sequence with the missing } represented by an empty opaque node.
Differential Revision: https://reviews.llvm.org/D130551
Motivating case: `foo bar;` is not a declaration of nothing with `foo` and `bar`
both types.
This is a common and critical ambiguity, clangd/AST.cpp has 20% fewer
ambiguous nodes (1674->1332) after this change.
Differential Revision: https://reviews.llvm.org/D130337
llvm::sort is beneficial even when we use the iterator-based overload,
since it can optionally shuffle the elements (to detect
non-determinism). However llvm::sort is not usable everywhere, for
example, in compiler-rt.
Reviewed By: nhaehnle
Differential Revision: https://reviews.llvm.org/D130406
- the grammar ambiguity is eliminated by a guard;
- modify the guard function signatures, now all parameters are folded in
to a single object, avoid a long parameter list (as we will add more
parameters in the near future);
Reviewed By: sammccall
Differential Revision: https://reviews.llvm.org/D130160
After this, NUMERIC_CONSTANT and strings should parse only one way.
There are 8 types of literals, and 24 valid (literal, TokenKind) pairs.
This means adding 8 new named guards (or 24, if we want to assert the token).
It seems fairly clear to me at this point that the guard names are unneccesary
indirection: the guards are in fact coupled to the rule signature.
(Also add the zero guard I forgot in the previous patch.)
Differential Revision: https://reviews.llvm.org/D130066
This eliminates some simple-declaration/function-definition false
parses.
- implement a function to determine whether a declarator ForestNode is a
function declarator;
- extend the standard declarator to two guarded function-declarator and
non-function-declarator nonterminals;
Differential Revision: https://reviews.llvm.org/D129222
This first version only uses bracket matching. We plan to extend this to
use DirectiveTree as well.
Also includes changes to Token to allow retrieving corresponding token
in token stream of original source file.
Differential Revision: https://reviews.llvm.org/D129648
`stripComments(cook(...))` is a common pattern being written.
Without this patch, this has a use-after-free issue (cook returns a temporary
TokenStream object which has its own payload, but the payload is not
shared with the one returned by stripComments).
Reviewed By: sammccall
Differential Revision: https://reviews.llvm.org/D125311
The idea is:
- a parse failure is detected when all heads die when trying to shift the next token
- we can recover by choosing a nonterminal we're partway through parsing, and
determining where it ends through nonlocal means (e.g. matching brackets)
- we can find candidates by walking up the stack from the (ex-)heads
- the token range is defined using heuristics attached to grammar rules
- the unparsed region is represented in the forest by an Opaque node
This patch has the core GLR functionality.
It does not allow recovery heuristics to be attached as extensions to
the grammar, but rather infers a brace-based heuristic.
Expected followups:
- make recovery heuristics grammar extensions (depends on D127448)
- add recovery to our grammar for bracketed constructs and sequence nodes
- change the structure of our augmented `_ := start` rules to eliminate some
special-cases in glrParse.
- (if I can work out how): avoid some spurious recovery cases described in comments
(Previously mistakenly committed as a0f4c10ae2)
Differential Revision: https://reviews.llvm.org/D128486
- Extend the GLR parser to allow conditional reduction based on the
guard functions;
- Implement two simple guards (contextual-override/final) for cxx.bnf;
- layering: clangPseudoCXX depends on clangPseudo (as the guard function need
to access the TokenStream);
Differential Revision: https://reviews.llvm.org/D127448
The actions table is very compact but the binary search to find the
correct action is relatively expensive.
A hashtable is faster but pretty large (64 bits per value, plus empty
slots, and lookup is constant time but not trivial due to collisions).
The structure in this patch uses 1.25 bits per entry (whether present or absent)
plus the size of the values, and lookup is trivial.
The Shift table is 119KB = 27KB values + 92KB keys.
The Goto table is 86KB = 30KB values + 57KB keys.
(Goto has a smaller keyspace as #nonterminals < #terminals, and more entries).
This patch improves glrParse speed by 28%: 4.69 => 5.99 MB/s
Overall the table grows by 60%: 142 => 228KB.
By comparison, DenseMap<unsigned, StateID> is "only" 16% faster (5.43 MB/s),
and results in a 285% larger table (547 KB) vs the baseline.
Differential Revision: https://reviews.llvm.org/D128485
- define a common data structure Language which is a compiled result of the
bnf grammar. It is defined in Language.h;
- creates a clangPseudoCLI lib which defines a grammar commandline flag and
expose a function to get the Language. It supports --grammar=cxx,
--grammmar=/path/to/file.bnf;
- use the clangPseudoCLI in clang-pseudo, fuzzer, and benchmark tools (
simplify the code and use the prebuilt cxx grammar);
Split out from https://reviews.llvm.org/D127448.
Differential Revision: https://reviews.llvm.org/D128679
Sam McCall