Commit Graph

809 Commits

Author SHA1 Message Date
Nico Weber c0cd5274cc [ast-dump-tool] fix regression if --empty-implementation but --json-input-path is not
Looks like this broke in one of the relands of
https://reviews.llvm.org/D93164
2021-03-15 08:48:26 -04:00
Stephen Kelly 19740652c4 [AST] Add generator for source location introspection
Generate a json file containing descriptions of AST classes and their
public accessors which return SourceLocation or SourceRange.

Use the JSON file to generate a C++ API and implementation for accessing
the source locations and method names for accessing them for a given AST
node.

This new API can be used to implement 'srcloc' output in clang-query:

  http://ce.steveire.com/z/m_kTIo

The JSON file can also be used to generate bindings for other languages,
such as Python and Javascript:

  https://steveire.wordpress.com/2019/04/30/the-future-of-ast-matching

In this first version of this feature, only the accessors for Stmt
classes are generated, not Decls, TypeLocs etc.  Those can be added
after this change is reviewed, as this change is mostly about
infrastructure of these code generators.

Also in this version, the platforms/cmake configurations are excluded as
much as possible so that support can be added iteratively.  Currently a
break on any platform causes a revert of the entire feature.  This way,
the `OR WIN32` can be removed in a future commit and if it breaks the
buildbots, only that commit gets reverted, making the entire process
easier to manage.

Differential Revision: https://reviews.llvm.org/D93164
2021-03-15 10:52:44 +00:00
Stephen Kelly 6e303a982d Revert "[AST] Add generator for source location introspection"
This reverts commit 91abaa1f8d.
2021-03-15 01:16:10 +00:00
Stephen Kelly 370b9b4aea Revert "Attempt to fix ARM buildbot"
This reverts commit 12dac66f6b.
2021-03-15 01:16:07 +00:00
Stephen Kelly 12dac66f6b Attempt to fix ARM buildbot 2021-03-15 00:20:39 +00:00
Stephen Kelly 91abaa1f8d [AST] Add generator for source location introspection
Generate a json file containing descriptions of AST classes and their
public accessors which return SourceLocation or SourceRange.

Use the JSON file to generate a C++ API and implementation for accessing
the source locations and method names for accessing them for a given AST
node.

This new API can be used to implement 'srcloc' output in clang-query:

  http://ce.steveire.com/z/m_kTIo

The JSON file can also be used to generate bindings for other languages,
such as Python and Javascript:

  https://steveire.wordpress.com/2019/04/30/the-future-of-ast-matching

In this first version of this feature, only the accessors for Stmt
classes are generated, not Decls, TypeLocs etc.  Those can be added
after this change is reviewed, as this change is mostly about
infrastructure of these code generators.

Also in this version, the platforms/cmake configurations are excluded as
much as possible so that support can be added iteratively.  Currently a
break on any platform causes a revert of the entire feature.  This way,
the `OR WIN32` can be removed in a future commit and if it breaks the
buildbots, only that commit gets reverted, making the entire process
easier to manage.

Differential Revision: https://reviews.llvm.org/D93164
2021-03-15 00:00:29 +00:00
Stephen Kelly e312b4b6c7 Revert "[AST] Add generator for source location introspection"
This reverts commit 477e4b9746.
2021-03-14 22:51:45 +00:00
Stephen Kelly 9249861437 Revert "Ensure that cache variable is set when not building introspection"
This reverts commit 6b010c6f6e.
2021-03-14 22:51:40 +00:00
Stephen Kelly 6b010c6f6e Ensure that cache variable is set when not building introspection 2021-03-14 22:46:07 +00:00
Stephen Kelly 477e4b9746 [AST] Add generator for source location introspection
Generate a json file containing descriptions of AST classes and their
public accessors which return SourceLocation or SourceRange.

Use the JSON file to generate a C++ API and implementation for accessing
the source locations and method names for accessing them for a given AST
node.

This new API can be used to implement 'srcloc' output in clang-query:

  http://ce.steveire.com/z/m_kTIo

The JSON file can also be used to generate bindings for other languages,
such as Python and Javascript:

  https://steveire.wordpress.com/2019/04/30/the-future-of-ast-matching

In this first version of this feature, only the accessors for Stmt
classes are generated, not Decls, TypeLocs etc.  Those can be added
after this change is reviewed, as this change is mostly about
infrastructure of these code generators.

Also in this version, the platforms/cmake configurations are excluded as
much as possible so that support can be added iteratively.  Currently a
break on any platform causes a revert of the entire feature.  This way,
the `OR WIN32` can be removed in a future commit and if it breaks the
buildbots, only that commit gets reverted, making the entire process
easier to manage.

Differential Revision: https://reviews.llvm.org/D93164
2021-03-14 22:32:42 +00:00
Nikita Popov e0f70a8a97 Revert "[AST] Add generator for source location introspection"
Breaks the build ... somehow: https://llvm-compile-time-tracker.com/show_error.php?commit=77f7d2be214a1de29d583c75739f563593991fc3

FAILED: tools/clang/include/clang/Tooling/NodeIntrospection.inc
cd /root/llvm-compile-time-tracker/llvm-project-build/tools/clang/lib/Tooling && /usr/bin/cmake -E make_directory /root/llvm-compile-time-tracker/llvm-project-build/tools/clang/lib/Tooling/generated/ && /root/llvm-compile-time-tracker/llvm-project/clang/lib/Tooling/DumpTool/generate_cxx_src_locs.py --json-input-path /root/llvm-compile-time-tracker/llvm-project-build/ASTNodeAPI.json --output-file generated/NodeIntrospection.inc --empty-implementation 0 && /usr/bin/cmake -E copy_if_different /root/llvm-compile-time-tracker/llvm-project-build/tools/clang/lib/Tooling/generated/NodeIntrospection.inc /root/llvm-compile-time-tracker/llvm-project-build/tools/clang/include/clang/Tooling/NodeIntrospection.inc
/bin/sh: 1: /root/llvm-compile-time-tracker/llvm-project/clang/lib/Tooling/DumpTool/generate_cxx_src_locs.py: not found

This reverts commit cefe711135.
This reverts commit f72f122fee.
This reverts commit 970c21e345.
This reverts commit 77f7d2be21.
2021-03-14 17:05:08 +01:00
Stephen Kelly cefe711135 Fix license headers 2021-03-14 15:45:53 +00:00
Stephen Kelly f72f122fee Update python script per review comments
Wrap to 80 cols, introduce main() function and use argparse instead of
optparse.
2021-03-14 15:37:30 +00:00
Stephen Kelly 77f7d2be21 [AST] Add generator for source location introspection
Generate a json file containing descriptions of AST classes and their
public accessors which return SourceLocation or SourceRange.

Use the JSON file to generate a C++ API and implementation for accessing
the source locations and method names for accessing them for a given AST
node.

This new API can be used to implement 'srcloc' output in clang-query:

  http://ce.steveire.com/z/m_kTIo

In this first version of this feature, only the accessors for Stmt
classes are generated, not Decls, TypeLocs etc.  Those can be added
after this change is reviewed, as this change is mostly about
infrastructure of these code generators.

Differential Revision: https://reviews.llvm.org/D93164
2021-03-14 10:54:33 +00:00
Stephen Kelly 14050ddc40 Revert "[AST] Add generator for source location introspection"
This reverts commit d627a27d26.

This fails to link on Windows somehow.
2021-03-10 23:36:06 +00:00
Stephen Kelly 774b707564 Revert "Workaround a -Wmisleading-indentation warning"
This reverts commit 5c22e2bec0.
2021-03-10 23:35:41 +00:00
Stephen Kelly 5c22e2bec0 Workaround a -Wmisleading-indentation warning
Because the generated code is not formatted, it can cause warnings.
2021-03-10 23:12:31 +00:00
Stephen Kelly d627a27d26 [AST] Add generator for source location introspection
Generate a json file containing descriptions of AST classes and their
public accessors which return SourceLocation or SourceRange.

Use the JSON file to generate a C++ API and implementation for accessing
the source locations and method names for accessing them for a given AST
node.

This new API can be used to implement 'srcloc' output in clang-query:

  http://ce.steveire.com/z/m_kTIo

In this first version of this feature, only the accessors for Stmt
classes are generated, not Decls, TypeLocs etc.  Those can be added
after this change is reviewed, as this change is mostly about
infrastructure of these code generators.

Differential Revision: https://reviews.llvm.org/D93164
2021-03-10 22:38:39 +00:00
Sam McCall 588db1ccff [clangd] Use flags from open files when opening headers they include
Currently our strategy for getting header compile flags is something like:

A) look for flags for the header in compile_commands.json
   This basically never works, build systems don't generate this info.
B) try to match to an impl file in compile_commands.json and use its flags
   This only (mostly) works if the headers are in the same project.
C) give up and use fallback flags
   This kind of works for stdlib in the default configuration, and
   otherwise doesn't.

Obviously there are big gaps here.

This patch inserts a new attempt between A and B: if the header is
transitively included by any open file (whether same project or not),
then we use its compile command.

This doesn't make any attempt to solve some related problems:
 - parsing non-self-contained header files in context (importing PP state)
 - using the compile flags of non-opened candidate files found in the index

Fixes https://github.com/clangd/clangd/issues/123
Fixes https://github.com/clangd/clangd/issues/695
See https://github.com/clangd/clangd/issues/519

Differential Revision: https://reviews.llvm.org/D97351
2021-03-01 09:43:59 +01:00
Haojian Wu 780ead41e0 [Syntax] No crash on OpaqueValueExpr.
OpaqueValueExpr doesn't correspond to the concrete syntax, it has
invalid source location, ignore them.

Reviewed By: kbobyrev

Differential Revision: https://reviews.llvm.org/D96112
2021-02-18 10:32:04 +01:00
Haojian Wu e159a3ced4 [Syntax] Remove a strict valid source location assertion for TypeLoc.
The EndLoc of a type loc can be invalid for broken code.

Also extend the existing test to support error code with `error-ok`
annotation.

Differential Revision: https://reviews.llvm.org/D96261
2021-02-11 09:53:52 +01:00
Haojian Wu 6c1a23303d [Syntax] Support condition for IfStmt.
Differential Revision: https://reviews.llvm.org/D95782
2021-02-04 09:15:30 +01:00
serge-sans-paille d47ee525f9 [clang-tooling] Prevent llvm::fatal_error on invalid CLI option
Fail gracefully instead. Prevent further misuse by enforcing the factory builder
instead of the constructor.

Differential Revision: https://reviews.llvm.org/D94420
2021-01-29 10:15:06 +01:00
Mikhail Maltsev a0e30914f8 [clang][Tooling] Get rid of a hack in SymbolOccurrences, NFCI
The class `SymbolOccurrences` can store either a single `SourceRange`
in-place or multiple `SourceRanges` on the heap. In the latter case
the number of source ranges is stored in the internal representation
of the beginning `SourceLocation` of the in-place `SourceRange`
object.

This change gets rid of such hack by placing `SourceRange` in a union
which holds either a valid `SourceRange` or an `unsigned int` (a number
of ranges).

The change also adds `static_assert`s that check that `SourceRange` and
`SourceLocation` are trivially destructible (this is required for the
current patch and for D94237 which has already been committed).

Reviewed By: MarkMurrayARM, simon_tatham

Differential Revision: https://reviews.llvm.org/D94599
2021-01-22 13:01:41 +00:00
Yitzhak Mandelbaum 1fabe6e519 [libTooling] Change `addInclude` to use expansion locs.
This patch changes the default range used to anchor the include insertion to use
an expansion loc.  This ensures that the location is valid, when the user relies
on the default range.

Driveby: extend a FIXME for a problem that was emphasized by this change; fix some spellings.

Differential Revision: https://reviews.llvm.org/D93703
2021-01-15 03:08:56 +00:00
Simon Pilgrim 236129fb44 [CompilationDatabase] Pass Twine by const reference instead of by value. NFCI. 2021-01-07 12:53:28 +00:00
Yitzhak Mandelbaum 675a2973ee [libTooling] Add support for smart pointers to relevant Transformer `Stencil`s.
Stencils `maybeDeref` and `maybeAddressOf` are designed to handle nodes that may
be pointers. Currently, they only handle native pointers. This patch extends the
support to recognize smart pointers and handle them as well.

Differential Revision: https://reviews.llvm.org/D93637
2021-01-05 17:57:41 +00:00
Alexander Kornienko 027899dab6 Remove references to the ast_type_traits namespace
Follow up to cd62511496 /
https://reviews.llvm.org/D74499

Reviewed By: aaron.ballman

Differential Revision: https://reviews.llvm.org/D92994
2020-12-11 00:58:46 +01:00
Duncan P. N. Exon Smith 2ea8c69eff Tooling: Migrate to FileEntryRef in clang::tooling::applyAllReplacements, NFC
Migrate to the `FileEntryRef` overload of `SourceManager::createFileID`.
2020-12-09 16:31:38 -08:00
Duncan P. N. Exon Smith 5207f19d10 ADT: Allow IntrusiveRefCntPtr construction from std::unique_ptr, NFC
Allow a `std::unique_ptr` to be moved into the an `IntrusiveRefCntPtr`,
and remove a couple of now-unnecessary `release()` calls.

Differential Revision: https://reviews.llvm.org/D92888
2020-12-08 17:33:19 -08:00
Yitzhak Mandelbaum e6bc4a71e3 [libTooling] Add `describe` combinator for formatting AST nodes for diagnostics.
This new stencil combinator is intended for use in diagnostics and the like.

Differential Revision: https://reviews.llvm.org/D92658
2020-12-07 16:08:05 +00:00
Sam McCall a1cb9cbf5c Add ability to load a FixedCompilationDatabase from a buffer.
Previously, loading one from a file meant allowing the library to do the IO.
Clangd would prefer to do such IO itself (e.g. to allow caching).

Differential Revision: https://reviews.llvm.org/D92640
2020-12-07 13:07:10 +01:00
mydeveloperday 8668eae2ad [clang-format] Add option for case sensitive regexes for sorted includes
I think the title says everything.

Reviewed By: MyDeveloperDay

Patch By:  HazardyKnusperkeks

Differential Revision: https://reviews.llvm.org/D91507
2020-12-05 16:33:21 +00:00
Sam McCall 650e04e179 [Tooling] JSONCompilationDatabase::loadFromBuffer retains the buffer, copy it.
This function doesn't seem to be used in-tree outside tests.
However clangd wants to use it soon, and having the CDB be self-contained seems
reasonable.

Differential Revision: https://reviews.llvm.org/D92646
2020-12-04 21:54:55 +01:00
Yitzhak Mandelbaum fdff677a95 [libTooling] Remove deprecated Clang Transformer declarations
A number of declarations were leftover after the move from `clang::tooling` to
`clang::transformer`. This patch removes those declarations and upgrades the
handful of references to the deprecated declarations.

Differential Revision: https://reviews.llvm.org/D92340
2020-11-30 20:15:26 +00:00
Yitzhak Mandelbaum 88e6208562 [libTooling] Update Transformer's `node` combinator to include the trailing semicolon for decls.
Currently, `node` only includes the semicolon for (some) statements. However,
declarations have the same issue of (potentially) trailing semicolons, so `node`
should behave the same for them.

Differential Revision: https://reviews.llvm.org/D91872
2020-11-20 18:11:50 +00:00
Thorsten Schütt 955341a722 test commit
add whitespace
2020-11-18 09:20:37 +01:00
Nathan James d44edfc109
[clang][NFC] Use SmallString instead of SmallVector<char
Simplifies code in some places and is more explicit about what is being used.
No additional includes were added here so no impact on compile time.
2020-11-17 13:02:58 +00:00
Nathan James 0a6e051a9b
[NFC] Small refactor to CompilationDatabase 2020-11-14 14:03:35 +00:00
Sylvain Audi 79105e4644 [clang-scan-deps] Fix for input file given as relative path in compilation database "command" entry.
Differential Revision: https://reviews.llvm.org/D91204
2020-11-12 08:48:17 -05:00
Sam McCall 1630e50874 [Syntax] Tablegen literal expressions.
Non-mechanical changes:
 - Added FIXME to StringLiteral to cover multi-token string literals.
 - LiteralExpression::getLiteralToken() is gone. (It was never called)
   This is because we don't codegen methods in Alternatives
   It's conceptually suspect if we consider multi-token string literals, though.

Differential Revision: https://reviews.llvm.org/D91277
2020-11-12 01:26:02 +01:00
Sam McCall ea4d24c899 [Syntax] Tablegen Sequence classes. NFC
Similar to the previous patch, this doesn't convert *all* the classes that
could be converted. It also doesn't enforce any new invariants etc.

It *does* include some data we don't use yet: specific token types that are
allowed and optional/required status of sequence items. (Similar to Dmitri's
prototype). I think these are easier to add as we go than later, and serve
a useful documentation purpose.

Differential Revision: https://reviews.llvm.org/D90659
2020-11-11 16:29:19 +01:00
Sam McCall 138189ee33 [Syntax] Tablegen operator<<(NodeKind). NFC
Differential Revision: https://reviews.llvm.org/D90662
2020-11-11 16:02:01 +01:00
Sam McCall 454579e46a Reland [Syntax] Add minimal TableGen for syntax nodes. NFC
This reverts commit 09c6259d6d.
(Fixed side-effecting code being buried in an assert)
2020-11-11 11:24:47 +01:00
Sam McCall 09c6259d6d Revert "[Syntax] Add minimal TableGen for syntax nodes. NFC"
This reverts commit 55120f74ca.

Segfaults during build:
http://lab.llvm.org:8011/#/builders/36/builds/1310
2020-11-09 23:59:11 +01:00
Sam McCall 55120f74ca [Syntax] Add minimal TableGen for syntax nodes. NFC
So far, only used to generate Kind and implement classof().

My plan is to have this general-purpose Nodes.inc in the style of AST
DeclNodes.inc etc, and additionally a special-purpose backend generating
the actual class definitions. But baby steps...

Differential Revision: https://reviews.llvm.org/D90540
2020-11-09 23:45:50 +01:00
Kirill Bobyrev 142c6f82fd
[clang] Simplify buildSyntaxTree API
Follow-up on https://reviews.llvm.org/D88553#inline-837013

Reviewed By: sammccall

Differential Revision: https://reviews.llvm.org/D90672
2020-11-09 22:49:54 +01:00
Duncan P. N. Exon Smith c56ec7bede Tooling: Remove dead code for ToolingInvocation::mapVirtualFile
Follows through on c4cb3b10dc8c50e46c9fb1b7ae95e3c3c94975d3's FIXME
dating back to 2015. Anyone using this should migrate to
InMemoryFileSystem and/or ClangTool::mapVirtualFile.

Differential Revision: https://reviews.llvm.org/D90885
2020-11-09 15:45:35 -05:00
Eduardo Caldas 23657d9cc3 [SyntaxTree] Add reverse links to syntax Nodes.
Rationale:
Children of a syntax tree had forward links only, because there was no
need for reverse links.

This need appeared when we started mutating the syntax tree.
On a forward list, to remove a target node in O(1) we need a pointer to the node before the target. If we don't have this "before" pointer, we have to find it, and that requires O(n).
So in order to remove a syntax node from a tree, we would similarly need to find the node before to then remove. This is both not ergonomic nor does it have a good complexity.

Differential Revision: https://reviews.llvm.org/D90240
2020-11-05 09:33:53 +00:00
Sam McCall dd6f7ee05e [Syntax] DeclaratorList is a List
I think this was just an oversight.

Differential Revision: https://reviews.llvm.org/D90541
2020-11-03 03:29:06 +01:00