Commit Graph

632 Commits

Author SHA1 Message Date
Dmitri Gribenko 73f423e739 Parenthesized a logical expression.
This change fixes a warning on the ppc64be buildbot.
2019-12-20 14:36:21 +01:00
Dmitri Gribenko 9caf12e849 Use isa instead of dyn_cast 2019-12-20 14:36:21 +01:00
Kadir Cetinkaya c2377eae28
[clang][Tooling] Prefer -x over -std when interpolating
Summary:
Currently interpolation logic prefers -std over -x. But the latter is a
more strong signal, so this patch inverts the order and only makes use of -std
if -x didn't exist.

Fixes https://github.com/clangd/clangd/issues/185

Thanks @sammccall for tracking this down!

Reviewers: sammccall

Subscribers: ilya-biryukov, usaxena95, cfe-commits, sammccall

Tags: #clang

Differential Revision: https://reviews.llvm.org/D71727
2019-12-20 11:10:36 +01:00
Ilya Biryukov c1bbefef9d [Syntax] Use a hash table to search for tokens by their location
This is both more efficient and avoids corner cases in
`SourceManager::isBeforeInTranslationUnit`.

The change is trivial and clearly a performance improvement on the hot
path of building the syntax tree, so sending without review.
2019-12-18 12:24:00 +01:00
Ilya Biryukov 1ad15046dc [Syntax] Allow to mutate syntax trees
Summary:
This patch adds facilities to mutate the syntax trees and produce
corresponding text replacements.

The public interface of the syntax library now includes facilities to:
    1. perform type-safe modifications of syntax trees,
    2. compute textual replacements to apply the modifications,
    3. create syntax trees not backed by the source code.

For each of the three, we only add a few example transformations in this
patch to illustrate the idea, support for more kinds of nodes and
transformations will be done in follow-up patches.

The high-level mutation operations are implemented on top of operations
that allow to arbitrarily change the trees. They are considered to be
implementation details and are not available to the users of the
library.

Reviewers: sammccall, gribozavr2

Reviewed By: gribozavr2

Subscribers: merge_guards_bot, mgorny, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D64573
2019-12-18 12:19:03 +01:00
Sam McCall 22f8125088 [Tooling/Syntax] Helpers to find spelled tokens touching a location.
Summary: Useful when positions are used to target nodes, with before/after ambiguity.

Reviewers: ilya-biryukov, kbobyrev

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D71356
2019-12-13 16:57:03 +01:00
Sam McCall 3f8da5d091 [Tooling/Syntax] Helpers to find spelled tokens touching a location.
Summary: Useful when positions are used to target nodes, with before/after ambiguity.

Reviewers: ilya-biryukov, kbobyrev

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D71356
2019-12-12 12:59:50 +01:00
Ilya Biryukov be14a22b47 [Syntax] Build nodes for simple cases of top level declarations
Summary:
More complicated nodes (e.g. template declarations) will be implemented
in the follow-up patches.

Reviewers: gribozavr2

Reviewed By: gribozavr2

Subscribers: merge_guards_bot, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D70856
2019-12-12 08:04:22 +01:00
Michael Spencer 5bcd34a03f Revert "[clang][clang-scan-deps] Aggregate the full dependency information."
This reverts commit f978ea4983.

It broke clang-ppc64be-linux, but not sure why yet.
2019-12-11 16:35:55 -08:00
Michael Spencer f978ea4983 [clang][clang-scan-deps] Aggregate the full dependency information.
Differential Revision: https://reviews.llvm.org/D70268
2019-12-11 14:40:51 -08:00
Sam McCall 966fac1941 [clang][Tooling] Fix potential UB in ExpandResponseFilesCompilationDatabase
Summary:
`vector::assign` will cause UB at here.

fixes: https://github.com/clangd/clangd/issues/223

Reviewers: kadircet, sammccall, hokein

Reviewed By: sammccall

Subscribers: merge_guards_bot, ilya-biryukov, usaxena95, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D71172
2019-12-09 12:24:23 +01:00
Reid Kleckner 60573ae6fe Remove Expr.h include from ASTContext.h, NFC
ASTContext.h is popular, prune its includes. Expr.h brings in Attr.h,
which is also expensive.

Move BlockVarCopyInit to Expr.h to accomplish this.
2019-12-06 15:30:49 -08:00
Kadir Cetinkaya b3b3778303
Reapply "[llvm][Support] Take in CurrentDirectory as a parameter in ExpandResponseFiles"
Attemps to fix windows buildbots.
2019-12-04 17:00:47 +01:00
Kadir Cetinkaya 0f12f9096e
Revert "[llvm][Support] Take in CurrentDirectory as a parameter in ExpandResponseFiles"
This reverts commit 75656005db.
2019-12-04 15:58:01 +01:00
Kadir Cetinkaya 75656005db
[llvm][Support] Take in CurrentDirectory as a parameter in ExpandResponseFiles
Summary:
This is a follow-up to D70769 and D70222, which allows propagation of
current directory down to ExpandResponseFiles for handling of relative paths.

Previously clients had to mutate FS to achieve that, which is not thread-safe
and can even be thread-hostile in the case of real file system.

Reviewers: sammccall

Subscribers: hiraditya, cfe-commits, llvm-commits

Tags: #clang, #llvm

Differential Revision: https://reviews.llvm.org/D70857
2019-12-04 15:13:12 +01:00
Kadir Cetinkaya 45ef055d4f
[clang][Tooling] Add support for .rsp files in compile_commands.json
Summary:
Add support for .rsp files.

Fixes https://github.com/clangd/clangd/issues/81

Patch By: liu hui(@lh123)

Reviewers: sammccall, ilya-biryukov, hokein, kadircet

Reviewed By: kadircet

Subscribers: merge_guards_bot, mgorny, MaskRay, jkorous, arphaman, kadircet, usaxena95, cfe-commits

Tags: #clang-tools-extra, #clang

Differential Revision: https://reviews.llvm.org/D70222
2019-12-04 15:13:12 +01:00
Sam McCall c9c714c705 Reland [clangd] Rethink how SelectionTree deals with macros and #includes.
This reverts commit 905b002c13.

Avoid tricky (and invalid) comparator for std::set.
2019-12-03 17:53:43 +01:00
Sam McCall 905b002c13 Revert "[clangd] Rethink how SelectionTree deals with macros and #includes."
This reverts commit 19daa21f84.

It causes a bunch of failures on a bot that I've been unable to
reproduce so far:
http://45.33.8.238/mac/3308/step_7.txt
2019-11-29 19:59:02 +01:00
Sam McCall 19daa21f84 [clangd] Rethink how SelectionTree deals with macros and #includes.
Summary:
The exclusive-claim model is successful at resolving conflicts over tokens
between parent/child or siblings. However claims at the spelled-token
level do the wrong thing for macro expansions, where siblings can be
equally associated with the macro invocation.
Moreover, any model that only uses the endpoints in a range can fail when
a macro invocation occurs inside the node.

To address this, we use the existing TokenBuffer in more depth.
Claims are expressed in terms of expanded tokens, so there is no need to worry
about macros, includes etc.

Once we know which expanded tokens were claimed, they are mapped onto
spelled tokens for hit-testing.
This mapping is fairly flexible, currently the handling of macros is
pretty simple (map macro args onto spellings, other macro expansions onto the
macro name token).
This mapping is in principle token-by-token for correctness (though
there's some batching for performance).

The aggregation of the selection enum is now more principled as we need to be
able to aggregate several hit-test results together.

For simplicity i removed the ability to determine selectedness of TUDecl.
(That was originally implemented in 90a5bf92ff97b1, but doesn't seem to be very
important or worth the complexity any longer).

The expandedTokens(SourceLocation) helper could be added locally, but seems to
make sense on TokenBuffer.

Fixes https://github.com/clangd/clangd/issues/202
Fixes https://github.com/clangd/clangd/issues/126

Reviewers: hokein

Subscribers: MaskRay, jkorous, arphaman, kadircet, usaxena95, cfe-commits, ilya-biryukov

Tags: #clang

Differential Revision: https://reviews.llvm.org/D70512
2019-11-29 15:21:13 +01:00
Ilya Biryukov e702bdb859 [Syntax] Build SimpleDeclaration node that groups multiple declarators
Summary:
Also remove the temporary TopLevelDeclaration node and add
UnknownDeclaration to represent other unknown nodes.

See the follow-up change for building more top-level declarations.
Adding declarators is also pretty involved and will be done in another
follow-up patch.

Reviewers: gribozavr2

Reviewed By: gribozavr2

Subscribers: merge_guards_bot, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D70787
2019-11-29 11:39:45 +01:00
Ilya Biryukov 302cb3bc3d [Syntax] Add a comment explaining the pointer keys in std::map<Token*, ...>. NFC 2019-11-29 09:49:24 +01:00
Ilya Biryukov def65bb4f5 [Syntax] Remove unused parameter from `TreeBuilder::markChildToken`. NFC 2019-11-29 09:45:06 +01:00
Duncan P. N. Exon Smith 20d51b2f14 clang/Modules: Rename CompilerInstance::ModuleManager, NFC
Fix the confusing naming of `CompilerInstance::ModuleManager`.  This is
actually an instance of `ASTReader`, which contains an instance of
`ModuleManager`.  I have to assume there was a point in the past where
they were just one class, but it's been pretty confusing for a while.  I
think it's time to fix it.

The new name is `TheASTReader`; the annoying `The` prefix is so that we
don't shadow the `ASTReader` class.  I tried out `ASTRdr` but that
seemed less clear, and this choice matches `ThePCHContainerOperations`
just a couple of declarations below.

Also rename `CompilerInstance::getModuleManager` and
`CompilerInstance::createModuleManager` to `*ASTReader`, making some
cases of `getModuleManager().getModuleManager()` a little more clear.

https://reviews.llvm.org/D70583
2019-11-22 18:24:11 -08:00
Yitzhak Mandelbaum 01e8dd2e7a [libTooling] Add stencil combinators for nodes that may be pointers or values.
Summary:
Adds combinators `maybeDeref` and `maybeAddressOf` to provide a uniform way to handle
nodes which may be bound to either a pointer or a value (most often in the
context of member expressions). Such polymorphism is already supported by
`access`; these combinators extend it to more general uses.

Reviewers: gribozavr

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D70554
2019-11-22 12:36:40 -05:00
Yitzhak Mandelbaum dd471dbe99 [libTooling] Extend `buildASTFromCodeWithArgs` to take files argument.
Summary:
Adds an optional parameter to `buildASTFromCodeWithArgs` that allows the user to
pass additional files that the main code needs to compile. This change makes
`buildASTFromCodeWithArgs` consistent with `runToolOnCodeWithArgs`.

Patch by Alexey Eremin.

Reviewers: gribozavr

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D70175
2019-11-19 12:18:36 -05:00
Kadir Cetinkaya 16bdcc809c
[clang][Tooling] Filter flags that generate output in SyntaxOnlyAdjuster
Summary:
Flags that generate output could result in failures when creating
syntax only actions. This patch introduces initial logic for filtering out
those. The first such flag is "save-temps", which saves intermediate
files(bitcode, assembly, etc.) into a specified directory.

Fixes https://github.com/clangd/clangd/issues/191

Reviewers: hokein

Subscribers: ilya-biryukov, usaxena95, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D70173
2019-11-13 15:03:30 +01:00
mydeveloperday 335ac2eb66 Allow additional file suffixes/extensions considered as source in main include grouping
Summary:
By additional regex match, grouping of main include can be enabled in files that are not normally considered as a C/C++ source code.
For example, this might be useful in templated code, where template implementations are being held in *Impl.hpp files.
On the occassion, 'assume-filename' option description was reworded as it was misleading. It has nothing to do with `style=file` option and it does not influence sourced style filename.

Reviewers: rsmith, ioeric, krasimir, sylvestre.ledru, MyDeveloperDay

Reviewed By: MyDeveloperDay

Subscribers: MyDeveloperDay, cfe-commits

Patch by:  furdyna

Tags: #clang

Differential Revision: https://reviews.llvm.org/D67750
2019-11-12 21:26:52 +00:00
Yitzhak Mandelbaum 489449c28a [libTooling] Further simplify `Stencil` type and introduce `MatchComputation`.
Summary:
This revision introduces a new interface `MatchComputation` which generalizes
the `Stencil` interface and replaces the `std::function` interface of
`MatchConsumer`. With this revision, `Stencil` (as an abstraction) becomes just
one collection of implementations of
`MatchComputation<std::string>`. Correspondingly, we remove the `Stencil` class
entirely in favor of a simple type alias, deprecate `MatchConsumer` and change
all functions that accepted `MatchConsumer<std::string>` to use
`MatchComputation<std::string>` instead.

Reviewers: gribozavr

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D69802
2019-11-11 12:44:15 -05:00
Haojian Wu 118f7836a6 [clang-rename] Respect the traversal scope when traversing the entire AST.
Summary:
This should be NFC to clang-rename, by default the traversal scope is
TUDecl. Traversing the TUDecl in clangd is a performance cliff, we should
avoid it.

Reviewers: ilya-biryukov

Subscribers: kadircet, usaxena95, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D69892
2019-11-07 10:43:54 +01:00
Ilya Biryukov 96065cf79f [Syntax] Silence "unused function" warning in no-assert builds. NFC
A helper `isImpicitExpr` is only used inside assert.
2019-11-07 09:37:25 +01:00
Yitzhak Mandelbaum 6c683aa8d7 [libTooling] Fix breakage from change #84922 2019-11-06 11:31:35 -05:00
Yitzhak Mandelbaum ce2b5cb6de [libTooling] Simplify type structure of `Stencil`s.
Summary:
Currently, stencils are defined as a sequence of `StencilParts`. This
differentiation adds an unneeded layer of complexity to the definition of
Stencils. This change significantly simplifies the type structure: a stencil is
now conceptually any object implementing `StencilInterface` and `Stencil` is
just a thin wrapper for pointers to this interface.

To account for the sequencing that was supported by the old `Stencil` type, we
introduce a sequencing class that implements `StencilInterface`. That is,
sequences are just another kind of Stencil and no longer have any special
status.

Corresponding to this change in the type structure, we change the way `cat` is
used (and defined). `cat` bundles multiple features: it builds a stencil from a
sequence of subcomponents and admits multiple different types for its arguments,
while coercing them into the right type. Previously, `cat` was also used to
coerce a single `StencilPart` into a `Stencil`. With that distinction gone, many
uses of `cat` (e.g. in the tests) are unnecessary and have, therefore, been
removed.

Reviewers: gribozavr

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D69613
2019-11-06 10:28:34 -05:00
Ilya Biryukov 58fa50f437 [Syntax] Add nodes for most common statements
Summary:
Most of the statements mirror the ones provided by clang AST.
Major differences are:
  - expressions are wrapped into 'ExpressionStatement' instead of being
    a subclass of statement,
  - semicolons are always consumed by the leaf expressions (return,
    expression satement, etc),
  - some clang statements are not handled yet, we wrap those into an
    UnknownStatement class, which is not present in clang.

We also define an 'Expression' and 'UnknownExpression' classes in order
to produce 'ExpressionStatement' where needed. The actual implementation
of expressions is not yet ready, it will follow later.

Reviewers: sammccall

Reviewed By: sammccall

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D63835
2019-11-06 10:56:06 +01:00
Michael Spencer d816d9bdc5 [clang][ScanDeps] Fix issue with multiple commands with the same input.
Previously, given a CompilationDatabase with two commands for the same
source file we would report that file twice with the union of the
dependencies for each command both times.

This was due to the way `ClangTool` runs actions given an input source
file (see the comment in `DependencyScanningTool.cpp`). This commit adds
a `SingleCommandCompilationDatabase` that is created with each
`CompileCommand` in the original CDB, which is then used for each
`ClangTool` invocation. This gives us a single run of
`DependencyScanningAction` per `CompileCommand`.

I looked at using `AllTUsToolExecutor` which is a parallel tool
executor, but I'm not sure it's suitable for `clang-scan-deps` as it
does a lot more sharing of state than `AllTUsToolExecutor` expects.

Differential Revision: https://reviews.llvm.org/D69643
2019-10-31 14:22:01 -07:00
Michael Spencer 17b64e36e9 [clang][ScanDeps] Fix shadowing warning. 2019-10-30 15:53:38 -07:00
Michael Spencer 33a745e6fe [clang][clang-scan-deps] Add support for extracting full module dependencies.
This is a recommit of d8a4ef0e68 with the nondeterminism fixed.

This adds experimental support for extracting a Clang module dependency graph
from a compilation database. The output format is experimental and will change.
It is currently a concatenation of JSON outputs for each compilation. Future
patches will change this to deduplicate modules between compilations.

Differential Revision: https://reviews.llvm.org/D69420
2019-10-30 15:27:27 -07:00
Michael Spencer 1c88d66223 Revert "[clang][clang-scan-deps] Add support for extracting full module dependencies."
This reverts commit d8a4ef0e68.

This commit broke some of the bots. I believe it's due to nondeterminism. Will fix and recommit.
2019-10-30 15:16:38 -07:00
Michael Spencer d8a4ef0e68 [clang][clang-scan-deps] Add support for extracting full module dependencies.
This adds experimental support for extracting a Clang module dependency graph
from a compilation database. The output format is experimental and will change.
It is currently a concatenation of JSON outputs for each compilation. Future
patches will change this to deduplicate modules between compilations.

Differential Revision: https://reviews.llvm.org/D69420
2019-10-30 14:46:17 -07:00
Michael Spencer 8da20560ab [clang][DependencyScanning] 80-col. 2019-10-25 15:43:57 -07:00
Haojian Wu 3d9632a997 [clang-rename] NFC, make getCanonicalSymbolDeclaration robust on nullptr input. 2019-10-25 14:33:04 +02:00
Michael Spencer 7af309a4ed [clang][DependencyScanning] clang-format. 2019-10-24 18:28:01 -07:00
Michael Spencer 9ab6d8236b [clang-scan-deps] Add basic support for modules.
This fixes two issues that prevent simple uses of modules from working.

* We would previously minimize _every_ file opened by clang, even module maps
  and module pcm files. Now we only minimize files with known extensions. It
  would be better if we knew which files clang intended to open as a source
  file, but this works for now.

* We previously cached every lookup, even failed lookups. This is a problem
  because clang stats the module cache directory before building a module and
  creating that directory. If we cache that failure then the subsequent pcm
  load doesn't see the module cache and fails.

Overall this still leaves us building minmized modules on disk during scanning.
This will need to be improved eventually for performance, but this is correct,
and works for now.

Differential Revision: https://reviews.llvm.org/D68835
2019-10-24 16:19:11 -07:00
Kousik Kumar fb042b094f Refactor DependencyScanningTool to its own file
Summary:
There's no behavior change - just moving DependencyScanningTool to its own file
since this tool can be reused across both clang-scan-deps binary and an interface
exposed as part of libClang APIs.

Reviewers: arphaman, jkorous, Bigcheese, dexonsmith

Subscribers: mgorny, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D69186

llvm-svn: 375483
2019-10-22 05:05:18 +00:00
Bjorn Pettersson 1f43ea41c3 Prune Pass.h include from DataLayout.h. NFCI
Summary:
Reduce include dependencies by no longer including Pass.h from
DataLayout.h. That include seemed irrelevant to DataLayout, as
well as being irrelevant to several users of DataLayout.

Reviewers: rnk

Reviewed By: rnk

Subscribers: mehdi_amini, hiraditya, cfe-commits, llvm-commits

Tags: #clang, #llvm

Differential Revision: https://reviews.llvm.org/D69261

llvm-svn: 375436
2019-10-21 17:51:54 +00:00
Michael Liao 7e5d5ee873 [tooling] Relax an assert when multiple GPU targets are specified.
llvm-svn: 375245
2019-10-18 15:03:34 +00:00
Kousik Kumar 9e7e36d4c2 Revert "Include sanitize blacklist and other extra deps as part of scan-deps output"
This test is failing on Windows bots, revert for now (will check the right fix and retry the patch).

Summary: This reverts commit 962ca076e5.

Reviewers: Bigcheese, jkorous, arphaman

Subscribers: dexonsmith, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D69079

llvm-svn: 375079
2019-10-17 04:50:12 +00:00
Kousik Kumar 962ca076e5 Include sanitize blacklist and other extra deps as part of scan-deps output
Summary:
Clang's -M mode includes these extra dependencies in its output and clang-scan-deps
should have equivalent behavior, so adding these extradeps to output just like
how its being done for ".d" file generation mode.

Reviewers: arphaman, dexonsmith, Bigcheese, jkorous

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D69017

llvm-svn: 375074
2019-10-17 02:14:44 +00:00
Yitzhak Mandelbaum 8bb47cd8c3 [libTooling] Put all Transformer declarations in a single namespace.
Summary:
This revision introduces a new namespace, `clang::transformer`, to hold
the declarations for the Transformer library.

Reviewers: gribozavr

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D68876

llvm-svn: 374962
2019-10-16 01:06:46 +00:00
Yitzhak Mandelbaum e38c36b7b0 [libTooling] Move `RewriteRule` abstraction into its own header and impl.
Summary: Move the `RewriteRule` class and related declarations into its own set
of files (header, implementation). Only the `Transformer` class is left in the
Transformer-named files. This change clarifies the distinction between the
`RewriteRule` class, which is essential to the Transformer library, and the
`Transformer` class, which is only one possible `RewriteRule` interpreter
(compare to `TransformerClangTidyCheck`, a clang-tidy based interpreter).

Reviewers: gribozavr

Subscribers: jfb, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D68795

llvm-svn: 374558
2019-10-11 14:43:46 +00:00
Yitzhak Mandelbaum cf2438ec13 [libTooling] Change Stencil equality to use `toString()`
Summary:
Removes the `isEqual` method from StencilPartInterface and modifies equality to
use the string representation returned by the `toString` method for comparison.

This means the `run` and `selection` stencils return true by default, and
clients should be cautious in relying on equality operator for comparison of
stencils containing parts generated by these functions.

It also means we no longer need the custom RTTI support (typeId() and
down_cast()), so it has been removed.

Patch by Harshal T. Lehri.

Reviewers: gribozavr

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D68825

llvm-svn: 374552
2019-10-11 14:02:03 +00:00
Michael Liao 1f62ae8635 [tooling] Fix assertion on MacOSX.
llvm-svn: 374478
2019-10-10 23:45:20 +00:00
Michael Liao 1ccb39bbd8 [tooling] Teach Tooling to understand compilation with offloading.
Summary:
- So far, we only recognize the host compilation with offloading and
  skip the offloading part.

Reviewers: tra, yaxunl

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D68660

llvm-svn: 374470
2019-10-10 23:05:55 +00:00
Michael J. Spencer 2f56266234 [ScanDeps] clang-format, 80 cols.
llvm-svn: 374439
2019-10-10 20:19:02 +00:00
Kousik Kumar 4abac53302 In openFileForRead don't cache erroneous entries if the error relates to them being directories. Add tests.
Summary:
It seems that when the CachingFileSystem is first given a file to open that is actually a directory, it incorrectly
caches that path to be errenous and throws an error when subsequently a directory open call is made for the same
path.
This change makes it so that we do NOT cache a path if it turns out we asked for a file when its a directory.

Reviewers: arphaman

Subscribers: dexonsmith, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D68193

llvm-svn: 374366
2019-10-10 15:29:01 +00:00
Yitzhak Mandelbaum fbdf83521b [libTooling] Move Transformer files to their own directory/library.
Summary:
The Transformer library has been growing inside of
lib/Tooling/Refactoring. However, it's not really related to anything else in
that directory. This revision moves all Transformer-related files into their own
include & lib directories.  A followup revision will (temporarily) add
forwarding headers to help any users migrate their code to the new location.

Reviewers: gribozavr

Subscribers: mgorny, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D68637

llvm-svn: 374271
2019-10-10 02:34:47 +00:00
Yitzhak Mandelbaum d5b983555f [libTooling] Add `toString` method to the Stencil class
Summary:
`toString` generates a string representation of the stencil.

Patch by Harshal T. Lehri.

Reviewers: gribozavr

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D68574

llvm-svn: 373916
2019-10-07 16:20:22 +00:00
Haojian Wu 5d862c042b [clang-rename] Fix a crash when renaming a class without definition.
Reviewers: sammccall

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D68459

llvm-svn: 373748
2019-10-04 14:09:31 +00:00
Yitzhak Mandelbaum e80e889695 [libTooling] Add various Stencil combinators for expressions.
Summary:
This revision adds three new Stencil combinators:
* `expression`, which idiomatically constructs the source for an expression,
  including wrapping the expression's source in parentheses if needed.
* `deref`, which constructs an idiomatic dereferencing expression.
* `addressOf`, which constructs an idiomatic address-taking expression.

Reviewers: gribozavr

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D68315

llvm-svn: 373593
2019-10-03 13:01:00 +00:00
Benjamin Kramer 340406a740 [Stencil] Hide implementaion detai. NFC.
llvm-svn: 373504
2019-10-02 18:33:44 +00:00
Yitzhak Mandelbaum db24ef509e [libTooling] Transformer: refine `SourceLocation` specified as anchor of changes.
Summary: Every change triggered by a rewrite rule is anchored at a particular
location in the source code.  This patch refines how that location is chosen and
defines it as an explicit function so it can be shared by other Transformer
implementations.

This patch was inspired by a bug found by a clang tidy, wherein two changes were
anchored at the same location (the expansion loc of the macro) resulting in the
discarding of the second change.

Reviewers: gribozavr

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D66652

llvm-svn: 373093
2019-09-27 15:26:04 +00:00
Yitzhak Mandelbaum ea966c1bc0 [libTooling] Add `run` combinator to Stencils.
Summary:
This revision adds `run`, a StencilPart that runs a user-defined function that
computes a result over `MatchFinder::MatchResult`.

Reviewers: gribozavr

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D67969

llvm-svn: 372936
2019-09-26 00:53:56 +00:00
Paul Hoad 52e44b1423 [clang-format] Modified SortIncludes and IncludeCategories to priority for sorting #includes within the Group Category.
Summary:
This new Style rule is made as a part of adding support for NetBSD KNF in clang-format. NetBSD have it's own priority of includes which should be followed while formatting NetBSD code. This style sorts the Cpp Includes according to the priorities of NetBSD, as mentioned in the [Style Guide](http://cvsweb.netbsd.org/bsdweb.cgi/src/share/misc/style?rev=HEAD&content-type=text/x-cvsweb-markup)
 The working of this Style rule shown below:

**Configuration:**
This revision introduces a new field under IncludeCategories named `SortPriority` which defines the priority of ordering the `#includes` and the `Priority` will define the categories for grouping the `#include blocks`.

Reviewers: cfe-commits, mgorny, christos, MyDeveloperDay

Reviewed By: MyDeveloperDay

Subscribers: lebedev.ri, rdwampler, christos, mgorny, krytarowski

Patch By: Manikishan

Tags: #clang, #clang-format

Differential Revision: https://reviews.llvm.org/D64695

llvm-svn: 372919
2019-09-25 20:33:01 +00:00
Yitzhak Mandelbaum 148fd9f9c1 [libTooling] Introduce the MatchConsumer abstraction
Summary:
This revision introduces a separate (small) library for the `MatchConsumer`
abstraction: computations over AST match results.  This abstraction is central
to the Transformer framework, and there deserves being defined explicitly.

Reviewers: gribozavr

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D67961

llvm-svn: 372870
2019-09-25 13:34:04 +00:00
Yitzhak Mandelbaum 0a81b4ed97 [libTooling] Add `access` and `ifBound` combinators to Stencil library.
Summary:
This revision add the `access` and `ifBound` combinators to the Stencil library:
* `access` -- constructs an idiomatic expression for accessing a member (a
  `MemberExpr`).
* `ifBound` -- chooses between two `StencilParts` based on the whether an id is
  bound in the match (corresponds to the combinator of the same name in
  RangeSelector).

Reviewers: gribozavr

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D67633

llvm-svn: 372605
2019-09-23 13:21:42 +00:00
Yitzhak Mandelbaum 1588c08735 [libTooling] Introduce new library of source-code builders.
Summary:
Introduces facilities for easily building source-code strings, including
idiomatic use of parentheses and the address-of, dereference and member-access
operators (dot and arrow) and queries about need for parentheses.

Reviewers: gribozavr

Subscribers: mgorny, cfe-commits, ilya-biryukov

Tags: #clang

Differential Revision: https://reviews.llvm.org/D67632

llvm-svn: 372595
2019-09-23 12:40:10 +00:00
Alex Lorenz 172e8a7a5d [clang-scan-deps] strip the --serialize-diagnostics argument
This ensures that clang-scan-deps won't write out diagnostics when
scanning dependencies.

llvm-svn: 372444
2019-09-21 00:17:26 +00:00
Yitzhak Mandelbaum eff88e42f7 [libTooling] Add `ifBound`, `elseBranch` RangeSelector combinators.
Summary:
Adds two new combinators and corresponding tests to the RangeSelector library.
* `ifBound` -- conditional evaluation of range-selectors, based on whether a
   given node id is bound in the match.
* `elseBranch` -- selects the source range of the else and its statement.

Reviewers: gribozavr

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D67621

llvm-svn: 372410
2019-09-20 17:11:03 +00:00
Alex Lorenz ee30b0ecc2 [clang-scan-deps] Fix for headers having the same name as a directory
Scan deps tool crashes when called on a C++ file, containing an include
that has the same name as a directory.
The tool crashes since it finds foo/dir and tries to read that as a file and fails.

Patch by: kousikk (Kousik Kumar)

Differential Revision: https://reviews.llvm.org/D67091

llvm-svn: 371903
2019-09-13 22:12:02 +00:00
Alex Lorenz 57a014d31f [clang-scan-deps] remove dots and dots dots from the reported file dependency paths
This resolves differences observed on LLVM + Clang when running the comparison between canonical
dependencies (full preprocessing, no file manager reused), and dependencies obtained
when the file manager was reused between the full preprocessing invocations.

llvm-svn: 371751
2019-09-12 18:03:24 +00:00
Alex Lorenz 428d92832c [clang-scan-deps] cast Result to ErrorOr<unique_ptr<vfs::File>> explicitly to avoid s390x-linux buildbot failure
llvm-svn: 371664
2019-09-11 21:00:13 +00:00
Alex Lorenz ca6e60971e [clang-scan-deps] add skip excluded conditional preprocessor block preprocessing optimization
This commit adds an optimization to clang-scan-deps and clang's preprocessor that skips excluded preprocessor
blocks by bumping the lexer pointer, and not lexing the tokens until reaching appropriate #else/#endif directive.
The skip positions and lexer offsets are computed when the file is minimized, directly from the minimized tokens.

On an 18-core iMacPro with macOS Catalina Beta I got 10-15% speed-up from this optimization when running clang-scan-deps on
the compilation database for a recent LLVM and Clang (3511 files).

Differential Revision: https://reviews.llvm.org/D67127

llvm-svn: 371656
2019-09-11 20:40:31 +00:00
Ilya Biryukov 433927595d [Driver] Use shared singleton instance of DriverOptTable
Summary:
This significantly reduces the time required to run clangd tests, by
~10%.

Should also have an effect on other tests that run command-line parsing
multiple times inside a single invocation.

Reviewers: gribozavr, sammccall

Reviewed By: sammccall

Subscribers: kadircet, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D67163

llvm-svn: 370908
2019-09-04 14:26:28 +00:00
Alex Lorenz c4130cf132 [clang-scan-deps] NFC, remove outdated implementation comment
There's no need to purge symlinked entries in the FileManager,
as the new FileEntryRef API allows us to compute dependencies more
accurately when the FileManager is reused.

llvm-svn: 370493
2019-08-30 17:34:22 +00:00
Dmitri Gribenko b22804b354 [Tooling] Migrated APIs that take ownership of objects to unique_ptr
Subscribers: jkorous, arphaman, kadircet, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D66960

llvm-svn: 370451
2019-08-30 09:29:34 +00:00
Alex Lorenz ca4216abde [clang-scan-deps] NFC, refactor the DependencyScanningWorker to use a consumer
to report the dependencies to the client

This will allow the scanner to report modular dependencies to the consumer.
This will also allow the scanner to accept regular cc1 clang invocations, e.g.
in an implementation of a libclang C API for clang-scan-deps, that I will add
follow-up patches for in the future.

llvm-svn: 370425
2019-08-30 01:25:57 +00:00
Alex Lorenz 3944c9638e [clang-scan-deps] reuse the file manager across invocations of
the dependency scanner on a single worker thread

This behavior can be controlled using the new `-reuse-filemanager` clang-scan-deps
option. By default the file manager is reused.

The added test/ClangScanDeps/symlink.cpp is able to pass with
the reused filemanager after the related FileEntryRef changes
landed earlier. The test test/ClangScanDeps/subframework_header_dir_symlink.m
still fails when the file manager is reused (I run the FileCheck with not to
make it PASS). I will address this in a follow-up patch that improves
the DirectoryEntry name modelling in the FileManager.

llvm-svn: 370420
2019-08-29 22:56:38 +00:00
Dmitri Gribenko 907452107d Changed FrontendActionFactory::create to return a std::unique_ptr
Subscribers: jkorous, arphaman, kadircet, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D66947

llvm-svn: 370379
2019-08-29 16:38:36 +00:00
Dmitri Gribenko 1fac68b0dc ArrayRef'ized CompilerInvocation::CreateFromArgs
Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D66797

llvm-svn: 370122
2019-08-27 22:13:31 +00:00
Johan Vikstrom 6687fde07a [Syntax] Added function to get macro expansion tokens to TokenBuffer.
Summary:
Returns the first token in every mapping where the token is an identifier.
This API is required to be able to highlight macro expansions in clangd.

Reviewers: hokein, ilya-biryukov

Subscribers: kadircet, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D66470

llvm-svn: 369385
2019-08-20 13:34:01 +00:00
David L. Jones 4ed5521cad [Tooling] Add a hack to work around issues with matcher binding in r368681.
The change in r368681 contains a (probably unintentional) behavioral change for
rewrite rules with a single matcher. Previously, the single matcher would not
need to be bound (`joinCaseMatchers` returned it directly), even though a final
DynTypeMatcher was created and bound by `buildMatcher`. With the new change, a
single matcher will be bound, in addition to the final binding (which is now in
`buildMatchers`, but happens roughly at the same point in the overall flow).

This patch simply duplicates the "final matcher" trick: it creates an extra
DynTypedMatcher for each rewrite rule case matcher, and unconditionally makes it
bindable. This is probably not the right long-term fix, but it does allow
existing code to continue to work with this interface.

Subscribers: cfe-commits, gribozavr, ymandel

Tags: #clang

Differential Revision: https://reviews.llvm.org/D66273

llvm-svn: 368958
2019-08-15 04:10:11 +00:00
Jonas Devlieghere 2b3d49b610 [Clang] Migrate llvm::make_unique to std::make_unique
Now that we've moved to C++14, we no longer need the llvm::make_unique
implementation from STLExtras.h. This patch is a mechanical replacement
of (hopefully) all the llvm::make_unique instances across the monorepo.

Differential revision: https://reviews.llvm.org/D66259

llvm-svn: 368942
2019-08-14 23:04:18 +00:00
Yitzhak Mandelbaum 385a451844 [libTooling] Fix code to avoid unused-function warning after r368681.
llvm-svn: 368862
2019-08-14 15:20:06 +00:00
Shaurya Gupta 36ca1e6399 [Tooling] Added DeclStmtClass to ExtractionSemicolonPolicy
Since the DeclStmt range includes the semicolon, it doesn't need a
semicolon at the end during extraction

llvm-svn: 368850
2019-08-14 13:37:39 +00:00
Shaurya Gupta 52d0cfc91e [Refactor] Moving SourceExtraction header from lib to include
Summary:
- Moved the SourceExtraction header from lib to include so that it can be used in clangd.

Reviewers: arphaman

Subscribers: ilya-biryukov, dexonsmith, kadircet, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D65878

llvm-svn: 368743
2019-08-13 20:21:00 +00:00
Yitzhak Mandelbaum 42b957aad7 [libTooling] In Transformer, generalize `applyFirst` to admit rules with incompatible matchers.
Summary:
This patch removes an (artificial) limitation of `applyFirst`, which requires
that all of the rules' matchers can be grouped together in a single `anyOf()`.
This change generalizes the code to group the matchers into separate `anyOf`s
based on compatibility. Correspondingly, `buildMatcher` is changed to
`buildMatchers`, to allow for returning a set of matchers rather than just one.

Reviewers: gribozavr

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D65877

llvm-svn: 368681
2019-08-13 12:31:29 +00:00
Shaurya Gupta 7137736e49 [Extract] Fixed SemicolonExtractionPolicy for SwitchStmt and SwitchCase
Reviewers: arphaman, sammccall

Subscribers: dexonsmith, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D65883

llvm-svn: 368267
2019-08-08 08:37:49 +00:00
Diego Astiazaran ba55970c15 [Tooling] Expose ExecutorConcurrency option.
D65628 requires a flag to specify the number of threads for a clang-doc step. It would be good to use ExecutorConcurrency after exposing it instead of creating a new one that has the same purpose.

Differential Revision: https://reviews.llvm.org/D65833

llvm-svn: 368196
2019-08-07 18:35:28 +00:00
Alex Lorenz e1f4c4aad2 [clang-scan-deps] Implementation of dependency scanner over minimized sources
This commit implements the fast dependency scanning mode in clang-scan-deps: the
preprocessing is done on files that are minimized using the dependency directives source minimizer.

A shared file system cache is used to ensure that the file system requests and source minimization
is performed only once. The cache assumes that the underlying filesystem won't change during the course
of the scan (or if it will, it will not affect the output), and it can't be evicted. This means that the
service and workers can be used for a single run of a dependency scanner, and can't be reused across multiple,
incremental runs. This is something that we'll most likely support in the future though.
Note that the driver still utilizes the underlying real filesystem.

This commit is also still missing the fast skipped PP block skipping optimization that I mentioned at EuroLLVM talk.
Additionally, the file manager is still not reused by the threads as well.

Differential Revision: https://reviews.llvm.org/D63907

llvm-svn: 368086
2019-08-06 20:43:25 +00:00
Ilya Biryukov bfbf6b6cab [Syntax] Do not add a node for 'eof' into the tree
Summary:
While useful as a sentinel value when iterating over tokens, having
'eof' in the tree, seems to do more harm than good.

Reviewers: sammccall

Reviewed By: sammccall

Subscribers: javed.absar, kristof.beyls, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D64576

llvm-svn: 368062
2019-08-06 17:07:58 +00:00
Rainer Orth 09d890d728 Move LangStandard*, InputKind::Language to Basic
This patch is a prerequisite for using LangStandard from Driver in
https://reviews.llvm.org/D64793.

It moves LangStandard* and InputKind::Language to Basic.  It is mostly
mechanical, with only a few changes of note:

- enum Language has been changed into enum class Language : uint8_t to
  avoid a clash between OpenCL in enum Language and OpenCL in enum
  LangFeatures and not to increase the size of class InputKind.

- Now that getLangStandardForName, which is currently unused, also checks
  both canonical and alias names, I've introduced a helper getLangKind
  which factors out a code pattern already used 3 times.

The patch has been tested on x86_64-pc-solaris2.11, sparcv9-sun-solaris2.11,
and x86_64-pc-linux-gnu.

There's a companion patch for lldb which uses LangStandard.h
(https://reviews.llvm.org/D65717).

While polly includes isl which in turn uses InputKind::C, that part of the
code isn't even built inside the llvm tree.  I've posted a patch to allow
for both InputKind::C and Language::C upstream
(https://groups.google.com/forum/#!topic/isl-development/6oEvNWOSQFE).

Differential Revision: https://reviews.llvm.org/D65562

llvm-svn: 367864
2019-08-05 13:59:26 +00:00
Harlan Haskins 8d323d1506 [clang] Adopt new FileManager error-returning APIs
Update the callers of FileManager::getFile and FileManager::getDirectory to handle the new llvm::ErrorOr-returning methods.

Signed-off-by: Harlan Haskins <harlan@apple.com>
llvm-svn: 367616
2019-08-01 21:31:56 +00:00
Yitzhak Mandelbaum 3f1ab737e2 [LibTooling] Relax Transformer to allow rewriting macro expansions
Summary:
Currently, Transformer rejects any changes to source locations inside macro
expansions. This change relaxes that constraint to allow rewrites when the
entirety of the expansion is replaced, since that can be mapped to replacing the
entirety of the expansion range in the file source.  This change makes
Transformer consistent with the handling of edit ranges in `clang::edit::Commit`
(which is used, for example, for applying `FixItHint`s from diagnostics).

Reviewers: ilya-biryukov

Subscribers: gribozavr, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D64518

llvm-svn: 366473
2019-07-18 17:44:54 +00:00
Yitzhak Mandelbaum 2e97a1e19e [LibTooling] Add function to translate and validate source range for editing
Summary:
Adds the function `getRangeForEdit` to validate that a given source range is
editable and, if needed, translate it into a range in the source file (for
example, if it's sourced in macro expansions).

Reviewers: ilya-biryukov

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D64924

llvm-svn: 366469
2019-07-18 17:26:57 +00:00
Sam McCall 5d9d7c59ee Re-land [JSONCompilationDatabase] Strip distcc/ccache/gomacc wrappers from parsed commands.
Use //net/dir like other test cases for windows compatibility

llvm-svn: 365975
2019-07-12 23:38:31 +00:00
Russell Gallop 614a78c15a Revert "[JSONCompilationDatabase] Strip distcc/ccache/gomacc wrappers from parsed commands."
New test is failing on Windows bot

This reverts commit 9c0391b36a.

llvm-svn: 365906
2019-07-12 15:15:56 +00:00
Sam McCall 9c0391b36a [JSONCompilationDatabase] Strip distcc/ccache/gomacc wrappers from parsed commands.
Summary:
It's common to use compiler wrappers by setting CC="gomacc clang++".
This results in both args appearing in compile_commands.json, and clang's driver
can't handle this.

This patch attempts to recognize this pattern (by looking for well-known
wrappers) and dropping argv0 in this case.

It conservatively ignores other cases for now:
 - wrappers with unknown names
 - wrappers that accept -flags
 - wrappers where the compiler to use is implied (usually cc or gcc)

This is done at the JSONCompilationDatabase level rather than somewhere more
fundamental, as (hopefully) this isn't a general conceptual problem, but a messy
aspect of the nature of the ecosystem around compile_commands.json.
i.e. compilation databases more tightly tied to the build system should not have
this problem.

Reviewers: phosek, klimek

Subscribers: llvm-commits

Tags: #llvm

Differential Revision: https://reviews.llvm.org/D64297

llvm-svn: 365887
2019-07-12 10:11:40 +00:00
Ilya Biryukov 625a0f70a9 [Syntax] Add assertion to catch invalid tokens early. NFC
To help with identifiying root cause of a crash we are seeing.

llvm-svn: 365599
2019-07-10 08:24:42 +00:00
Ilya Biryukov 51dad4196e [Syntax] Move roles into a separate enum
To align with reviewer's suggestions.

llvm-svn: 365479
2019-07-09 13:31:43 +00:00
Ilya Biryukov 9b3f38f990 Reland r365355: [Syntax] Introduce syntax trees
With a fix to a PS4 buildbot crash.

llvm-svn: 365466
2019-07-09 11:32:13 +00:00
Simon Pilgrim 7e3b22758d Revert rL365355 : [Syntax] Introduce syntax trees
Summary:
A tooling-focused alternative to the AST. This commit focuses on the
memory-management strategy and the structure of the AST.

More to follow later:
  - Operations to mutate the syntax trees and corresponding textual
    replacements.
  - Mapping between clang AST nodes and syntax tree nodes.
  - More node types corresponding to the language constructs.

Reviewers: sammccall

Reviewed By: sammccall

Subscribers: llvm-commits, mgorny, cfe-commits

Tags: #clang, #llvm

Differential Revision: https://reviews.llvm.org/D61637
........
Fixes buildbots which were crashing on SyntaxTests.exe

llvm-svn: 365465
2019-07-09 11:26:35 +00:00