AST file more lazy, so that we don't eagerly load that information for
all known identifiers each time a new AST file is loaded. The eager
reloading made some sense in the context of precompiled headers, since
very few identifiers were defined before PCH load time. With modules,
however, a huge amount of code can get parsed before we see an
@import, so laziness becomes important here.
The approach taken to make this information lazy is fairly simple:
when we load a new AST file, we mark all of the existing identifiers
as being out-of-date. Whenever we want to access information that may
come from an AST (e.g., whether the identifier has a macro definition,
or what top-level declarations have that name), we check the
out-of-date bit and, if it's set, ask the AST reader to update the
IdentifierInfo from the AST files. The update is a merge, and we now
take care to merge declarations before/after imports with declarations
from multiple imports.
The results of this optimization are fairly dramatic. On a small
application that brings in 14 non-trivial modules, this takes modules
from being > 3x slower than a "perfect" PCH file down to 30% slower
for a full rebuild. A partial rebuild (where the PCH file or modules
can be re-used) is down to 7% slower. Making the PCH file just a
little imperfect (e.g., adding two smallish modules used by a bunch of
.m files that aren't in the PCH file) tips the scales in favor of the
modules approach, with 24% faster partial rebuilds.
This is just a first step; the lazy scheme could possibly be improved
by adding versioning, so we don't search into modules we already
searched. Moreover, we'll need similar lazy schemes for all of the
other lookup data structures, such as DeclContexts.
llvm-svn: 143100
preprocessed entities that are #included in the range that we are interested.
This is useful when we are interested in preprocessed entities of a specific file, e.g
when we are annotating tokens. There is also an optimization where we cache the last
result of PreprocessingRecord::getPreprocessedEntitiesInRange and we re-use it if
there is a call with the same range as before.
rdar://10313365
llvm-svn: 142887
CoreFoundation object-transfer properties audited, and add a #pragma
to cause them to be automatically applied to functions in a particular
span of code. This has to be implemented largely in the preprocessor
because of the requirement that the region be entirely contained in
a single file; that's hard to impose from the parser without registering
for a ton of callbacks.
llvm-svn: 140846
PreprocessingRecord's getPreprocessedEntitiesInRange.
Also remove all the stuff that were added in ASTUnit that are unnecessary now
that we do a binary search for preprocessed entities and deserialize only
what is necessary.
llvm-svn: 140063
which will do a binary search and return a pair of iterators
for preprocessed entities in the given source range.
Source ranges of preprocessed entities are stored twice currently in
the PCH/Module file but this will be fixed in a subsequent commit.
llvm-svn: 140058
-Use an array of offsets for all preprocessed entities
-Get rid of the separate array of offsets for just macro definitions;
for references to macro definitions use an index inside the preprocessed
entities array.
-Deserialize each preprocessed entity lazily, at first request; not in bulk.
Paves the way for binary searching of preprocessed entities that will offer
efficiency and will simplify things on the libclang side a lot.
llvm-svn: 139809
target triple to separate modules built under different
conditions. The hash is used to create a subdirectory in the module
cache path where other invocations of the compiler (with the same
version, language options, etc.) can find the precompiled modules.
llvm-svn: 139662
but there is a corresponding umbrella header in a framework, build the
module on-the-fly so it can be immediately loaded at the import
statement. This is very much proof-of-concept code, with details to be
fleshed out over time.
llvm-svn: 139558
where the compiler will look for module files. Eliminates the
egregious hack where we looked into the header search paths for
modules.
llvm-svn: 139538
keyword. We now handle this keyword in HandleIdentifier, making a note
for ourselves when we've seen the __import_module__ keyword so that
the next lexed token can trigger a module import (if needed). This
greatly simplifies Preprocessor::Lex(), and completely erases the 5.5%
-Eonly slowdown Argiris noted when I originally implemented
__import_module__. Big thanks to Argiris for noting that horrible
regression!
llvm-svn: 139265
Previously we would cut off the source file buffer at the code-completion
point; this impeded code-completion inside C++ inline methods and,
recently, with buffering ObjC methods.
Have the code-completion inserted into the source buffer so that it can
be buffered along with a method body. When we actually hit the code-completion
point the cut-off lexing or parsing.
Fixes rdar://10056932&8319466
llvm-svn: 139086
and language-specific initialization. Use this to allow ASTUnit to
create a preprocessor object *before* loading the AST file. No actual
functionality change.
llvm-svn: 138983
LangOptions, rather than making distinct copies of
LangOptions. Granted, LangOptions doesn't actually get modified, but
this will eventually make it easier to construct ASTContext and
Preprocessor before we know all of the LangOptions.
llvm-svn: 138959
include guards don't show up as macro definitions in every translation
unit that imports a module. Macro definitions can, however, be
exported with the intentionally-ugly #__export_macro__
directive. Implement this feature by not even bothering to serialize
non-exported macros to a module, because clients of that module need
not (should not) know that these macros even exist.
llvm-svn: 138943
existing practice with Python extension modules. Not that Python
extension modules should be using a double-underscored identifier
anyway, but...
llvm-svn: 138870
__import__ within the preprocessor, since the prior one foolishly
assumed that Preprocessor::Lex() was re-entrant. We now handle
__import__ at the top level (only), after macro expansion. This should
fix the buildbot failures.
llvm-svn: 138704
loads the named module. The syntax itself is intentionally hideous and
will be replaced at some later point with something more
palatable. For now, we're focusing on the semantics:
- Module imports are handled first by the preprocessor (to get macro
definitions) and then the same tokens are also handled by the parser
(to get declarations). If both happen (as in normal compilation),
the second one is redundant, because we currently have no way to
hide macros or declarations when loading a module. Chris gets credit
for this mad-but-workable scheme.
- The Preprocessor now holds on to a reference to a module loader,
which is responsible for loading named modules. CompilerInstance is
the only important module loader: it now knows how to create and
wire up an AST reader on demand to actually perform the module load.
- We search for modules in the include path, using the module name
with the suffix ".pcm" (precompiled module) for the file name. This
is a temporary hack; we hope to improve the situation in the
future.
llvm-svn: 138679
Currently getMacroArgExpandedLocation is very inefficient and for the case
of a location pointing at the main file it will end up checking almost all of
the SLocEntries. Make it faster:
-Use a map of macro argument chunks to their expanded source location. The map
is for a single source file, it's stored in the file's ContentCache and lazily
computed, like the source lines cache.
-In SLocEntry's FileInfo add an 'unsigned NumCreatedFIDs' field that keeps track
of the number of FileIDs (files and macros) that were created during preprocessing
of that particular file SLocEntry. This is useful when computing the macro argument
map in skipping included files while scanning for macro arg FileIDs that lexed from
a specific source file. Due to padding, the new field does not increase the size
of SLocEntry.
llvm-svn: 138225
for tokens that are lexed consecutively from the same FileID, instead of creating
a SLocEntry for each token. e.g for
assert(foo == bar);
there will be a single SLocEntry for the "foo == bar" chunk and locations
for the 'foo', '==', 'bar' tokens will point inside that chunk.
For parsing SemaExpr.cpp, this reduced the number of SLocEntries by 25%.
llvm-svn: 138129
1. Be more tolerant of comments in -CC (comment-preserving) mode. We were missing a few cases.
2. Make sure to expand the second FOO in "#if defined FOO FOO". (See also
r97253, which addressed the case of "#if defined(FOO FOO".)
Fixes PR10286.
llvm-svn: 136748
entities generated directly by the preprocessor from those loaded from
the external source (e.g., the ASTReader). By separating these two
sets of entities into different vectors, we allow both to grow
independently, and eliminate the need for preallocating all of the
loaded preprocessing entities. This is similar to the way the recent
SourceManager refactoring treats FileIDs and the source location
address space.
As part of this, switch over to building a continuous range map to
track preprocessing entities.
llvm-svn: 135646
variants to 'expand'. This changed a couple of public APIs, including
one public type "MacroInstantiation" which is now "MacroExpansion". The
rest of the codebase was updated to reflect this, especially the
libclang code. Two of the C++ (and thus easily changed) libclang APIs
were updated as well because they pertained directly to the old
MacroInstantiation class.
No functionality changed.
llvm-svn: 135139
'expand'. Also update the public API it provides to the new term, and
propagate that update to the various clients.
No functionality changed.
llvm-svn: 135138
When a macro instantiation occurs, reserve a SLocEntry chunk with length the
full length of the macro definition source. Set the spelling location of this chunk
to point to the start of the macro definition and any tokens that are lexed directly
from the macro definition will get a location from this chunk with the appropriate offset.
For any tokens that come from argument expansion, '##' paste operator, etc. have their
instantiation location point at the appropriate place in the instantiated macro definition
(the argument identifier and the '##' token respectively).
This improves macro instantiation diagnostics:
Before:
t.c:5:9: error: invalid operands to binary expression ('struct S' and 'int')
int y = M(/);
^~~~
t.c:5:11: note: instantiated from:
int y = M(/);
^
After:
t.c:5:9: error: invalid operands to binary expression ('struct S' and 'int')
int y = M(/);
^~~~
t.c:3:20: note: instantiated from:
\#define M(op) (foo op 3);
~~~ ^ ~
t.c:5:11: note: instantiated from:
int y = M(/);
^
The memory savings for a candidate boost library that abuses the preprocessor are:
- 32% less SLocEntries (37M -> 25M)
- 30% reduction in PCH file size (900M -> 635M)
- 50% reduction in memory usage for the SLocEntry table (1.6G -> 800M)
llvm-svn: 134587
Previously macro expanded tokens were added to Preprocessor's bump allocator and never released,
even after the TokenLexer that were lexing them was finished, thus they were wasting memory.
A very "useful" boost library was causing clang to eat 1 GB just for the expanded macro tokens.
Introduce a special cache that works like a stack; a TokenLexer can add the macro expanded tokens
in the cache, and when it finishes, the tokens are removed from the end of the cache.
Now consumed memory by expanded tokens for that library is ~ 1.5 MB.
Part of rdar://9327049.
llvm-svn: 134105
1. We would assume that the length of the string literal token was at least 2
2. We would allocate a buffer with size length-2
And when the stars aligned (one of which would be an invalid source location due to stale PCH)
The length would be 0 and we would try to allocate a 4GB buffer.
Add checks for this corner case and a bunch of asserts.
(We really really should have had an assert for 1.).
Note that there's no test case since I couldn't get one (it was major PITA to reproduce),
maybe later.
llvm-svn: 131492
__has_extension is a function-like macro which takes the same set
of feature identifiers as __has_feature. It evaluates to 1 if the
feature is supported by Clang in the current language (either as a
language extension or a standard language feature) or 0 if not.
At the same time, add support for the C1X feature identifiers
c_generic_selections (renamed from generic_selections) and
c_static_assert, and document them.
Patch by myself and Jean-Daniel Dupas.
llvm-svn: 131308
CXTranslationUnit_NestedMacroInstantiations, which indicates whether
we want to see "nested" macro instantiations (e.g., those that occur
inside other macro instantiations) within the detailed preprocessing
record. Many clients (e.g., those that only care about visible tokens)
don't care about this information, and in code that uses preprocessor
metaprogramming, this information can have a very high cost.
Addresses <rdar://problem/9389320>.
llvm-svn: 130990
which determines whether a particular file is actually a header that
is intended to be guarded from multiple inclusions within the same
translation unit.
llvm-svn: 130808
includes get resolved, especially when they are found relatively to
another include file. We also try to get it working for framework
includes, but that part of the code is untested, as I don't have a code
base that uses it.
llvm-svn: 130246
This change requires making a bunch of fundamental Clang structures (optionally) reference counted to allow correct
ownership semantics of these objects (e.g., ASTContext) to play out between an active ASTUnit and CompilerInstance
object.
llvm-svn: 128011
clients to observe the exact path through which an #included file was
located. This is very useful when trying to record and replay inclusion
operations without it beind influenced by the aggressive caching done
inside the FileManager to avoid redundant system calls and filesystem
operations.
The work to compute and return this is only done in the presence of
callbacks, so it should have no effect on normal compilation.
Patch by Manuel Klimek.
llvm-svn: 127742
The previous name was inaccurate as this token in fact appears at
the end of every preprocessing directive, not just macro definitions.
No functionality change, except for a diagnostic tweak.
llvm-svn: 126631
AST/PCH files more lazy:
- Don't preload all of the file source-location entries when reading
the AST file. Instead, load them lazily, when needed.
- Only look up header-search information (whether a header was already
#import'd, how many times it's been included, etc.) when it's needed
by the preprocessor, rather than pre-populating it.
Previously, we would pre-load all of the file source-location entries,
which also populated the header-search information structure. This was
a relatively minor performance issue, since we would end up stat()'ing
all of the headers stored within a AST/PCH file when the AST/PCH file
was loaded. In the normal PCH use case, the stat()s were cached, so
the cost--of preloading ~860 source-location entries in the Cocoa.h
case---was relatively low.
However, the recent optimization that replaced stat+open with
open+fstat turned this into a major problem, since the preloading of
source-location entries would now end up opening those files. Worse,
those files wouldn't be closed until the file manager was destroyed,
so just opening a Cocoa.h PCH file would hold on to ~860 file
descriptors, and it was easy to blow through the process's limit on
the number of open file descriptors.
By eliminating the preloading of these files, we neither open nor stat
the headers stored in the PCH/AST file until they're actually needed
for something. Concretely, we went from
*** HeaderSearch Stats:
835 files tracked.
364 #import/#pragma once files.
823 included exactly once.
6 max times a file is included.
3 #include/#include_next/#import.
0 #includes skipped due to the multi-include optimization.
1 framework lookups.
0 subframework lookups.
*** Source Manager Stats:
835 files mapped, 3 mem buffers mapped.
37460 SLocEntry's allocated, 11215575B of Sloc address space used.
62 bytes of files mapped, 0 files with line #'s computed.
with a trivial program that uses a chained PCH including a Cocoa PCH
to
*** HeaderSearch Stats:
4 files tracked.
1 #import/#pragma once files.
3 included exactly once.
2 max times a file is included.
3 #include/#include_next/#import.
0 #includes skipped due to the multi-include optimization.
1 framework lookups.
0 subframework lookups.
*** Source Manager Stats:
3 files mapped, 3 mem buffers mapped.
37460 SLocEntry's allocated, 11215575B of Sloc address space used.
62 bytes of files mapped, 0 files with line #'s computed.
for the same program.
llvm-svn: 125286
When we are in code-completion mode, skip parsing of all function bodies except the one where the
code-completion point resides.
For big .cpp files like 'SemaExpr.cpp' the improvement makes a huge difference, in some cases cutting down
code-completion time -62% !
We don't get diagnostics for the bodies though, so modify the code-completion tests that check for errors.
See rdar://8814203.
llvm-svn: 122765
Diagnostic pragmas are broken because we don't keep track of the diagnostic state changes and we only check the current/latest state.
Problems manifest if a diagnostic is emitted for a source line that has different diagnostic state than the current state; this can affect
a lot of places, like C++ inline methods, template instantiations, the lexer, etc.
Fix the issue by having the Diagnostic object keep track of the source location of the pragmas so that it is able to know what is the diagnostic state at any given source location.
Fixes rdar://8365684.
llvm-svn: 121873
trap the serialized preprocessing records (macro definitions, macro
instantiations, macro definitions) from the generation of the
precompiled preamble, then replay those when walking the list of
preprocessed entities. This eliminates a bug where clang_getCursor()
wasn't able to find preprocessed-entity cursors in the preamble.
llvm-svn: 120396
FileSystemOpts through a ton of apis, simplifying a lot of code.
This also fixes a latent bug in ASTUnit where it would invoke
methods on FileManager without creating one in some code paths
in cindextext.
llvm-svn: 120010
and use a better and more general approach, where NullStmt has a flag to indicate whether it was preceded by an empty macro.
Thanks to Abramo Bagnara for the hint!
llvm-svn: 119887
than a Token that holds the same information all in one easy-to-use
package. There's no technical reason to prefer the former -- the
information comes from a Token originally -- and it's clumsier to use,
so I've changed the code to use tokens everywhere.
Approved by clattner
llvm-svn: 119845
-Move the stuff of Diagnostic related to creating/querying diagnostic IDs into a new DiagnosticIDs class.
-DiagnosticIDs can be shared among multiple Diagnostics for multiple translation units.
-The rest of the state in Diagnostic object is considered related and tied to one translation unit.
-Have Diagnostic point to the SourceManager that is related with. Diagnostic can now accept just a
SourceLocation instead of a FullSourceLoc.
-Reflect the changes to various interfaces.
llvm-svn: 119730
The callback info for #if/#elif is not great -- ideally it would give
us a list of tokens in the #if, or even better, a little parse tree.
But that's a lot more work. Instead, clients can retokenize using
Lexer::LexFromRawLexer().
Reviewed by nlewycky.
llvm-svn: 118318
When -working-directory is passed in command line, file paths are resolved relative to the specified directory.
This helps both when using libclang (where we can't require the user to actually change the working directory)
and to help reproduce test cases when the reproduction work comes along.
--FileSystemOptions is introduced which controls how file system operations are performed (currently it just contains
the working directory value if set).
--FileSystemOptions are passed around to various interfaces that perform file operations.
--Opening & reading the content of files should be done only through FileManager. This is useful in general since
file operations will be abstracted in the future for the reproduction mechanism.
FileSystemOptions is independent of FileManager so that we can have multiple translation units sharing the same
FileManager but with different FileSystemOptions.
Addresses rdar://8583824.
llvm-svn: 118203
load identifiers without loading their corresponding macro
definitions. This is likely to improve PCH performance slightly, and
reduces deserialization stack depth considerably when using
preprocessor metaprogramming.
llvm-svn: 117750
inclusion directives, keeping track of every #include, #import,
etc. in the translation unit. We keep track of the source location and
kind of the inclusion, how the file name was spelled, and the
underlying file to which the inclusion resolved.
llvm-svn: 116952
Now MICache is a linked list (per the FIXME), where we tradeoff between MacroInfo objects being in MICache
and MIChainHead. MacroInfo objects in the MICache chain are already "Destroy()'ed", so they can be reused. When
inserting into MICache, we need to remove them from the regular linked list so that they aren't destroyed more than
once.
llvm-svn: 116869
list of allocated MacroInfos. This requires only 1 extra pointer per MacroInfo object, and allows us to blow them
away in one place. This fixes an elusive memory leak with MacroInfos (whose exact location I couldn't still figure
out despite substantial digging).
Fixes <rdar://problem/8361834>.
llvm-svn: 116842
spelled (#pragma, _Pragma, __pragma). In -E mode, use that information
to add appropriate newlines when translating _Pragma and __pragma into
#pragma, like GCC does. Fixes <rdar://problem/8412013>.
llvm-svn: 113553
The extra data stored on user-defined literal Tokens is stored in extra
allocated memory, which is managed by the PreprocessorLexer because there isn't
a better place to put it that makes sure it gets deallocated, but only after
it's used up. My testing has shown no significant slowdown as a result, but
independent testing would be appreciated.
llvm-svn: 112458
reparsing an ASTUnit. When saving a preamble, create a buffer larger
than the actual file we're working with but fill everything from the
end of the preamble to the end of the file with spaces (so the lexer
will quickly skip them). When we load the file, create a buffer of the
same size, filling it with the file and then spaces. Then, instruct
the lexer to start lexing after the preamble, therefore continuing the
parse from the spot where the preamble left off.
It's now possible to perform a simple preamble build + parse (+
reparse) with ASTUnit. However, one has to disable a bunch of checking
in the PCH reader to do so. That part isn't committed; it will likely
be handled with some other kind of flag (e.g., -fno-validate-pch).
As part of this, fix some issues with null termination of the memory
buffers created for the preamble; we were trying to explicitly
NULL-terminate them, even though they were also getting implicitly
NULL terminated, leading to excess warnings about NULL characters in
source files.
llvm-svn: 109445
is present.
Rather than using clang_getCursorExtent(), which requires
us to lex the token at the ending position to determine its
length. Then, we'd be comparing [a, b) source ranges that cover the
characters in the range rather than the normal behavior for Clang's
source ranges, which covers the tokens in the range. However, relexing
causes us to read the source file (which may come from a precompiled
header), which is rather unfortunate and affects performance.
In the new scheme, we only use Clang-style source ranges that cover
the tokens in the range. At the entry points where this matters
(clang_annotateTokens, clang_getCursor), we make sure to move source
locations to the start of the token.
Addresses most of <rdar://problem/8049381>.
llvm-svn: 109134
which is the part of the file that contains all of the initial
comments, includes, and preprocessor directives that occur before any
of the actual code. Added a new -print-preamble cc1 action that is
only used for testing.
llvm-svn: 108913
When loading the PCH, IdentifierInfos that are associated with pragmas cause declarations that use these identifiers to be deserialized (e.g. the "clang" pragma causes the "clang" namespace to be loaded).
We can avoid this if we just use StringRefs for the pragmas.
As a bonus, since we don't have to create and pass IdentifierInfos, the pragma interfaces get a bit more simplified.
llvm-svn: 108237
diagnostics. That would be while we're parsing string literals for the
sole purpose of producing a diagnostic about them. Fixes
<rdar://problem/8026030>.
llvm-svn: 104684
and diagnostic groups. This allows the compiler to group
diagnostics together (e.g. "Logic Warning",
"Format String Warning", etc) like the static analyzer does.
This is not exposed through anything in the compiler yet.
llvm-svn: 103051
instantiations when we have the corresponding macro definition and by
removing macro definition information from our table when the macro is
undefined.
llvm-svn: 99004
record (which includes all macro instantiations and definitions). As
with all lay deserialization, this introduces a new external source
(here, an external preprocessing record source) that loads all of the
preprocessed entities prior to iterating over the entities.
The preprocessing record is an optional part of the precompiled header
that is disabled by default (enabled with
-detailed-preprocessing-record). When the preprocessor given to the
PCH writer has a preprocessing record, that record is written into the
PCH file. When the PCH reader is given a PCH file that contains a
preprocessing record, it will be lazily loaded (which, effectively,
implicitly adds -detailed-preprocessing-record). This is the first
case where we have sections of the precompiled header that are
added/removed based on a compilation flag, which is
unfortunate. However, this data consumes ~550k in the PCH file for
Cocoa.h (out of ~9.9MB), and there is a non-trivial cost to gathering
this detailed preprocessing information, so it's too expensive to turn
on by default. In the future, we should investigate a better encoding
of this information.
llvm-svn: 99002
eliminating the extra PopulatePreprocessingRecord object. This will
become useful once we start writing the preprocessing record to
precompiled headers.
llvm-svn: 98966
preprocessing record. Use that link with clang_getCursorReferenced()
and clang_getCursorDefinition() to match instantiations of a macro to
the definition of the macro.
llvm-svn: 98842
the macro definitions and macro instantiations that are found
during preprocessing. Preprocessing records are *not* generated by
default; rather, we provide a PPCallbacks subclass that hooks into the
existing callback mechanism to record this activity.
The only client of preprocessing records is CIndex, which keeps track
of macro definitions and instantations so that they can be exposed via
cursors. At present, only token annotation uses these facilities, and
only for macro instantiations; both will change in the near
future. However, with this change, token annotation properly annotates
macro instantiations that do not produce any tokens and instantiations
of macros that are later undef'd, improving our consistency.
Preprocessing directives that are not macro definitions are still
handled by clang_annotateTokens() via re-lexing, so that we don't have
to track every preprocessing directive in the preprocessing record.
Performance impact of preprocessing records is still TBD, although it
is limited to CIndex and therefore out of the path of the main compiler.
llvm-svn: 98836
presence or absence of header map arguments when using the precompiled
header would cause Clang to get confused about which headers had
already been included/imported, along with their controlling
macros. The fundamental problem is that the serialization of the
header search information was relying on the UIDs of FileEntry objects
at PCH generation time and PCH load time to be equivalent, which
effectively means that we had to probe the same files in the same
order. Differing header map arguments caused an extra FileEntry
lookup, but it's easy to imagine other minor command-line arguments
triggering this problem.
Header-search information is now encoded along with the
source-location entry for a file, so that we register information
about a file's properties as a header at the same time we create the
FileEntry for that file.
Fixes <rdar://problem/7743243>.
llvm-svn: 98636
SourceManager's getBuffer() and, therefore, could fail, along with
Preprocessor::getSpelling(). Use the Invalid parameters in the literal
parsers (string, floating point, integral, character) to make them
robust against errors that stem from, e.g., PCH files that are not
consistent with the underlying file system.
I still need to audit every use caller to all of these routines, to
determine which ones need specific handling of error conditions.
llvm-svn: 98608
region of interest (if provided). Implement clang_getCursor() in terms
of this traversal rather than using the Index library; the unified
cursor visitor is more complete, and will be The Way Forward.
Minor other tweaks needed to make this work:
- Extend Preprocessor::getLocForEndOfToken() to accept an offset
from the end, making it easy to move to the last character in the
token (rather than just past the end of the token).
- In Lexer::MeasureTokenLength(), the length of whitespace is zero.
llvm-svn: 94200
definitions from a precompiled header. This ensures that
code-completion with macro names behaves the same with or without
precompiled headers.
llvm-svn: 92497
We creating and free thousands of MacroArgs objects (and the related
std::vectors hanging off them) for the testcase in PR5610 even though
there are only ~20 live at a time. This doesn't actually use the
cache yet.
llvm-svn: 91391
files with the contents of an arbitrary memory buffer. Use this new
functionality to drastically clean up the way in which we handle file
truncation for code-completion: all of the truncation/completion logic
is now encapsulated in the preprocessor where it belongs
(<rdar://problem/7434737>).
llvm-svn: 90300
in diagnostics when we fail to open a file. This allows us to
report things like:
$ clang test.c -I.
test.c:2:10: fatal error: error opening file './foo.h': Permission denied
#include "foo.h"
^
llvm-svn: 90276
stat a file but where mmaping it fails. In this case, we emit an
error like:
t.c:1:10: fatal error: error opening file '../../foo.h'
instead of "cannot find file".
llvm-svn: 90110
handling template template parameters properly. This refactoring:
- Parses template template arguments as id-expressions, representing
the result of the parse as a template name (Action::TemplateTy)
rather than as an expression (lame!).
- Represents all parsed template arguments via a new parser-specific
type, ParsedTemplateArgument, which stores the kind of template
argument (type, non-type, template) along with all of the source
information about the template argument. This replaces an ad hoc
set of 3 vectors (one for a void*, which was either a type or an
expression; one for a bit telling whether the first was a type or
an expression; and one for a single source location pointing at
the template argument).
- Moves TemplateIdAnnotation into the new Parse/Template.h. It never
belonged in the Basic library anyway.
llvm-svn: 86708
a little fuzzy, but conceptually it's just uniquing the identifier.
Chris, please review. I debated splitting into const/non-const versions where
the const one propogated constness to the resulting IdentifierInfo*.
llvm-svn: 86106
operators, e.g.,
operator+<int>
which now works in declarators, id-expressions, and member access
expressions. This commit only implements the non-dependent case, where
we can resolve the template-id to an actual declaration.
llvm-svn: 85966
-code-completion-at=filename:line:column
which performs code completion at the specified location by truncating
the file at that position and enabling code completion. This approach
makes it possible to run multiple tests from a single test file, and
gives a more natural command-line interface.
llvm-svn: 82571
essence, code completion is triggered by a magic "code completion"
token produced by the lexer [*], which the parser recognizes at
certain points in the grammar. The parser then calls into the Action
object with the appropriate CodeCompletionXXX action.
Sema implements the CodeCompletionXXX callbacks by performing minimal
translation, then forwarding them to a CodeCompletionConsumer
subclass, which uses the results of semantic analysis to provide
code-completion results. At present, only a single, "printing" code
completion consumer is available, for regression testing and
debugging. However, the design is meant to permit other
code-completion consumers.
This initial commit contains two code-completion actions: one for
member access, e.g., "x." or "p->", and one for
nested-name-specifiers, e.g., "std::". More code-completion actions
will follow, along with improved gathering of code-completion results
for the various contexts.
[*] In the current -code-completion-dump testing/debugging mode, the
file is truncated at the completion point and EOF is translated into
"code completion".
llvm-svn: 82166
declaration in the AST.
The new ASTContext::getCommentForDecl function searches for a comment
that is attached to the given declaration, and returns that comment,
which may be composed of several comment blocks.
Comments are always available in an AST. However, to avoid harming
performance, we don't actually parse the comments. Rather, we keep the
source ranges of all of the comments within a large, sorted vector,
then lazily extract comments via a binary search in that vector only
when needed (which never occurs in a "normal" compile).
Comments are written to a precompiled header/AST file as a blob of
source ranges. That blob is only lazily loaded when one requests a
comment for a declaration (this never occurs in a "normal" compile).
The indexer testbed now supports comment extraction. When the
-point-at location points to a declaration with a Doxygen-style
comment, the indexer testbed prints the associated comment
block(s). See test/Index/comments.c for an example.
Some notes:
- We don't actually attempt to parse the comment blocks themselves,
beyond identifying them as Doxygen comment blocks to associate them
with a declaration.
- We won't find comment blocks that aren't adjacent to the
declaration, because we start our search based on the location of
the declaration.
- We don't go through the necessary hops to find, for example,
whether some redeclaration of a declaration has comments when our
current declaration does not. Similarly, we don't attempt to
associate a \param Foo marker in a function body comment with the
parameter named Foo (although that is certainly possible).
- Verification of my "no performance impact" claims is still "to be
done".
llvm-svn: 74704
Implement support for C++ Substitution Failure Is Not An Error
(SFINAE), which says that errors that occur during template argument
deduction do *not* produce diagnostics and do not necessarily make a
program ill-formed. Instead, template argument deduction silently
fails. This is currently implemented for template argument deduction
during matching of class template partial specializations, although
the mechanism will also apply to template argument deduction for
function templates. The scheme is simple:
- If we are in a template argument deduction context, any diagnostic
that is considered a SFINAE error (or warning) will be
suppressed. The error will be propagated up the call stack via the
normal means.
- By default, all warnings and errors are SFINAE errors. Add the
NoSFINAE class to a diagnostic in the .td file to make it a hard
error (e.g., for access-control violations).
Note that, to make this fully work, every place in Sema that emits an
error *and then immediately recovers* will need to check
Sema::isSFINAEContext() to determine whether it must immediately
return an error rather than recovering.
llvm-svn: 73332
preprocessor and initialize it early in clang-cc. This
ensures that __has_builtin works in all modes, not just
when ASTContext is around.
llvm-svn: 73319
builtin preprocessor macro. This appears to work with two caveats:
1) builtins are registered in -E mode, and 2) target-specific builtins
are unconditionally registered even if they aren't supported by the
target (e.g. SSE4 builtin when only SSE1 is enabled).
llvm-svn: 73289
PCH file. In the Cocoa-prefixed "Hello, World" benchmark, this takes
us from reading 503 identifiers down to 37 and from 470 macros down to
4. It also results in an 8% performance improvement.
llvm-svn: 70094
This allows it to accurately measure tokens, so that we get:
t.cpp:8:13: error: unknown type name 'X'
static foo::X P;
~~~~~^
instead of the woefully inferior:
t.cpp:8:13: error: unknown type name 'X'
static foo::X P;
~~~~ ^
Most of this is just plumbing to push the reference around.
llvm-svn: 69099
buffer generated for the current translation unit. If they are
different, complain and then ignore the PCH file. This effectively
checks for all compilation options that somehow would affect
preprocessor state (-D, -U, -include, the dreaded -imacros, etc.).
When we do accept the PCH file, throw away the contents of the
predefines buffer rather than parsing them, since all of the results
of that parsing are already stored in the PCH file. This eliminates
the ugliness with the redefinition of __builtin_va_list, among other
things.
llvm-svn: 68838
PCH. This works now, except for limitations not being able to do things
with identifiers. The basic example in the testcase works though.
llvm-svn: 68832
improvement, source locations read from the PCH file will properly
resolve to the source files that were used to build the PCH file
itself.
Once we have the preprocessor state stored in the PCH file, source
locations that refer to macro instantiations that occur in the PCH
file should have the appropriate instantiation information.
llvm-svn: 68758
- Make the Diagnostic::Level for PTH errors to be specified by the caller
clang (driver):
- Set the PTHManager diagnostic level to "Diagnostic::Error" for -include-pth
(a hard error) and Diagnostic::Warning for -token-cache (we can still
proceed).
llvm-svn: 67462
std::vector<int>::allocator_type
When we parse a template-id that names a type, it will become either a
template-id annotation (which is a parsed representation of a
template-id that has not yet been through semantic analysis) or a
typename annotation (where semantic analysis has resolved the
template-id to an actual type), depending on the context. We only
produce a type in contexts where we know that we only need type
information, e.g., in a type specifier. Otherwise, we create a
template-id annotation that can later be "upgraded" by transforming it
into a typename annotation when the parser needs a type. This occurs,
for example, when we've parsed "std::vector<int>" above and then see
the '::' after it. However, it means that when writing something like
this:
template<> class Outer::Inner<int> { ... };
We have two tokens to represent Outer::Inner<int>: one token for the
nested name specifier Outer::, and one template-id annotation token
for Inner<int>, which will be passed to semantic analysis to define
the class template specialization.
Most of the churn in the template tests in this patch come from an
improvement in our error recovery from ill-formed template-ids.
llvm-svn: 65467
to being allocated from the same bumpptr that the MacroInfo objects
themselves are.
This speeds up -Eonly cocoa.h pth by ~4%, fsyntax-only is barely measurable.
llvm-svn: 65195
escapes in the string for subtoken positioning. This gives
us working examples like:
t.m:5:16: warning: field width should have type 'int', but argument has type 'unsigned int'
printf("\n\n%*d", (unsigned) 1, 1);
^ ~~~~~~~~~~~~
where before the caret pointed two spaces to the left.
llvm-svn: 64940
Now instead of just tracking the expansion history, also track the full
range of the macro that got replaced. For object-like macros, this doesn't
change anything. For _Pragma and function-like macros, this means we track
the locations of the ')'.
This is required for PR3579 because apparently GCC uses the line of the ')'
of a function-like macro as the location to expand __LINE__ to.
llvm-svn: 64601
to use this stat information in the PTH file using a 'StatSysCallCache' object.
Performance impact (Cocoa.h, PTH):
- number of stat calls reduces from 1230 to 425
- fsyntax-only: time improves by 4.2%
We can reduce the number of stat calls to almost zero by caching negative stat
calls and directory stat calls in the PTH file as well.
llvm-svn: 64353
actually *slightly* slower than the binary search. Since this is algorithmically
better, further performance tuning should be able to make this faster.
llvm-svn: 64326
Performance impact for -fsyntax-only on Cocoa.h (with Cocoa.h in the PTH file):
- PTH generation time improves by 5%
- PTH reading improves by 0.3%.
llvm-svn: 63072
Token now has a class of kinds for "literals", which include
numeric constants, strings, etc. These tokens can optionally have
a pointer to the start of the token in the lexer buffer. This
makes it faster to get spelling and do other gymnastics, because we
don't have to go through source locations.
This change is performance neutral, but will make other changes
more feasible down the road.
llvm-svn: 63028
ground work for implementing #line, and fixes the "out of macro ID's"
problem.
There is nothing particularly tricky about the code, other than the
very performance sensitive SourceManager::getFileID() method.
llvm-svn: 62978
method. This lets us clean up the interface and make it more obvious that
this method is *really really* _Pragma specific.
Note that _Pragma handling uglifies the Lexer in the critical path. It would
be very interesting to consider making _Pragma remapping be a new special
lexer class of its own.
llvm-svn: 62425
"FileID" a concept that is now enforced by the compiler's type checker
instead of yet-another-random-unsigned floating around.
This is an important distinction from the "FileID" currently tracked by
SourceLocation. *That* FileID may refer to the start of a file or to a
chunk within it. The new FileID *only* refers to the file (and its
#include stack and eventually #line data), it cannot refer to a chunk.
FileID is a completely opaque datatype to all clients, only SourceManager
is allowed to poke and prod it.
llvm-svn: 62407
the "physical" location of tokens, refer to the "spelling" location.
This is more concrete and useful, tokens aren't really physical objects!
llvm-svn: 62309
- IdentifierInfo can now (optionally) have its string data not be
co-located with itself. This is for use with PTH. This aspect is a
little gross, as getName() and getLength() now make assumptions
about a possible alternate representation of IdentifierInfo.
Perhaps we should make IdentifierInfo have virtual methods?
IdentifierTable:
- Added class "IdentifierInfoLookup" that can be used by
IdentifierTable to perform "string -> IdentifierInfo" lookups using
an auxilliary data structure. This is used by PTH.
- Perform tests show that IdentifierTable::get() does not slow down
because of the extra check for the IdentiferInfoLookup object (the
regular StringMap lookup does enough work to mitigate the impact of
an extra null pointer check).
- The upshot is that now that some IdentifierInfo objects might be
owned by the IdentiferInfoLookup object. This should be reviewed.
PTH:
- Modified PTHManager::GetIdentifierInfo to *not* insert entries in
IdentifierTable's string map, and instead create IdentifierInfo
objects on the fly when mapping from persistent IDs to
IdentifierInfos. This saves a ton of work with string copies,
hashing, and StringMap lookup and resizing. This change was
motivated because when processing source files in the PTH cache we
don't need to do any string -> IdentifierInfo lookups.
- PTHManager now subclasses IdentifierInfoLookup, allowing clients of
IdentifierTable to transparently use IdentifierInfo objects managed
by the PTH file. PTHManager resolves "string -> IdentifierInfo"
queries by doing a binary search over a sorted table of identifier
strings in the PTH file (the exact algorithm we use can be changed
as needed).
These changes lead to the following performance changes when using PTH on Cocoa.h:
- fsyntax-only: 10% performance improvement
- Eonly: 30% performance improvement
llvm-svn: 62273
- Use canonical FileID when using getSpelling() caching. This
addresses some cache misses we were seeing with -fsyntax-only on
Cocoa.h
- Added Preprocessor::getPhysicalCharacterAt() utility method for
clients to grab the first character at a specified sourcelocation.
This uses the PTH spelling cache.
- Modified Sema::ActOnNumericConstant() to use
Preprocessor::getPhysicalCharacterAt() instead of
SourceManager::getCharacterData() (to get PTH hits).
These changes cause -fsyntax-only to not page in any sources from
Cocoa.h. We see a speedup of 27%.
llvm-svn: 62193
- Refactor caching logic into a helper class PTHSpellingSearch
- Allow "random accesses" in the spelling cache, thus catching the remaining
cases where 'getSpelling' wasn't hitting the PTH cache
For -Eonly, PTH, Cocoa.h:
- This reduces wall time by 3% (user time unchanged, sys time reduced)
- This reduces the amount of paged source by 1112K.
The remaining 1112K still being paged in is from somewhere else
(investigating).
llvm-svn: 62009
performance gain. Here's what we see for -Eonly on Cocoa.h (using PTH):
- wall time decreases by 21% (26% speedup overall)
- system time decreases by 35%
- user time decreases by 6%
These reductions are due to not paging source files just to get spellings for
literals. The solution in place doesn't appear to be 100% yet, as we still see
some of the pages for source files getting mapped in. Using -print-stats, we see
that SourceManager maps in 7179K less bytes of source text (reduction of 75%).
Will investigate why the remaining 25% are getting paged in.
With these changes, here's how PTH compares to non-PTH on Cocoa.h:
-Eonly: PTH takes 64% of the time as non-PTH (54% speedup)
-fsyntax-only: PTH takes 89% of the time as non-PTH (11% speedup)
llvm-svn: 61913
- Added stub PTHLexer::getSpelling() that will be used for fetching cached
spellings from the PTH file. This doesn't do anything yet.
- Added a hook in Preprocessor::getSpelling() to call PTHLexer::getSpelling()
when using a PTHLexer.
- Updated PTHLexer to read the offsets of spelling tables in the PTH file.
llvm-svn: 61911
- In PTHLexer::Lex read all of the token data from PTH file before
constructing the token. The idea is to enhance locality.
- Do not use Read8/Read32 in PTHLexer::Lex. Inline these operations manually.
- Change PTHManager::ReadIdentifierInfo() to PTHManager::GetIdentifierInfo().
They are functionally the same except that PTHLexer::Lex() reads the
persistent id.
These changes result in a 3.3% speedup for PTH on Cocoa.h (-Eonly).
llvm-svn: 61363
- Embed 'eom' tokens in PTH file.
- Use embedded 'eom' tokens to not lazily generate them in the PTHLexer.
This means that PTHLexer can always advance to the next token after
reading a token (instead of buffering tokens using a copy).
- Moved logic of 'ReadToken' into Lex. GetToken & ReadToken no longer exist.
- These changes result in a 3.3% speedup (-Eonly) on Cocoa.h.
- The code is a little gross. Many cleanups are possible and should be done.
llvm-svn: 61360
become useful or correct until we (1) parse template arguments
correctly, (2) have some way to turn template-ids into types,
declarators, etc., and (3) have a real representation of templates.
llvm-svn: 61208
- Added virtual method 'getSourceLocation()' (no arguments) that gets the location of the next "observable" location (e.g., next character, next token).
PPLexerChange.cpp:
- Implemented FIXME by using PreprocessorLexer::getSourceLocation() to get the location in the file we are returning to after lexing a #included file. This appears to be slightly faster than having the branch (i.e., 'if(CurLexer)'). It's also not a really hot part of the Preprocessor.
llvm-svn: 60860
- Added method "setPTHManager" that will be called by the driver to install
a PTHManager for the Preprocessor.
- Fixed some comments.
- Added EnterSourceFileWithPTH to mirror EnterSourceFileWithLexer.
llvm-svn: 60437
its call sites. This makes it more explicit when the hasError flag is
getting set and removes a confusing difference in behavior between
PP.Diag and Diag in this code.
llvm-svn: 59863
(and carefully calculated) effect of allowing the compiler to reason
about the aliasing properties of DiagnosticBuilder object better,
allowing the whole thing to be promoted to registers instead of
resulting in a ton of stack traffic.
While I'm not very concerned about the performance of the Diag() method
invocations, I *am* more concerned about their code size and impact on the
non-diagnostic code. This patch shrinks the clang executable (in
release-asserts mode with gcc-4.2) from 14523980 to 14519816 bytes. This
isn't much, but it shrinks the lexer from 38192 to 37776, PPDirectives.o
from 31116 to 28868 bytes, etc.
llvm-svn: 59862
one for building up the diagnostic that is in flight (DiagnosticBuilder)
and one for pulling structured information out of the diagnostic when
formatting and presenting it.
There is no functionality change with this patch.
llvm-svn: 59849
- Move out logic for handling the end-of-file to LexEndOfFile (to match the Lexer) class. The logic now mirrors the Lexer class more, which allows us to pass most of the Preprocessor test cases.
llvm-svn: 59768
- Rename 'CurToken' and 'LastToken' to 'CurTokenIdx' and 'LastTokenIdx'
respectively.
- Add helper methods GetToken(), AdvanceToken(), AtLastToken() to abstract away
details of the token stream. This also allows us to easily replace their
implementation later.
llvm-svn: 59733
- This is fairly gross but although the code is conceptually the
same, introducting the union causes gcc 4.2 on x86 (darwin, if that
matters) to pessimize LexTokenInternal which is critical to our
preprocessor performance.
This speeds up -Eonly lexing of Cocoa.h by ~4.7% in my timings and
reduces the code size of LexTokenInternal by 8.6%.
llvm-svn: 59725
- Add variants of IsNonPragmaNonMacroLexer to accept an IncludeMacroStack entry
(simplifies some uses).
- Use IsNonPragmaNonMacroLexer in Preprocessor::LookupFile.
- Add 'FileID' to PreprocessorLexer, and have Preprocessor query this fileid
when looking up the FileEntry for a file
Performance testing of -Eonly on Cocoa.h shows no performance regression because
of this patch.
llvm-svn: 59666
With this snippet:
void f(a::b);
An assert is hit:
Assertion failed: CachedTokens[CachedLexPos-1].getLocation() == Tok.getAnnotationEndLoc() && "The annotation should be until the most recent cached token", file ..\..\lib\Lex\PPCaching.cpp, line 98
Introduce Preprocessor::RevertCachedTokens that reverts a specific number of tokens when backtracking is enabled.
llvm-svn: 59636
- Add variants of IsNonPragmaNonMacroLexer to accept an IncludeMacroStack entry
(simplifies some uses).
- Use IsNonPragmaNonMacroLexer in Preprocessor::LookupFile.
Performance testing of -Eonly on Cocoa.h shows no performance regression because
of this patch.
llvm-svn: 59574
- Add static method to test if the current lexer is a non-macro/non-pragma
lexer.
- Refactor some code in PPLexerChange to use this static method.
- No performance change.
llvm-svn: 59486
PreprocessorLexer, which will either be a 'Lexer' or 'PTHLexer'.
- Added stub field 'CurPTHLexer' to keep track of the current PTHLexer.
- Modified IncludeStackInfo to track both the current PTHLexer and
current PreprocessorLexer.
llvm-svn: 59472
representing the names of declarations in the C family of
languages. DeclarationName is used in NamedDecl to store the name of
the declaration (naturally), and ObjCMethodDecl is now a NamedDecl.
llvm-svn: 59441
PreprocessorLexer now has a virtual method "IndirectLex" which allows it to call the lex method of its subclasses. This is not for performance intensive operations.
llvm-svn: 59185
even whitespace, as tokens from the file. This is enabled with
L->SetKeepWhitespaceMode(true) on a raw lexer. In this mode, you too
can use clang as a really complex version of 'cat' with code like this:
Lexer RawLex(SourceLocation::getFileLoc(SM.getMainFileID(), 0),
PP.getLangOptions(), File.first, File.second);
RawLex.SetKeepWhitespaceMode(true);
Token RawTok;
RawLex.LexFromRawLexer(RawTok);
while (RawTok.isNot(tok::eof)) {
std::cout << PP.getSpelling(RawTok);
RawLex.LexFromRawLexer(RawTok);
}
This will emit exactly the input file, with no canonicalization or other
translation. Realistic clients actually do something with the tokens of
course :)
llvm-svn: 57401
same we we do an unterminated string or character literal. This makes
it so we can guarantee that the lexer never calls into the
preprocessor (which would be suicide for a raw lexer).
llvm-svn: 57395
using LexRawToken, create one and use LexFromRawLexer. This avoids
twiddling the RawLexer flag around and simplifies some code (even
speeding raw lexing up a tiny bit).
This change also improves the token paster to use a Lexer on the stack
instead of new/deleting it.
llvm-svn: 57393
to whether the fileid is a 'extern c system header' in addition to whether it
is a system header, most of this is spreading plumbing around. Once we have that,
PPLexerChange bases its "file enter/exit" notifications to PPCallbacks to
base the system header state on FileIDInfo instead of HeaderSearch. Finally,
in Preprocessor::HandleIncludeDirective, mirror logic in GCC: the system headerness
of a file being entered can be set due to the #includer or the #includee.
llvm-svn: 56688
- Added as private members for each because it is not clear where to
put the common definition. Perhaps the IdentifierInfos all of these
"pseudo-keywords" should be collected into one place (this would
KnownFunctionIDs and Objective-C property IDs, for example).
Remove Token::isNamedIdentifier.
- There isn't a good reason to use strcmp when we have interned
strings, and there isn't a good reason to encourage clients to do
so.
llvm-svn: 54794
* Move FormatError() from TextDiagnostic up to DiagClient, remove now
empty class TextDiagnostic
* Make DiagClient optional for Diagnostic
This fixes the following problems:
* -html-diags (and probably others) does now output the same set of
warnings as console clang does
* nothing crashes if one forgets to call setHeaderSearch() on
TextDiagnostic
* some code duplication is removed
llvm-svn: 54620
1) New public methods added:
-EnableBacktrackAtThisPos
-DisableBacktrack
-Backtrack
-isBacktrackEnabled
2) LookAhead() implementation is replaced with a more efficient one.
3) LookNext() is removed.
llvm-svn: 54611
related to pp-expressions. Doing so is pretty simple and this
patch implements it, yielding nice diagnostics like:
t.c:2:7: error: division by zero in preprocessor expression
#if 1 / (0 + 0)
~ ^ ~~~~~~~
t.c:5:14: error: expected ')' in preprocessor expression
#if (412 + 42
~~~~~~~~^
t.c:5:5: error: to match this '('
#if (412 + 42
^
t.c:10:10: warning: left side of operator converted from negative value to unsigned: -42 to 18446744073709551574
#if (-42 + 0U) / -2
~~~ ^ ~~
t.c:10:16: warning: right side of operator converted from negative value to unsigned: -2 to 18446744073709551614
#if (-42 + 0U) / -2
~~~~~~~~~~ ^ ~~
5 diagnostics generated.
llvm-svn: 50638
clang.cpp: InitializePreprocessor now makes a copy of the contents of PredefinesBuffer and
passes it to the preprocessor object.
clang.cpp: DriverPreprocessorFactory now calls "InitializePreprocessor" instead of this being done in main().
html::HighlightMacros() now takes a PreprocessorFactory, allowing it to conjure up a new
Preprocessor to highlight macros.
class HTMLDiagnostics now takes a PreprocessorFactory* that it can use for html::HighlightMacros().
Updated clients of HTMLDiagnostics to use this new interface.
llvm-svn: 49875
theoretically useful, but not useful in practice. It adds a bunch of
complexity, and not much value. It's best to nuke it. One big advantage
is that it means the target interfaces will soon lose their SLoc arguments
and target queries can never emit diagnostics anymore (yay). Removing this
also simplifies some of the core preprocessor which should make it slightly
faster.
Ted, I didn't simplify TripleProcessor, which can now have at most one
triple, and can probably just be removed. Please poke at it when you have
time.
llvm-svn: 47930
- More enum signeness bitfield fixes (MSVC treats enums as signed).
- Fixed in Lex/HeaderSearch.cpp an unsafe copy between two
HeaderSearch::PerFileInfo entries in a common vector. The copy involved two
calls to getFileInfo() within the assignment; these calls could have
side-effects that enlarged the internal vector, and with MSVC this would
invalidate one of the values in the assignment.
Patch by Argiris Kirtzidis!
llvm-svn: 47536
Since this behavior is useful for most classes, we might consider adding a simple 3 method class that implements the behavior. Ted said that Boost has such a class.
llvm-svn: 46654
incorrectly apply the multiple include optimization to files with
guards like:
#if !defined(x) MACRO
where MACRO could expand to different things in different contexts.
Thanks Neil!
llvm-svn: 45716
both Preprocessor and ASTContext, we no longer need to explicitly pass
MainFileID around in function calls that also pass either Preprocessor or
ASTContext. This resulted in some nice cleanups in the ASTConsumers and the
driver.
llvm-svn: 45228
contents of the header map. Look ma, no assumptions about input data
here (aka, corrupt header maps can't crash the compiler - crazy thought).
llvm-svn: 45122
the internal representation. This also fixes a bug where -I foo -F foo would
not search foo as both a normal and framework include dir.
llvm-svn: 45092
NumericLiteralParser::GetFloatValue(). Upon method return, this flag has the value
true if the returned APFloat can exactly represent the number in the parsed text,
and false otherwise.
Modified the implementation of GetFloatValue() to parse literals using APFloat's
convertFromString method (which allows us to set the value of isExact).
llvm-svn: 44339
predefined macros. Previously, these were handled by the driver,
now they are handled by the preprocessor.
Some fallout of this:
1. Instead of preprocessing two buffers (the predefines, then the
main source file) we now start preprocessing the main source
file and inject the predefines as a "psuedo #include" from the
main source file.
2. #1 allows us to nuke the Lexer::IsMainFile flag and simplify
Preprocessor::isInPrimaryFile.
3. The driver doesn't have to know about standard #defines, the
preprocessor knows, which is nice for people wanting to define
their own drivers.
4. This allows us to put normal tokens in the predefine buffer,
for example a definition for __builtin_va_list that is
target-specific, and a typedef for id in objc.
llvm-svn: 42818
stuff like this:
// If we don't have a comma, it is either the end of the list (a ';') or
// an error, bail out.
if (Tok.isNot(tok::comma))
break;
instead of:
// If we don't have a comma, it is either the end of the list (a ';') or
// an error, bail out.
if (Tok.getKind() != tok::comma)
break;
There is obviously no functionality change, but the code reads a bit better and is
more terse.
llvm-svn: 42795
Now instead of IdentifierInfo knowing anything about MacroInfo,
only the preprocessor knows. This makes MacroInfo truly private
to the Lex library (and its direct clients) instead of being
accessed in the Basic library.
llvm-svn: 42727
(because all bitfields now fit in 32 bits). This shrinks the identifier
table for carbon.h from 1634428 to 1451424 bytes (12%) and has no impact
on compile time.
llvm-svn: 42723
moves the MacroInfo pointer to a side hash table (which currently
lives in IdentifierTable.cpp). This removes a pointer from
Identifier info, but doesn't shrink it, as it requires a new bit
be added. This strange approach with the 'hasmacro' bit is needed
to not lose preprocessor performance.
llvm-svn: 42722
fit in 32-bits, shrinking IdentifierInfo by a word.
This shrinks the total size of the identifier pool from
1817264 to 1634428 bytes (11%) on carbon.h.
llvm-svn: 42719
number of arguments now and does the right thing, but the nullary/unary
accessors are preserved as convenience functions. This allows us to
slightly simplify clients.
llvm-svn: 42716
Selector's instead of requiring void* to be used. I converted one use of
DenseSet<void*> over to use DenseSet<Selector> but the others should change
as well.
llvm-svn: 42645
- Add SelectorTable, which enables us to remove MultiKeywordSelector from the public header.
- Remove FoldingSet from IdentifierInfo.h and Preprocessor.h.
- Remove Parser::ObjcGetUnarySelector and Parser::ObjcGetKeywordSelector, they are subsumed by SelectorTable.
- Add MultiKeywordSelector to IdentifierInfo.cpp.
- Move a bunch of selector related methods from ParseObjC.cpp to IdentifierInfo.cpp.
- Added some comments.
llvm-svn: 42643
This motivated implementing a devious clattner inspired solution:-)
This approach uses a small value "Selector" class to point to an IdentifierInfo for the 0/1 case. For multi-keyword selectors, we instantiate a MultiKeywordSelector object (previously known as SelectorInfo). Now, the incremental cost for selectors is only 24,800 for Cocoa.h! This saves 156,592 bytes, or 86%!! The size reduction is also the result of getting rid of the AST slot, which was not strictly necessary (we will associate a selector with it's method using another table...most likely in Sema).
This change was critical to make now, before we have too many clients.
I still need to add some comments to the Selector class...will likely add later today/tomorrow.
llvm-svn: 42452
#1: It is cleaner. I never "liked" storing keyword selectors (i.e. foo:bar:baz) in the IdentifierTable.
#2: It is more space efficient. Since Cocoa keyword selectors can be quite long, this technique is space saving. For Cocoa.h, pulling the keyword selectors out saves ~180k. The cost of the SelectorInfo data is ~100k. Saves ~80k, or 43%.
#3: It results in many API simplifications. Here are some highlights:
- Removed 3 actions (ActOnKeywordMessage, ActOnUnaryMessage, & one flavor of ObjcBuildMethodDeclaration that was specific to unary messages).
- Removed 3 funky structs from DeclSpec.h (ObjcKeywordMessage, ObjcKeywordDecl, and ObjcKeywordInfo).
- Removed 2 ivars and 2 constructors from ObjCMessageExpr (fyi, this space savings has not been measured).
I am happy with the way it turned out (though it took a bit more hacking than I expected). Given the central role of selectors in ObjC, making sure this is "right" will pay dividends later.
Thanks to Chris for talking this through with me and suggesting this approach.
llvm-svn: 42395
Rationale:
We currently have a separate table to unique ObjC selectors. Since I don't need all the instance data in IdentifierInfo, I thought this would save space (and make more sense conceptually).
It turns out the cost of having duplicate entries for unary selectors (i.e. names without colons) outweighs the cost difference between the IdentifierInfo & SelectorInfo structures. Here is the data:
Two tables:
*** Selector/Identifier Stats:
# Selectors/Identifiers: 51635
Bytes allocated: 1999824
One table:
*** Identifier Table Stats:
# Identifiers: 49500
Bytes allocated: 1990316
llvm-svn: 42139
- Add SelectorInfo/SelectorTable classes, modeled after IdentifierInfo/IdentifierTable.
- Add SelectorTable instance to ASTContext, created lazily through ASTContext::getSelectorInfo().
- Add SelectorInfo slot to ObjcMethodDecl.
- Add helper function to derive a SelectorInfo from ObjcKeywordInfo.
Misc: Got the Decl stats stuff up and running again...it was missing support for ObjC AST's.
llvm-svn: 42023
2) Add support for lexing imaginary constants (a GCC extension):
t.c:5:10: warning: imaginary constants are an extension
A = 1.0iF;
^
3) Make the 'invalid suffix' diagnostic pointer more accurate:
t.c:6:10: error: invalid suffix 'qF' on floating constant
A = 1.0qF;
^
instead of:
t.c:6:10: error: invalid suffix 'qF' on floating constant
A = 1.0qF;
^
llvm-svn: 41411
memorybuffer instead of a pointer to the memorybuffer itself. This
reduces coupling and eliminates a pointer dereference on a hot path.
This speeds up -Eonly on 483.xalancbmk by 2.1%
llvm-svn: 40394
fileid/offset pair, it now contains a bit discriminating between
mapped locations and file locations. This separates the tables for
macros and files in SourceManager, and allows better separation of
concepts in the rest of the compiler. This allows us to have *many*
macro instantiations before running out of 'addressing space'.
This is also more efficient, because testing whether something is a
macro expansion is now a bit test instead of a table lookup (which
also used to require having a srcmgr around, now it doesn't).
This is fully functional, but there are several refinements and
optimizations left.
llvm-svn: 40103
specifying the start of a token and a logical (phase 3) character number,
returns a sloc representing the input character corresponding to it.
llvm-svn: 39905
out of the llvm namespace. This makes the clang namespace be a sibling of
llvm instead of being a child.
The good thing about this is that it makes many things unambiguous. The
bad things is that many things in the llvm namespace (notably data structures
like smallvector) now require an llvm:: qualifier. IMO, libsystem and libsupport
should be split out of llvm into their own namespace in the future, which will fix
this issue.
llvm-svn: 39659
Submitted by:
Reviewed by:
Fixed a bug in the parser's handling of attributes on pointer declarators.
For example, the following code was producing a syntax error...
int *__attribute(()) foo;
attrib.c:10:25: error: expected identifier or '('
int *__attribute(()) foo;
^
Changed Parser::ParseTypeQualifierListOpt to not consume the token following
an attribute declaration.
Also added LexerToken::getName() convenience method...useful when tracking
down errors like this.
llvm-svn: 39602
Submitted by:
Reviewed by:
Move string literal parsing from Sema=>LiteralSupport. This consolidates
all the quirky parsing code within the Lexer subsystem (yeah!). This
simplifies Sema and (more importantly) allows future parsers
(i.e. subclasses of Action) to benefit from this code.
llvm-svn: 39354
Submitted by:
Reviewed by:
-Converted the preprocessor to use NumericLiteralParser.
-Several minor changes to LiteralSupport interface/implementation.
-Added an error diagnostic for floating point usage in pp expr's.
llvm-svn: 39352
Submitted by:
Reviewed by:
Moved numeric literal support from SemaExpr.cpp to LiteralSupport.[h,cpp]
in Lex. This will allow it to be used by both Sema and Preprocessor (and
should be the last major refactoring of this sub-system).. Over
time, it will be reused by anyone implementing an actions module (i.e.
any subclass of llvm::clang::Action. Minor changes to IntegerLiteral in Expr.h.
More to come...
llvm-svn: 39351
whose decl objects are lazily created the first time they are referenced.
Builtin functions are described by the clang/AST/Builtins.def file, which
makes it easy to add new ones.
This is missing two important pieces:
1. Support for the rest of the gcc builtins.
2. Support for target-specific builtins (e.g. __builtin_ia32_emms).
Just adding this builtins reduces the number of implicit function definitions
by 6, reducing the # diagnostics from 550 to 544 when parsing carbon.h.
I need to add all the i386-specific ones to eliminate several hundred more.
ugh.
llvm-svn: 39327
of having a loose collection of function pointers. This also allows clients to
maintain state, and reduces the size of the Preprocessor.h interface.
llvm-svn: 39203
filenames (and also '#pragma GCC dependency' of course). Now, assuming
no cleaning is needed, we can go all the way from lexing the filename to
doing filename lookups with no mallocs. This speeds up user PP time from
0.077 to 0.075s for Cocoa.h (2.6%).
llvm-svn: 39092
Douglas Gregor