Commit Graph

11852 Commits

Author SHA1 Message Date
serge-sans-paille 16544cbe64 [iwyu] Move <cmath> out of llvm/Support/MathExtras.h
Interestingly, MathExtras.h doesn't use <cmath> declaration, so move it out of
that header and include it when needed.

No functional change intended, but there's no longer a transitive include
fromMathExtras.h to cmath.
2022-09-28 20:49:01 +02:00
Vitaly Buka 07cf1a25a3 [NFC][StackLifetime] Rename local variable
The next patch will require more generic name.
2022-09-27 23:25:39 -07:00
Vitaly Buka c1bf3df576 [NFC][StackLifetime] Remove local variable 2022-09-27 23:07:03 -07:00
Philip Reames f6d110e26f [LAA] Make getPtrStride return Option instead of overloading zero as error value [nfc]
This is purely NFC restructure in advance of a change which actually exposes zero strides.  This is mostly because I find this interface confusing each time I look at it.
2022-09-27 15:55:44 -07:00
Nikita Popov 18b7c44086 [BasicAA] Use ScopeExit to clear Visited set (NFC) 2022-09-27 10:26:30 +02:00
Florian Hahn 275bee32ad
[LoopUnroll] Forget block and loop dispositions during unrolling.
After unrolling a loop, the block and loop dispositions need to be
cleared. As we don't know which SCEVs in the loop/blocks may be
impacted, completely clear the cache. This should also fix some cases
where deleted loops remained in the LoopDispositions cache.

This fixes a verification failure surfaced by D134531.

I am planning on reviewing/updating the existing uses of
forgetLoopDispositions to check if they should be replaced by
forgetBlockAndLoopDispositions.

Reviewed By: nikic

Differential Revision: https://reviews.llvm.org/D134612
2022-09-27 08:49:04 +01:00
Sanjay Patel 222e1c73f3 [InstSimplify] don't commute constant expression operand in min/max calls
The test shows that we would fail to consistently fold the
instruction based on the max value operand. This is also
the root cause for issue #57986, but I'll add an instcombine
test + assert for that exact problem in another commit.
2022-09-26 16:01:09 -04:00
Sanjay Patel e9e994838a [InstSimplify] rearrange matching for select-of-min/max folds; NFC
This makes the code a little shorter and should be easier to extend
for a pattern like in issue #42100.
2022-09-26 15:02:40 -04:00
Sanjay Patel 096f1c4db4 [InstSimplify] remove redundant predicate check; NFC
It's still possible that there's a simpler way to specify
the conditions needed for this set of folds, but "getStrictPred"
converts >= to > for example, so there's no need to explicitly
check that.
2022-09-26 15:02:40 -04:00
Kazu Hirata e2398a4d7c [Analysis] Introduce getStaticBonusApplied (NFC)
InlineCostCallAnalyzer encourages inlining of the last call to the
static function by subtracting LastCallToStaticBonus from Cost.

This patch introduces getStaticBonusApplied to make available the
amount of LastCallToStaticBonus applied.

The intent is to allow the module inliner to determine whether
inlining a given call site is expected to reduce the caller size with
an expression like:

  IC.getCost() + IC.getStaticBonusApplied() < 0

This patch does not add a use of getStaticBonus yet.

Differential Revision: https://reviews.llvm.org/D134373
2022-09-25 23:21:40 -07:00
Sanjay Patel b0bfefb6ec [InstSimplify] fold redundant select of min/max, part 2
This extends e5d15e1162 to handle the inverse predicates
(there's probably a more elegant way to specify the preds).

These patterns correspond to the existing simplify:
max (min X, Y), X --> X
...and extra preds for (non)equality.

The tests cycle through all 10 icmp preds for each min/max
variant with 4 swapped operand patterns each (and the min/max
operands are commuted in every other test within those).

Some Alive2 examples to verify:
https://alive2.llvm.org/ce/z/XMvEKQ
https://alive2.llvm.org/ce/z/QpMChr
2022-09-25 07:06:43 -04:00
Sanjay Patel e5d15e1162 [InstSimplify] fold redundant select of min/max
This is similar to the existing simplify:
max (max X, Y), X --> max X, Y
...but the select condition can be one of
several predicates as shown in the tests.

The tests cycle through all 10 icmp preds for
each min/max variant with 4 swapped operand
patterns each (and the min/max operands are
commuted in every other test within those).

Some Alive2 examples to verify:
https://alive2.llvm.org/ce/z/lCAQm4
https://alive2.llvm.org/ce/z/kzxVXC
2022-09-24 11:34:05 -04:00
Yi Kong 32994b7357 Make MLIR model URLs cache variables
This allows us to directly use the models published on Github.

Differential Revision: https://reviews.llvm.org/D134566
2022-09-23 15:21:53 -07:00
Teresa Johnson b1926f308f Restore "[MemProf] Memprof profile matching and annotation"
This reverts commit 794b7ea960, and
thus restores commit a212d8da94, and
follow on fixes 0cd6763fa9,
e9ff53d42f, and
37c6a25e9a.

Use a hash function (BLAKE3) instead of hash_combine/hash_code which are
not guaranteed to be stable across executions.

Additionally, it adds a "REQUIRES: x86_64-linux" to the tests that have
raw profile inputs to avoid failures on big endian bots.

Reviewers: snehasish, davidxl

Subscribers: llvm-commits

Differential Revision: https://reviews.llvm.org/D128142
2022-09-23 11:38:47 -07:00
Simon Pilgrim c94cbc343e Fix gcc warning about ambiguous if-else chain
Fixes warnings introduced by D111968
2022-09-23 14:36:28 +01:00
Simon Pilgrim a6e9141505 [TTI] Add OperandValueProperties::OP_NegatedPowerOf2 enum (PR51436)
The mul by constant costmodels handle power-of-2 constants, but not negated-power-of-2, despite the backends handling both.

This patch adds the OperandValueProperties::OP_NegatedPowerOf2 enum and wires it for use for basic mul cost analysis and SLP handling.

Fixes #50778

Differential Revision: https://reviews.llvm.org/D111968
2022-09-23 14:03:18 +01:00
Nikita Popov 6c6b48434e [BasicAA] Clean up calculation of FMRB from attributes
The current implementation for call sites is pretty convoluted
when you take the underlying implementation of the used APIs
into account. We will query the call site attributes, and then
fall back to the function attributes while taking into account
operand bundles. However, getModRefBehavior() already has it's
own (more accurate) logic for combining call-site FMRB with
function FMRB.

Clean this up by extracting a function that only fetches FMRB
from attributes, which can be directly used in getModRefBehavior()
for functions, and needs to be combined with an operand-bundle
respecting fallback in the call site case.

One caveat (that makes this non-NFC) is that CallBase function
attribute lookups allow using attributes from functions with
mismatching signature. To ensure we don't regress quality, do
the same for the function FMRB fallback.
2022-09-23 12:05:35 +02:00
Teresa Johnson 794b7ea960 Revert "[MemProf] Memprof profile matching and annotation"
This reverts commit a212d8da94, and follow
on fixes 0cd6763fa9,
e9ff53d42f, and
37c6a25e9a.

After re-reading the documentation for hash_combine, I don't think this
is the appropriate hash function to use for computing the hash to use as
a stack id in the metadata, since it is not guaranteed to produce stable
values across executions. I have not hit this problem, but plan to
switch to using an MD5 hash. I am hitting an issue with one of the bots
(https://lab.llvm.org/buildbot/#/builders/171/builds/20732)
where the values produced are only the lower 32 bits of the expected
hash values, however, which I assume is related to the implementation of
hash_combine and hash_code.

I believe I fixed all of the other bot failures with the follow on fixes,
which I'll merge into the new version before reapplying.
2022-09-22 16:08:03 -07:00
Arthur Eubanks a8f1da128d [LazyCallGraph] Handle spurious ref edges when deleting a dead function
Spurious ref edges are ref edges that still exist in the call graph even
though the corresponding IR reference no longer exists. This can cause
issues when deleting a dead function which has a spurious ref edge
pointed at it because currently we expect the dead function's RefSCC to
be trivial.

In the case that the dead function's RefSCC is not trivial, remove all
ref edges from other nodes in the RefSCC to it.

Removing a ref edge can result in splitting RefSCCs. There's actually no
reason to revisit those RefSCCs because currently we only run passes on
SCCs, and we've already added all SCCs in the RefSCC to the worklist.
(as opposed to removing the ref edge in
updateCGAndAnalysisManagerForPass() which can modify the call graph of
SCCs we have not visited yet). We also don't expect that RefSCC
refinement will allow us to glean any more information for optimization
use. Also, doing so would drastically increase the complexity of
LazyCallGraph::removeDeadFunction(), requiring us to return a list of
invalidated RefSCCs and new RefSCCs to add to the worklist.

Fixes #56503

Reviewed By: asbirlea

Differential Revision: https://reviews.llvm.org/D133907
2022-09-22 15:01:15 -07:00
Teresa Johnson 37c6a25e9a [MemProf] Fix buildbot error due to hasValue deprecation warning
Use has_value instead of hasValue to address a deprecation warning from
a212d8da94. E.g.:

https://lab.llvm.org/buildbot/#/builders/57/builds/22166/steps/5/logs/stdio
2022-09-22 13:26:53 -07:00
Teresa Johnson a212d8da94 [MemProf] Memprof profile matching and annotation
Profile matching and IR annotation for memprof profiles.

See also related RFCs:
RFC: Sanitizer-based Heap Profiler [1]
RFC: A binary serialization format for MemProf [2]
RFC: IR metadata format for MemProf [3]*

* Note that the IR metadata format has changed from the RFC during
implementation, as described in the preceeding patch adding the basic
metadata and verification support.

The matching is performed during the normal PGO annotation phase, to
ensure that the inlines applied in the IR at that point are a subset
of the inlines in the profiled binary and thus reflected in the
profile's call stacks. This is important because the call frames are
associated with functions in the profile based on the inlining in the
symbolized call stacks, and this simplifies locating the subset of
profile data relevant for matching onto each function's IR.

The PGOInstrumentationUse pass is enhanced to perform matching for
whatever combination of memprof and regular PGO profile data exists in
the profile.

Using the utilities introduced in D128854:
The memprof profile data for each context is converted to "cold" or
"notcold" based on parameterized thresholds for size, access count, and
lifetime. The memprof allocation contexts are trimmed to the minimal
amount of context required to uniquely identify whether the context is
cold or not cold. For allocations where all profiled contexts have the
same allocation type, no memprof metadata is attached and instead the
allocation call is directly annotated with an attribute specifying the
alloction type. This is the same attributed that will be applied to
allocation calls once cloned for different contexts, and later used
during LibCall simplification to emit allocation hints [4].

Depends on D128141 and D128854.

[1] https://lists.llvm.org/pipermail/llvm-dev/2020-June/142744.html
[2] https://lists.llvm.org/pipermail/llvm-dev/2021-September/153007.html
[3] https://discourse.llvm.org/t/rfc-ir-metadata-format-for-memprof/59165
[4] ab87cf382d

Differential Revision: https://reviews.llvm.org/D128142
2022-09-22 12:48:31 -07:00
Nikita Popov 60990d9042 [BasicAA] Move experimental.guard modelling to getModRefBehavior()
While we can't express this with attributes yet, we can model
these intrinsics as readonly + writing inaccessible memory (for
the control dependence) in FMRB. This way we don't need to
special-case them in getModRefInfo(), it falls out of the usual
logic.
2022-09-22 16:48:51 +02:00
Nikita Popov ab25ea6d35 [AA] Model operand bundles more precisely
Based on D130896, we can model operand bundles more precisely. In
addition to the baseline ModRefBehavior, a reading/clobbering operand
bundle may also read/write all locations. For example, a memcpy with
deopt bundle can read any memory, but only write argument memory.

This means that getModRefInfo() for memcpy with a pointer that does
not alias the arguments results in Ref, rather than ModRef, without
the need to implement any special handling.

Differential Revision: https://reviews.llvm.org/D130980
2022-09-22 11:15:20 +02:00
Nikita Popov 41dde5d858 [InstSimplify] Support vectors in simplifyWithOpReplaced()
We can handle vectors inside simplifyWithOpReplaced(), as long as
cross-lane operations are excluded. The equality can hold (or not
hold) for each vector lane independently, so we shouldn't use the
replacement value from other lanes.

I believe the only operations relevant here are shufflevector (where
all previous bugs were seen) and calls (which might use shuffle-like
intrinsics and would require more careful classification).

Differential Revision: https://reviews.llvm.org/D134348
2022-09-22 10:45:42 +02:00
Philip Reames 8c46881a53 [TTI] Recognize fp constants in getOperandInfo
We were recognizing vectors of floats, but not scalars.  That's a tad odd.
2022-09-21 14:34:34 -07:00
Arthur Eubanks f77342693b [CGSCC] Properly handle invalidating analyses for invalidated SCCs
Currently if we mark an SCC as invalid, if we haven't set UR.UpdatedC, we won't propagate the PreservedAnalyses up to the parent pass (adaptor/pass manager).

In the provided test case, we inline the function into itself then delete it as it has no users. The SCC is marked as invalid without providing a replacement UR.UpdatedC. Then the CGSCC pass manager and adaptor discard the PreservedAnalyses. Instead, handle PreservedAnalyses first before bailing due to the invalid SCC.

Fixes crashes due to out of date analyses.
2022-09-21 09:50:00 -07:00
Kazu Hirata 0a0ccc85bb [ModuleInliner] Factor out common code in InlineOrder.cpp (NFC)
This patch factors out common code in InlineOrder.cpp.

Without this patch, the model is to ask classes like SizePriority and
CostPriority to compare a pair of call sites:

  bool hasLowerPriority(const CallBase *L, const CallBase *R) const override {

while these priority classes have their own caches of priorities:

  DenseMap<const CallBase *, PriorityT> Priorities;

This model results in a lot of duplicate code like hasLowerPriority
and updateAndCheckDecreased.

This patch changes the model so that priority classes just have two
methods to compute a priority for a given call site and to compare two
previously computed priorities (as opposed to call sites).

Facilities like hasLowerPriority and updateAndCheckDecreased move to
PriorityInlineOrder along with the map from call sites to their
priorities.  PriorityInlineOrder becomes a template class so that it
can accommodate different priority classes.

Differential Revision: https://reviews.llvm.org/D134149
2022-09-21 08:50:30 -07:00
Graham Hunter 3c74ed9ee3 [LAA] Fix ICE with scAddExpr in forked pointers
The IR from https://github.com/llvm/llvm-project/issues/57368 results
in an assert firing when trying to create a runtime check for the
forked pointer. One of the forks is fine since it's loop invariant,
but the other is a scAddExpr (containing a scAddRecExpr, so not
invariant) when RtCheck::insert expects a scAddRecExpr.

This is a simple fix to just avoid forks which aren't AddRec or
loop invariant. We can allow it as a forked pointer later with
more work.

Reviewed By: fhahn

Differential Revision: https://reviews.llvm.org/D133020
2022-09-21 10:27:06 +01:00
Nikita Popov 17994ed919 [MemorySSA] Remove PerformedPhiTranslation flag
I believe this is no longer necessary, as the underlying problem
has been fixed in a different way: Nowadays, we will adjust the
location size to beforeOrAfterPointer() if the pointer is not loop
invariant. This makes merging results translated across loop
backedges safe.

The two tests in phi-translation.ll show an improvement while still
being correct: The loads in the loop no longer alias with noalias
pointers, but still alias with the store in the entry block (which
they originally did not -- this is the bug that
PerformedPhiTranslation originally fixed).

Differential Revision: https://reviews.llvm.org/D133404
2022-09-21 10:32:09 +02:00
Matt Arsenault 1a18fe65d3 Analysis: Remove redundant assertion
This assert guards the same assertion inside getTypeStoreSizeInBits
2022-09-20 09:39:45 -04:00
Matt Arsenault 1e1aefbf70 Analysis: Pass AssumptionCache through isKnownNonZero
Pass this through now that isDereferenceableAndAlignedPointer has
access to this.
2022-09-20 09:25:18 -04:00
Mateusz Mikuła 4f30c5808a [TargetLibraryInfo] Mark memrchr as unavailable on Windows
Otherwise LLVM will optimise strrchr into memrchr on Windows resulting in linker error:
```
$ cat memrchr_test.c
int main(int argc, char **argv) {
    return (long)strrchr("KkMm", argv[argc-1][0]);
}

$ clang memrchr_test.c -O
memrchr_test.c:3:12: warning: cast to smaller integer type 'long' from 'char *' [-Wpointer-to-int-cast]
    return (long)strrchr("KkMm", argv[argc-1][0]);
           ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 warning generated.
ld.lld: error: undefined symbol: memrchr
>>> referenced by D:/msys64/tmp/memrchr_test-e7aabd.o:(main)
clang: error: linker command failed with exit code 1 (use -v to see invocation)
```

Example taken from MSYS2 Discord and tested with windows-gnu target.

Reviewed By: aeubanks

Differential Revision: https://reviews.llvm.org/D134134
2022-09-20 10:50:31 +03:00
luxufan bfd31c6f12 [MemorySSA][NFC] Use const whenever possible
Differential Revision: https://reviews.llvm.org/D134162
2022-09-20 02:21:02 +00:00
Matt Arsenault 2adae8e1b7 VectorCombine: Pass through AssumptionCache 2022-09-19 19:25:22 -04:00
Matt Arsenault ce44357216 Analysis: Add AssumptionCache to isSafeToSpeculativelyExecute
Does not update any of the uses.
2022-09-19 19:25:22 -04:00
Matt Arsenault 0d8ffcc532 Analysis: Add AssumptionCache argument to isDereferenceableAndAlignedPointer
This does not try to pass it through from the end users.
2022-09-19 18:57:33 -04:00
Nikita Popov 36f325413e [SCEV] Don't verify dispositions of invalid loops
This should fix the expensive checks build. Ideally we would not
have invalid loops in LoopDispositions.
2022-09-19 15:07:44 +02:00
Max Kazantsev bb68b2402d [SCEV] Verify contents of loop disposition cache
It seems that it is sometimes broken. Initial motivation for this was
investigation of https://github.com/llvm/llvm-project/issues/56260, but
it also seems that we have found an unrelated bug in LoopFusion that leaves
broken caches.

Differential Revision: https://reviews.llvm.org/D134158
Reviewed By: nikic
2022-09-19 17:43:00 +07:00
Max Kazantsev 818b1ab84e [SCEV][NFC] Remove unused parameter from forgetLoopDispositions
Let's be honest about it, we don't drop loop dispositions for
particular loops. Remove the parameter that misleadingly makes
it apparent that we do.
2022-09-19 14:06:42 +07:00
Kazu Hirata 71b12030b9 [ModuleInliner] Capitalize a variable name (NFC) 2022-09-18 14:35:09 -07:00
Kazu Hirata 82293ed486 [ModuleInliner] Remove unused using declarations (NFC) 2022-09-18 14:27:06 -07:00
Kazu Hirata 00d982699b [ModuleInliner] Move getInlineCostWrapper to an anonymous namespace (NFC)
This patch moves getInlineCostWrapper to an anonymous namespace.
While I am at it, I'm moving the function closer to the beginning of
the file so that I can use it elsewhere in the file without a forward
declaration.
2022-09-18 14:09:21 -07:00
Kazu Hirata d3b95ecc98 [ModuleInliner] Remove InlineOrder::front (NFC)
InlineOrder::front is a remnant from the era when we had a nested
"while" loops in the module inliner, with the inner one grouping the
call sites with the same caller.

Now that we have a simple "while" loop draining the priority queue, we
can just use InlineOrder::pop.

Differential Revision: https://reviews.llvm.org/D134121
2022-09-18 08:49:44 -07:00
Kazu Hirata cf355bf36e [Analysis] Introduce isSoleCallToLocalFunction (NFC)
We check to see if a given CallBase is a sole call to a local function
at multiple places in InlineCost.cpp.  This patch factors out the
common code.

Differential Revision: https://reviews.llvm.org/D134114
2022-09-17 20:59:54 -07:00
Kazu Hirata 20d764aff0 [llvm] Don't including SetVector.h (NFC)
llvm/lib/ProfileData/RawMemProfReader.cpp uses SetVector without
including SetVector.h, so this patch adds an appropriate #include
there.
2022-09-17 12:36:43 -07:00
Kazu Hirata 5faf4bf195 [ModuleInliner] Move UseInlinePriority to InlineOrder.cpp (NFC)
UseInlinePriority specifies the priority function.  This patch
simplifies the code by moving UseInlinePriority closer to the actual
consumer -- the switch statement inside getInlineOrder.

Differential Revision: https://reviews.llvm.org/D134100
2022-09-17 11:41:28 -07:00
Kazu Hirata 6e30a9cc08 [Inliner] Retire DefaultInlineOrder (NFC)
DefaultInlineOrder was largely an exercise in generalizing the
traversal order of call sites within the inliner.

Now that the module inliner is starting to form its shape, there is no
point in sharing DefaultInlineOrder between the module inliner and the
CGSCC inliner.  DefaultInlineOrder and all the other inline orders are
mutually exclusive in the following sense:

- The use of DefaultInlineOrder doesn't make sense in the module
  inliner because there is no priority inherent in the order in which
  call sites are added to the list of call sites -- SmallVector.

- The use of any other inline order doesn't make sense in the CGSCC
  inliner because little prioritization can be done within one CGSCC.

This patch essentially reverts the addition of DefaultInlineOrder so
that the loop structure of Inliner.cpp looks like the state just
before we started working on the module inliner (circa June 2021).

At the same time, ww remove the choice of DefaultInlineOrder from
UseInlinePriority.

Differential Revision: https://reviews.llvm.org/D134080
2022-09-16 15:36:40 -07:00
Kazu Hirata e0bc76eb23 [ModuleInliner] Move InlinePriority and its derived classes to InlineOrder.cpp (NFC)
These classes are referred to only from getInlineOrder in
InlineOrder.cpp.  This patch hides the entire class declarations and
definitions in InlineOrder.cpp.

Differential Revision: https://reviews.llvm.org/D134056
2022-09-16 12:32:16 -07:00
Nikita Popov b1cd393f9e [AA] Tracking per-location ModRef info in FunctionModRefBehavior (NFCI)
Currently, FunctionModRefBehavior tracks whether the function reads
or writes memory (ModRefInfo) and which locations it can access
(argmem, inaccessiblemem and other). This patch changes it to track
ModRef information per-location instead.

To give two examples of why this is useful:

* D117095 highlights a weakness of ModRef modelling in the presence
  of operand bundles. For a memcpy call with deopt operand bundle,
  we want to say that it can read any memory, but only write argument
  memory. This would allow them to be treated like any other calls.
  However, we currently can't express this and have to say that it
  can read or write any memory.
* D127383 would ideally be modelled as a separate threadid location,
  where threadid Refs outside pre-split coroutines can be ignored
  (like other accesses to constant memory). The current representation
  does not allow modelling this precisely.

The patch as implemented is intended to be NFC, but there are some
obvious opportunities for improvements and simplification. To fully
capitalize on this we would also want to change the way we represent
memory attributes on functions, but that's a larger change, and I
think it makes sense to separate out the FunctionModRefBehavior
refactoring.

Differential Revision: https://reviews.llvm.org/D130896
2022-09-14 16:34:41 +02:00
Nikita Popov 1cfbbba15b [AA] Remove unnecessary intersections from getModRefBehavior() (NFC)
Intersection with other providers is performed by AAResults. Doing
this here is both pointless and confusing.
2022-09-14 14:26:39 +02:00
Nikita Popov 31cc0ab321 [BasicAA] Delay getAllocTypeSize() call (NFC)
This call is expensive, so don't perform it for zero indices.

Also rename the variable to use Alloc rather than Alloca, this
doesn't have anything to do with allocas in particular.
2022-09-13 10:24:50 +02:00
Matthias Gehre c1502425ba Move TargetTransformInfo::maxLegalDivRemBitWidth -> TargetLowering::maxSupportedDivRemBitWidth
Also remove new-pass-manager version of ExpandLargeDivRem because there is no way
yet to access TargetLowering in the new pass manager.

Differential Revision: https://reviews.llvm.org/D133691
2022-09-12 17:06:16 +01:00
zhongyunde 8a15695be2 [AA] Improve the BasicAA analysis capability
According https://discourse.llvm.org/t/memoryssa-does-the-accessedbetween-support-scalable-vector-pointer/65052,
scalable vector support in BasicAA is currently essentially limited,
and should be improved effectively for a constant offset GEP if the scalable index is zero, eg:
  getelementptr <vscale x 4 x i32>, ptr %p, i64 0, i64 %i

Reviewed By: nikic
Differential Revision: https://reviews.llvm.org/D133567
2022-09-12 19:41:17 +08:00
Junduo Dong 6975ab7126 [Clang] Reimplement time tracing of NewPassManager by PassInstrumentation framework
The previous implementation of time tracing in NewPassManager is direct but messive.

The key codes are like the demo below:
```
  /// Runs the function pass across every function in the module.
  PreservedAnalyses run(LazyCallGraph::SCC &C, CGSCCAnalysisManager &AM,
                        LazyCallGraph &CG, CGSCCUpdateResult &UR) {
      /// ...
      PreservedAnalyses PassPA;
      {
        TimeTraceScope TimeScope(Pass.name());
        PassPA = Pass.run(F, FAM);
      }
      /// ...
 }
```

It can be bothered to judge where should we add the tracing codes by hands.

With the PassInstrumentation framework, we can easily add `Before/After` callback
functions to add time tracing codes.

Differential Revision: https://reviews.llvm.org/D131960
2022-09-11 05:42:55 -07:00
Aiden Grossman ec83c7e358 [MLGO] Make TFLiteUtils throw an error if some features haven't been passed to the model
In the Tensorflow C lib utilities, an error gets thrown if some features
haven't gotten passed into the model (due to differences in ordering
which now don't exist with the transition to TFLite). However, this is
not currently the case when using TFLiteUtils. This patch makes some
minor changes to throw an error when not all inputs of the model have
been passed, which when not handled will result in a seg fault within
TFLite.

Reviewed By: mtrofin

Differential Revision: https://reviews.llvm.org/D133451
2022-09-10 22:59:03 +00:00
Nikita Popov a9f312c7f4 [AST] Use BatchAA in aliasesUnknownInst() (NFCI) 2022-09-09 15:54:48 +02:00
Mircea Trofin a219a8a822 [mlgo][nfc] Set logging level to warning or higher for TFLite 2022-09-08 12:10:56 -07:00
Nikita Popov 98a3a340c3 [ConstantExpr] Don't create fneg expressions
Don't create fneg expressions unless explicitly requested by IR or
bitcode.
2022-09-07 11:27:25 +02:00
Matthias Gehre 2090e85fee [llvm/CodeGen] Enable the ExpandLargeDivRem pass for X86, Arm and AArch64
This adds the ExpandLargeDivRem to the default pass pipeline.
The limit at which it expands div/rem instructions is configured
via a new TargetTransformInfo hook (default: no expansion)
X86, Arm and AArch64 backends implement this hook to expand div/rem
instructions with more than 128 bits.

Differential Revision: https://reviews.llvm.org/D130076
2022-09-06 15:32:04 +01:00
Sanjay Patel a8fcb51242 [InstSimplify] allow poison/undef in constant match for "C - X ==/!= X -> false/true"
This fold was added with 5e9522c311, but over-specified.
We can assume that an undef element is an odd number:
https://alive2.llvm.org/ce/z/djQmWU
2022-09-06 08:19:30 -04:00
eopXD ea3630e8d4 [CMake][MLGO] Fix cmake for MLGO
The if-statement should check whehter TFLITE is on or not rather than if the variable is specified.

Reviewed By: mtrofin

Differential Revision: https://reviews.llvm.org/D132902
2022-09-06 00:32:08 -07:00
Arthur Eubanks 7e3aa8f01a Revert "[LoopPassManager] Implement and use LoopNestAnalysis::run() instead of manually creating LoopNests"
This reverts commit 57fd866551.

Causes crashes, see comments in D132581.
2022-09-05 15:42:48 -07:00
LiaoChunyu 456c7ef68e [InstSimplify][NFC] shortened the code 2022-09-05 23:57:53 +08:00
LiaoChunyu 5e9522c311 [InstSimplify] Odd - X ==/!= X -> false/true
Reviewed By: spatel

Differential Revision: https://reviews.llvm.org/D132989
2022-09-05 23:51:45 +08:00
Kazu Hirata 3850edd9e0 Use llvm::count_if (NFC) 2022-09-03 11:17:35 -07:00
Simon Pilgrim e2d140e9c3 [TTI] Add isExpensiveToSpeculativelyExecute wrapper
CGP uses a raw `getInstructionCost(I, TargetTransformInfo::TCK_SizeAndLatency) >= TCC_Expensive` check to see if its better to move an expensive instruction used in a select behind a branch instead.

This is causing issues with upcoming improvements to TCK_SizeAndLatency costs on X86 as we need to use TCK_SizeAndLatency as an uop count (so its compatible with various target-specific buffer sizes - see D132288), but we can have instructions that have a low TCK_SizeAndLatency value but should still be treated as 'expensive' (FDIV for example) - by adding a isExpensiveToSpeculativelyExecute wrapper we can keep the current behaviour but still add an x86 override in a future patch when the cost tables are updated to compensate.
2022-09-03 13:12:22 +01:00
Arthur Eubanks 57fd866551 [LoopPassManager] Implement and use LoopNestAnalysis::run() instead of manually creating LoopNests
The current code is basically just emulating what the analysis manager does.

Reviewed By: asbirlea

Differential Revision: https://reviews.llvm.org/D132581
2022-09-02 10:55:53 -07:00
Rong Xu 0caa4a9559 [PGO] Support PGO annotation of CallBrInst
We currently instrument CallBrInst but do not annotate it with
the branch weight. This patch enables PGO annotation of CallBrInst.

Differential Revision: https://reviews.llvm.org/D133040
2022-09-01 14:13:50 -07:00
Nikita Popov 4f046bc8e0 [PHITranslateAddr] Require dominance when searching for translated address (PR57025)
This is a fix for PR57025 and an alternative to D131776. The problem
in the phi-translation-to-wrong-context.ll test case is that phi
translation of %gep.j into if2 pick %gep.i as the result. While this
instruction has the correct pointer address, it occurs in a context
where %i != 0. As such, we get a NoAlias result for the store in
if2, even though they do alias for %i == 0 (which is legal in the
original context of the pointer).

PHITranslateValue already has a MustDominate option, which can be
used to restrict PHI translation results to values that dominate the
translated-into block. However, this is more aggressive than what we
need and would significantly regress GVN results. In particular, if
we have a pointer value that does not require any translation, then
it is fine to continue using that value in the predecessor, because
the context is still correct for the original query. We only run into
problems if PHITranslateSubExpr() picks a completely random
instruction in a context that may have preconditions that do not hold.

Fix this by always performing the dominance checks in
PHITranslateSubExpr(), without enabling the more general MustDominate
requirement.

Fixes https://github.com/llvm/llvm-project/issues/57025. This also
fixes the test case for https://github.com/llvm/llvm-project/issues/30999,
but I'm not sure whether that's just the particular test case,
or a general solution to the problem.

Differential Revision: https://reviews.llvm.org/D132935
2022-09-01 16:26:42 +02:00
Pavel Samolysov 88581db62f [LazyCallGraph] Reformat the code in accordance with the code style. NFC
Also, some local variables were renamed in accordance with the code
style as well as `std::tie` occurrences and `.first`/`.second` member
uses were replaced with structure bindings.

Differential Revision: https://reviews.llvm.org/D132806
2022-08-30 11:06:42 +03:00
Kazu Hirata 6ed2cb4ad5 Revert "[llvm] Use llvm::is_contained (NFC)"
This reverts commit ebf574f59a.

This patch seems to cause build failures on Windows.
2022-08-28 18:52:49 -07:00
Kazu Hirata ebf574f59a [llvm] Use llvm::is_contained (NFC) 2022-08-28 17:35:03 -07:00
Kazu Hirata ce9f007c7c [llvm] Use llvm::find_if (NFC) 2022-08-28 10:41:48 -07:00
Kazu Hirata 7a617fdf39 Use std::gcd (NFC)
This patch replaces calls to GreatestCommonDivisor64 with std::gcd
where both arguments are known to be of unsigned types no larger than
64 bits in size.
2022-08-27 21:20:59 -07:00
Arthur Eubanks 7a94d189ad [LazyCallGraph] Update libcall list when replacing a libcall node's function
Otherwise when we visit all libcalls in
updateCGAndAnalysisManagerForPass(), the old libcall is dead and doesn't
have a node.

We treat libcalls conservatively in LazyCallGraph because any function
may introduce calls to them out of thin air.

It is weird to change the signature of a libcall since introducing calls
to the libcall with a different signature may break, but other passes
like deadargelim already do it, so let's preserve this behavior for now.

Fixes an issue found in D128830.

Reviewed By: psamolysov

Differential Revision: https://reviews.llvm.org/D132764
2022-08-27 10:57:53 -07:00
Mircea Trofin 3546b5c520 [mlgo] Fix flaky test
The source of the flakyness is internal uninitialized buffers due to
a dangling variable in the model.
2022-08-26 21:29:25 -07:00
Florian Hahn 9405af1c85
[LAA] Require AddRecs to be in the innermost loop for diff-checks.
The simpler diff-checks require pointers with add-recs from the same
innermost loop, but this property wasn't check completely. Add the
missing check to ensure both addrecs are in the innermost loop.

Fixes #57315.
2022-08-26 20:39:52 +01:00
Philip Reames 86b67a310d [LAA] Prune dependencies with distance large than access implied by trip count
When we have a dependency with a dependence distance which can only be hit on an iteration beyond the actual trip count of the loop, we can ignore that dependency when analyzing said loop. We already had this code, but had restricted it solely to unknown dependence distances. This change applies it to all dependence distances.

Without this code, we relied on the vectorizer reducing VF such that our infeasible dependence was respected. This usually worked out to about the same result, but not always. For fixed length vectorization, this could mean a smaller VF than optimal being chosen or additional runtime checks. For scalable vectorization - where the bounds on access implied by VF are broader - we could often not find a feasible VF at all.

Differential Revision: https://reviews.llvm.org/D131924
2022-08-25 14:24:13 -07:00
Sanjay Patel 4e44c22c97 [ValueTracking][InstCombine] restrict FP min/max matching to avoid miscompile
This is a long-standing FIXME with a non-FMF test that exposes
the bug as shown in issue #57357.

It's possible that there's still a way to miscompile by
mis-identifying/mis-folding FP min/max patterns, but
this patch only exposes a couple of seemingly minor
regressions while preventing the broken transform.
2022-08-25 16:52:40 -04:00
Florian Hahn c035efc814
[LAA] Cache PSE.getSE() in variable (NFC).
Preparation for follow-up patches will introduce additional uses
of SE.
2022-08-25 21:40:22 +01:00
Mircea Trofin b2b460b0a0 [mlgo] Fix tests
Missed a few tests in D119507
2022-08-24 17:31:40 -07:00
Mircea Trofin 5ce4c9aa04 [mlgo] Use TFLite for 'development' mode.
TLite is a lightweight, statically linkable[1], model evaluator, supporting a
subset of what the full tensorflow library does, sufficient for the
types of scenarios we envision having. It is also faster.

We still use saved models as "source of truth" - 'release' mode's AOT
starts from a saved model; and the ML training side operates in terms of
saved models.

Using TFLite solves the following problems compared to using the full TF
C API:

- a compiler-friendly implementation for runtime-loadable (as opposed
  to AOT-embedded) models: it's statically linked; it can be built via
  cmake;
- solves an issue we had when building the compiler with both AOT and
  full TF C API support, whereby, due to a packaging issue on the TF
  side, we needed to have the pip package and the TF C API library at
  the same version. We have no such constraints now.

The main liability is it supporting a subset of what the full TF
framework does. We do not expect that to cause an issue, but should that
be the case, we can always revert back to using the full framework
(after also figuring out a way to address the problems that motivated
the move to TFLite).

Details:

This change switches the development mode to TFLite. Models are still
expected to be placed in a directory - i.e. the parameters to clang
don't change; what changes is the directory content: we still need
an `output_spec.json` file; but instead of the saved_model protobuf and
the `variables` directory, we now just have one file, `model.tflite`.

The change includes a utility showing how to take a saved model and
convert it to TFLite, which it uses for testing.

The full TF implementation can still be built (not side-by-side). We
intend to remove it shortly, after patching downstream dependencies. The
build behavior, however, prioritizes TFLite - i.e. trying to enable both
full TF C API and TFLite will just pick TFLite.

[1] thanks to @petrhosek's changes to TFLite's cmake support and its deps!
2022-08-24 16:07:24 -07:00
Jakub Kuderski 6fa87ec10f [ADT] Deprecate is_splat and replace all uses with all_equal
See the discussion thread for more details:
https://discourse.llvm.org/t/adt-is-splat-and-empty-ranges/64692

Reviewed By: dblaikie

Differential Revision: https://reviews.llvm.org/D132335
2022-08-23 11:36:27 -04:00
Philip Reames c9608d57b8 [TTI] Plumb through OperandValueInfo in getMemoryOpCost [NFC]
This has the effect of exposing the power-of-two property for use in memory op costing, but no target actually uses it yet.  The main point of this change is simple consistency with the recently changes getArithmeticInstrCost, and to remove the last (interface) use of OperandValueKind.
2022-08-23 07:55:42 -07:00
Aditya Kumar 0af3ab02fd [NFC] LoopAccess: Move expressions close to usage
Avoids useless evaluation of these expressions.

Reviewed By: michaelmaitland, fhahn

Differential Revision: https://reviews.llvm.org/D132337
2022-08-23 07:08:42 -07:00
liqinweng 9181ab9223 [NFC]] Use llvm::all_of instead of std::all_of
Reviewed By: kazu

Differential Revision: https://reviews.llvm.org/D131886
2022-08-23 12:21:53 +08:00
Philip Reames 104fa367ee [TTI] Use OperandValueInfo in getArithmeticInstrCost implementation [NFC]
This change completes the process of replacing OperandValueKind and OperandValueProperties which were previously passed independently in this API with a single container class which contains both.

This is the change which motivated the whole sequence which preceeded it.  In an original spike version of this change, I'd noticed a nasty bug: I'd changed the signature without changing names, and as result, we silently passed additional information through a callsite which previously dropped the power-of-two fact.  This might be harmless in most cases, but at least a couple clearly dependend for correctness on not passing that property through.

I did my best to split off prior changes which reduced the scope of this one, and which made it possible to use compiler assistance.  For instance, every parameter which changes type in this change also changes name.  This was intentional to make sure that every call site possible effected must show up in the diff.  This let me audit each one closely.
2022-08-22 15:16:39 -07:00
Philip Reames 27d3321c4f [TTI] Use OperandValueInfo in getMemoryOpCost client api [nfc]
This removes the last use of OperandValueKind from the client side API, and (once this is fully plumbed through TTI implementation) allow use of the same properties in store costing as arithmetic costing.
2022-08-22 11:26:31 -07:00
Philip Reames 274f86e7a6 [TTI] Remove OperandValueKind/Properties from getArithmeticInstrCost interface [nfc]
This completes the client side transition to the OperandValueInfo version of this routine.  Backend TTI implementations still use the prior versions for now.
2022-08-22 11:06:32 -07:00
Philip Reames c42a5f1cc2 [TTI] Migrate getOperandInfo to OperandVaueInfo [nfc]
This is part of merging OperandValueKind and OperandValueProperties.
2022-08-22 10:19:02 -07:00
Philip Reames 5cd427106d [TTI] Start process of merging OperandValueKind and OperandValueProperties [nfc]
OperandValueKind and OperandValueProperties both provide facts about the operands of an instruction for purposes of cost modeling.  We've discussed merging them several times; before I plumb through more flags, let's go ahead and do so.

This change only adds the client side interface for getArithmeticInstrCost and makes a couple of minor changes in client code to prove that it works.  Target TTI implementations still use the split flags.  I'm deliberately splitting what could be one big change into a series of smaller ones so that I can lean on the compiler to catch errors along the way.
2022-08-22 09:48:15 -07:00
Max Kazantsev e587199a50 [SCEV] Prove condition invariance via context, try 2
Initial implementation had too weak requirements to positive/negative
range crossings. Not crossing zero with nuw is not enough for two reasons:

- If ArLHS has negative step, it may turn from positive to negative
  without crossing 0 boundary from left to right (and crossing right to
  left doesn't count for unsigned);
- If ArLHS crosses SINT_MAX boundary, it still turns from positive to
  negative;

In fact we require that ArLHS always stays non-negative or negative,
which an be enforced by the following set of preconditions:

- both nuw and nsw;
- positive step (looks liftable);

Because of positive step, boundary crossing is only possible from left
part to the right part. And because of no-wrap flags, it is guaranteed
to never happen.
2022-08-22 14:31:19 +07:00
Ting Wang d2d77e050b [PowerPC][Coroutines] Add tail-call check with call information for coroutines
Fixes #56679.

Reviewed By: ChuanqiXu, shchenz

Differential Revision: https://reviews.llvm.org/D131953
2022-08-21 22:20:40 -04:00
Simon Pilgrim 5263155d5b [CostModel] Add CostKind argument to getShuffleCost
Defaults to TCK_RecipThroughput - as most explicit calls were assuming TCK_RecipThroughput (vectorizers) or was just doing a before-vs-after comparison (vectorcombiner). Calls via getInstructionCost were just dropping the CostKind, so again there should be no change at this time (as getShuffleCost and its expansions don't use CostKind yet) - but it will make it easier for us to better account for size/latency shuffle costs in inline/unroll passes in the future.

Differential Revision: https://reviews.llvm.org/D132287
2022-08-21 10:54:51 +01:00
Kazu Hirata 258531b7ac Remove redundant initialization of Optional (NFC) 2022-08-20 21:18:28 -07:00
Sanjay Patel 2981a94902 [EarlyCSE][ConstantFolding] do not constant fold atan2(+/-0.0, +/-0.0), part 2
Follow-up to 7f1262a322. That patch avoided removing the
call, but it still allowed the constant-folded result. This
makes the behavior consistent with 1-arg libm folding: if the
call potentially raises an exception, then we just bail out.

It seems likely that there are other corner-cases like this,
but the tests are incomplete, so we have lived with these
discrepancies for a long time. This was untested before the
the constant folding was expanded in D127964.
2022-08-20 10:16:06 -04:00
Philip Reames b0a2c48e9f [tti] Consolidate getOperandInfo without OperandValueProperties copies [nfc] 2022-08-19 16:22:22 -07:00
Sanjay Patel 7f1262a322 [EarlyCSE][ConstantFolding] do not constant fold atan2(+/-0.0, +/-0.0)
These may raise an error (set errno) as discussed in the post-commit
comments for D127964, so we can't fold away the call and potentially
alter that behavior.
2022-08-19 12:27:29 -04:00
Alexey Bataev d53e245951 [COST][NFC]Introduce OperandValueKind in getMemoryOpCost, NFC.
Added OperandValueKind OpdInfo parameter to getMemoryOpCost functions to
better estimate cost with immediate values.

Part of D126885.
2022-08-19 07:33:00 -07:00
Max Kazantsev f798c042f4 Revert "[SCEV] Prove condition invariance via context"
This reverts commit a3d1fb3b59.

Reverting until investigation of https://github.com/llvm/llvm-project/issues/57247
has concluded.
2022-08-19 21:02:06 +07:00