Commit Graph

421 Commits

Author SHA1 Message Date
Augie Fackler 12c0bf8ba9 tests: add attributes that would normally come from inferattrs
As my goal is to remove at least _some_ functions from the static list
in MemoryBuiltins.cpp, these tests either need to run inferattrs or
statically declare these attributes to keep passing. A couple of tests
had alternate cases which are no longer meaningful, e.g.
`malloc-load-removal.ll`.

Differential Revision: https://reviews.llvm.org/D123087
2022-07-25 17:29:00 -04:00
Johannes Doerfert dfac030271 [Intrinsics] Add `nocallback` to the memset/cpy/move intrinsics
These were forgotten when D118680 was applied. Similar to D125937.

Differential Revision: https://reviews.llvm.org/D129516
2022-07-21 22:52:46 -05:00
Alexander Shaposhnikov e9afdf838e [GlobalOpt] Enable evaluation of atomic loads
Relax the check to allow evaluation of atomic loads
(but still skip volatile loads).

Test plan:
1/ ninja check-llvm check-clang
2/ Bootstrapped LLVM/Clang pass tests

Differential revision: https://reviews.llvm.org/D130211
2022-07-21 21:36:11 +00:00
Alexander Shaposhnikov 67f1fe8597 [GlobalOpt] Enable evaluation of atomic stores
Relax the check to allow evaluation of atomic stores
(but still skip volatile stores).

Test plan:
1/ ninja check-llvm check-clang
2/ Bootstrapped LLVM/Clang pass tests

Differential revision: https://reviews.llvm.org/D129841
2022-07-20 22:33:58 +00:00
Florian Hahn 2d5d6c343b
[GlobalOpt] Add more tests with large number of stores to globals.
Add a few more test cases for D129525.
2022-07-13 11:13:21 -07:00
Florian Hahn 139378ef8e
[GlobalOpt] Add test that requires splitting up global into many.
Add test that hits the limit introduced in 4796b4ae7b.
2022-07-11 16:34:07 -07:00
Nikita Popov 11950efe06 [ConstExpr] Remove div/rem constant expressions
D128820 stopped creating div/rem constant expressions by default;
this patch removes support for them entirely.

The getUDiv(), getExactUDiv(), getSDiv(), getExactSDiv(), getURem()
and getSRem() on ConstantExpr are removed, and ConstantExpr::get()
now only accepts binary operators for which
ConstantExpr::isSupportedBinOp() returns true. Uses of these methods
may be replaced either by corresponding IRBuilder methods, or
ConstantFoldBinaryOpOperands (if a constant result is required).

On the C API side, LLVMConstUDiv, LLVMConstExactUDiv, LLVMConstSDiv,
LLVMConstExactSDiv, LLVMConstURem and LLVMConstSRem are removed and
corresponding LLVMBuild methods should be used.

Importantly, this also means that constant expressions can no longer
trap! This patch still keeps the canTrap() method to minimize diff --
I plan to drop it in a separate NFC patch.

Differential Revision: https://reviews.llvm.org/D129148
2022-07-06 10:11:34 +02:00
Nikita Popov f65c88c42f [GlobalOpt] Fix memset handling in global ctor evaluation (PR55859)
The global ctor evaluator currently handles by checking whether the
memset memory is already zero, and skips it in that case. However,
it only actually checks the first byte of the memory being set.

This patch extends the code to check all bytes being set. This is
done byte-by-byte to avoid converting undef values to zeros in
larger reads. However, the handling is still not completely correct,
because there might still be padding bytes (though probably this
doesn't matter much in practice, as I'd expect global variable
padding to be zero-initialized in practice).

Mostly fixes https://github.com/llvm/llvm-project/issues/55859.

Differential Revision: https://reviews.llvm.org/D128532
2022-06-27 16:50:49 +02:00
Nikita Popov 771c46ad29 [GlobalOpt] Add tests for memset with non-zero value (NFC) 2022-06-27 16:36:25 +02:00
Arthur Eubanks e422c0d3b2 [GlobalOpt] Perform store->dominated load forwarding for stored once globals
The initial land incorrectly optimized forwarding non-Constants in non-nosync/norecurse functions. Bail on non-Constants since norecurse should cause global -> alloca promotion anyway.

The initial land also incorrectly assumed that StoredOnceStore was the only store to the global, but it actually means that only one value other than the global initializer is stored. Add a check that there's only one store.

Compile time tracker:
https://llvm-compile-time-tracker.com/compare.php?from=c80b88ee29f34078d2149de94e27600093e6c7c0&to=ef2c2b7772424b6861a75e794f3c31b45167304a&stat=instructions

Reviewed By: nikic, asbirlea, jdoerfert

Differential Revision: https://reviews.llvm.org/D128128
2022-06-24 09:09:26 -07:00
Nikita Popov 999aa6bf7f [GlobalOpt] Add tests for PR55859 (NFC) 2022-06-24 16:06:37 +02:00
Arthur Eubanks b257acd266 [test][GlobalOpt] Update precommitted test 2022-06-23 09:56:31 -07:00
Arthur Eubanks b5db65e0da Reland [GlobalOpt] Preserve CFG analyses
The only place we modify the CFG is when calling
removeUnreachableBlocks(), so insert a callback there which invalidates
analyses for that function (or recomputes DT in the legacy PM).

We may delete functions, make sure to clear analyses for those
functions. (this was missed in the original revision)

Small compile time wins across the board:
https://llvm-compile-time-tracker.com/compare.php?from=f444ea8ce0aaaa5ec1a4129809389da15cc41396&to=698f41f4fc26cbf1006ed5d88e9d658edfc5b749&stat=instructions

Reviewed By: nikic

Differential Revision: https://reviews.llvm.org/D128145
2022-06-21 09:19:59 -07:00
Arthur Eubanks 13ff7d6f39 Revert "[GlobalOpt] Perform store->dominated load forwarding for stored once globals"
This reverts commit 6f348b146b.

Am seeing internal test failures plus a linux kernel breakage reported due to this.
2022-06-20 10:26:47 -07:00
Arthur Eubanks 6f348b146b [GlobalOpt] Perform store->dominated load forwarding for stored once globals
Compile time tracker:
https://llvm-compile-time-tracker.com/compare.php?from=1e556f459b44dd0ca4073e932f66ecb6f40fe31a&to=6d7bed4e1e72c6a8592748626091274209740a40&stat=instructions

Reviewed By: nikic

Differential Revision: https://reviews.llvm.org/D128128
2022-06-19 10:27:20 -07:00
Arthur Eubanks 1e556f459b [test][GlobalOpt] Update precommitted test 2022-06-18 21:58:16 -07:00
Arthur Eubanks 9fac606af2 [test][GlobalOpt] Regenerate some tests 2022-06-18 21:34:38 -07:00
Arthur Eubanks 213d489707 [test][GlobalOpt] Precommit more tests
And fix up existing tests to not have so much UB.
2022-06-18 19:36:48 -07:00
Arthur Eubanks 07b9937d0c [test][GlobalOpt] Precommit test 2022-06-18 15:44:27 -07:00
Fangrui Song 3d6872b09f [GlobalOpt][test] Remove br i1 undef 2022-06-10 20:36:22 -07:00
Alexander Shaposhnikov badd088c57 [GlobalOpt] Enable optimization of constructors with different priorities
Adjust `optimizeGlobalCtorsList` to handle the case of different priorities.
This addresses the issue https://github.com/llvm/llvm-project/issues/55083.

Test plan: ninja check-all

Differential revision: https://reviews.llvm.org/D125278
2022-05-13 22:19:29 +00:00
Arthur Eubanks b07aab8fc1 [GlobalOpt] Iterate over replaced values deterministically to constprop
If there are pre-existing dead instructions, the order we visit replaced
values can cause us sometimes to not delete dead instructions.

The added test non-deterministically failed without the change.
2022-05-02 09:43:20 -07:00
Arthur Eubanks 4e65291837 [OpaquePtr][GlobalOpt] Don't attempt to evaluate global constructors with arguments
Previously all entries in global_ctors had to have the void()* type and
we'd skip evaluating bitcasted functions. With opaque pointers we may
see the function directly.

Fixes #55147.

Reviewed By: #opaque-pointers, nikic

Differential Revision: https://reviews.llvm.org/D124553
2022-04-27 19:00:44 -07:00
Nikita Popov db561064f6 [GlobalOpt] Handle non-instruction MTI source (PR54572)
This was reusing a cast to GlobalVariable to check for an
Instruction, which means we'll try to dereference a null pointer
if it's not actually a GlobalVariable. We should be casting
MTI->getSource() instead.

I don't think this problem is really specific to opaque pointers,
but it certainly makes it a lot easier to reproduce.

Fixes https://github.com/llvm/llvm-project/issues/54572.
2022-03-28 14:28:47 +02:00
Fangrui Song c6692f819e [GlobalOpt] Don't replace alias with aliasee if either alias/aliasee may be preemptible
Generalize D99629 for ELF. A default visibility non-local symbol is preemptible
in a -shared link. `isInterposable` is an insufficient condition.

Moreover, a non-preemptible alias may be referenced in a sub constant expression
which intends to lower to a PC-relative relocation. Replacing the alias with a
preemptible aliasee may introduce a linker error.

Respect dso_preemptable and suppress optimization to fix the abose issues. With
the change, `alias = 345` will not be rewritten to use aliasee in a `-fpic`
compile.
```
int aliasee;
extern int alias __attribute__((alias("aliasee"), visibility("hidden")));
void foo() { alias = 345; } // intended to access the local copy
```

While here, refine the condition for the alias as well.

For some binary formats like COFF, `isInterposable` is a sufficient condition.
But I think canonicalization for the changed case has little advantage, so I
don't bother to add the `Triple(M.getTargetTriple()).isOSBinFormatELF()` or
`getPICLevel/getPIELevel` complexity.

For instrumentations, it's recommended not to create aliases that refer to
globals that have a weak linkage or is preemptible. However, the following is
supported and the IR needs to handle such cases.
```
int aliasee __attribute__((weak));
extern int alias __attribute__((alias("aliasee")));
```

There are other places where GlobalAlias isInterposable usage may need to be
fixed.

Reviewed By: rnk

Differential Revision: https://reviews.llvm.org/D107249
2022-03-18 14:17:05 -07:00
Nikita Popov 067c035012 [GlobalOpt] Handle undef global_ctors gracefully
If there are no ctors, then this can have an arbirary zero-sized
value. The current code checks for null, but it could also be
undef or poison.

Replacing the specific null check with a check for
non-ConstantArray.
2022-03-10 16:02:12 +01:00
Arthur Eubanks f0b61f7957 Revert "[GlobalOpt] Don't replace alias with aliasee if either alias/aliasee may be preemptible"
This reverts commit 30e8f83c84.

Causes huge compile time regressions on certain large files. Will followup offline with author.
2022-03-03 11:04:14 -08:00
Fangrui Song 30e8f83c84 [GlobalOpt] Don't replace alias with aliasee if either alias/aliasee may be preemptible
Generalize D99629 for ELF. A default visibility non-local symbol is preemptible
in a -shared link. `isInterposable` is an insufficient condition.

Moreover, a non-preemptible alias may be referenced in a sub constant expression
which intends to lower to a PC-relative relocation. Replacing the alias with a
preemptible aliasee may introduce a linker error.

Respect dso_preemptable and suppress optimization to fix the abose issues. With
the change, `alias = 345` will not be rewritten to use aliasee in a `-fpic`
compile.
```
int aliasee;
extern int alias __attribute__((alias("aliasee"), visibility("hidden")));
void foo() { alias = 345; } // intended to access the local copy
```

While here, refine the condition for the alias as well.

For some binary formats like COFF, `isInterposable` is a sufficient condition.
But I think canonicalization for the changed case has little advantage, so I
don't bother to add the `Triple(M.getTargetTriple()).isOSBinFormatELF()` or
`getPICLevel/getPIELevel` complexity.

For instrumentations, it's recommended not to create aliases that refer to
globals that have a weak linkage or is preemptible. However, the following is
supported and the IR needs to handle such cases.
```
int aliasee __attribute__((weak));
extern int alias __attribute__((alias("aliasee")));
```

There are other places where GlobalAlias isInterposable usage may need to be
fixed.

Reviewed By: rnk

Differential Revision: https://reviews.llvm.org/D107249
2022-02-01 10:41:16 -08:00
Nikita Popov 236fbf571d [GlobalStatus] Skip non-pointer dead constant users
Constant expressions with a non-pointer result type used an early
exit that bypassed the later dead constant user check, and resulted
in different optimization outcomes depending on whether dead users
were present or not.

This fixes the issue reported in https://reviews.llvm.org/D117223#3287039.
2022-02-01 15:51:32 +01:00
Philip Reames 26049b8ce3 [GlobalOpt] Generalize malloc-to-global for any allocation function
We can generalize the malloc-to-global transform for other allocation functions which are both a) removable, and b) have a known initialization value.

One subtlety that I want to point out - mostly because I hadn't realized it was true until I took a closer look - is that the existing code doesn't prove that initialization/malloc happens only once. The initialization function can be called multiple times. This is correct without special handling for malloc as undef can map to any value previously written, but a non-undef initializing allocation it means we may end up memseting the new global repeatedly. In particular, this means it's not legal to fold the memset into the initializer of the global.

Differential Revision: https://reviews.llvm.org/D117503
2022-01-17 15:06:23 -08:00
Philip Reames 30715365d4 [test] precommit new test for D117503 2022-01-17 15:00:18 -08:00
Nikita Popov 499f1ca79f [GlobalOpt] Use generic type when converting malloc to global
The malloc to global transform currently determines the type of the
global by looking at bitcasts of the malloc. This is limited (the
transform fails if there are multiple different types) and
incompatible with opaque pointers.

My initial approach was to construct an appropriate struct type
based on usage in loads/stores. What this patch does instead is
to always create an [i8 x AllocSize] global, without trying to
guess types at all.

This does mean that other transforms that require a certain global
type may break. I fixed two of these in D117034 and D117223, which
I believe should be sufficient to avoid regressions. In particular,
the global SRA change should end up splitting the global into
naturally-typed sub-globals, at which point all other optimizations
should work.

Differential Revision: https://reviews.llvm.org/D117092
2022-01-17 09:55:33 +01:00
Nikita Popov 4796b4ae7b [GlobalOpt] Make global SRA offset based
Currently global SRA uses the GEP structure to determine how to
split the global. This patch instead analyses the loads and stores
that are performed on the global, and collects which types are used
at which offset, and then splits the global according to those.

This is both more general, and works fine with opaque pointers.
This is also closer to how ordinary SROA is performed.

Differential Revision: https://reviews.llvm.org/D117223
2022-01-17 09:28:36 +01:00
Nikita Popov be219323a2 [GlobalOpt] Add test for SRA with i8 array type (NFC) 2022-01-14 10:18:02 +01:00
Philip Reames 213193c184 [test] precommit coverage for D117249 2022-01-13 13:42:39 -08:00
Nikita Popov aba7c3c033 [ConstantFold] Check uniform value in ConstantFoldLoadFromConst()
This case is automatically handled if ConstantFoldLoadFromConstPtr()
is used. Make sure that ConstantFoldLoadFromConst() also handles it.
2022-01-13 14:40:19 +01:00
Nikita Popov 1cbb456123 [GlobalOpt] Fix global to select transform under opaque pointers
We need to check that the load/store type is also the same, as this
is no longer implicitly checked through the pointer type.
2022-01-13 11:13:06 +01:00
Nikita Popov f3e87176e1 [GlobalOpt] Support "stored once" optimization for different types
GlobalOpt can optimize a global with undef initializer and a single
store to put the stored value into the initializer instead. Currently,
this requires the type of the global and the store to match.

This patch extends support to cases with different types (but same
size), in which case we create a new global to replace the old one.

Differential Revision: https://reviews.llvm.org/D117034
2022-01-12 09:39:31 +01:00
Nikita Popov 94d6263391 [GlobalStatus] Look through non-constexpr casts
analyzeGlobal() looks through non-constexpr cast instructions when
looking for users. However, this particular place only strips the
casts again if they are constexprs. We should be looking through all
casts here.
2022-01-11 16:02:35 +01:00
Nikita Popov 3404127b4e [GlobalOpt] Regenerate test checks (NFC) 2022-01-11 15:34:34 +01:00
Nikita Popov 6e474d3308 [GlobalOpt][Evaluator] Fix off by one error in bounds check (PR53002)
We should bail out if the index is >= the size, not > the size.

Fixes https://github.com/llvm/llvm-project/issues/53002.
2022-01-05 14:06:02 +01:00
Nikita Popov 787f86e68c [GlobalOpt][Evaluator] Don't create bitcast for same type (PR52994)
isBitOrNoopPointerCastable() returns true if the types are the
same, but it's not actually possible to create a bitcast for all
such types. The assumption seems to be that the user will omit
creating the cast in that case, as it is unnecessary.

Fixes https://github.com/llvm/llvm-project/issues/52994.
2022-01-05 09:17:07 +01:00
Nikita Popov bbeaf2aac6 [GlobalOpt][Evaluator] Rewrite global ctor evaluation (fixes PR51879)
Global ctor evaluation currently models memory as a map from Constant*
to Constant*. For this to be correct, it is required that there is
only a single Constant* referencing a given memory location. The
Evaluator tries to ensure this by imposing certain limitations that
could result in ambiguities (by limiting types, casts and GEP formats),
but ultimately still fails, as can be seen in PR51879. The approach
is fundamentally fragile and will get more so with opaque pointers.

My original thought was to instead store memory for each global as an
offset => value representation. However, we also need to make sure
that we can actually rematerialize the modified global initializer
into a Constant in the end, which may not be possible if we allow
arbitrary writes.

What this patch does instead is to represent globals as a MutableValue,
which is either a Constant* or a MutableAggregate*. The mutable
aggregate exists to allow efficient mutation of individual aggregate
elements, as mutating an element on a Constant would require interning
a new constant. When a write to the Constant* is made, it is converted
into a MutableAggregate* as needed.

I believe this should make the evaluator more robust, compatible
with opaque pointers, and a bit simpler as well.

Fixes https://github.com/llvm/llvm-project/issues/51221.

Differential Revision: https://reviews.llvm.org/D115530
2022-01-04 09:30:54 +01:00
Nikita Popov 2926d6d335 [ConstantFold][GlobalOpt] Don't create x86_mmx null value
This fixes the assertion failure reported at
https://reviews.llvm.org/D114889#3198921 with a straightforward
check, until the cleaner fix in D115924 can be reapplied.
2021-12-21 09:11:41 +01:00
Nikita Popov aeb36ae0f4 Revert "[ConstantFolding] Unify handling of load from uniform value"
This reverts commit 9fd4f80e33.

This breaks SingleSource/Regression/C/gcc-c-torture/execute/pr19687.c
in test-suite. Either the test is incorrect, or clang is generating
incorrect union initialization code. I've submitted
https://reviews.llvm.org/D115994 to fix the test, assuming my
interpretation is correct. Reverting this in the meantime as it
may take some time to resolve.
2021-12-18 20:46:52 +01:00
Nikita Popov 9fd4f80e33 [ConstantFolding] Unify handling of load from uniform value
There are a number of places that specially handle loads from a
uniform value where all the bits are the same (zero, one, undef,
poison), because we a) don't care about the load offset in that
case and b) it bypasses casts that might not be legal generally
but do work with uniform values.

We had multiple implementations of this, with a different set of
supported values each time, as well as incomplete type checks in
some cases. In particular, this fixes the assertion reported in
https://reviews.llvm.org/D114889#3198921, as well as a similar
assertion that could be triggered via constant folding.

Differential Revision: https://reviews.llvm.org/D115924
2021-12-17 17:05:06 +01:00
Nikita Popov fcf7490028 [GlobalOpt] Add test for PR51879 (NFC) 2021-12-10 16:18:18 +01:00
Nikita Popov a0ff26e08c [GlobalOpt] Fix assertion failure during instruction deletion
This fixes the assertion failure reported in https://reviews.llvm.org/D114889#3166417,
by making RecursivelyDeleteTriviallyDeadInstructionsPermissive()
more permissive. As the function accepts a WeakTrackingVH, even if
originally only Instructions were inserted, we may end up with
different Value types after a RAUW operation. As such, we should
not assume that the vector only contains instructions.

Notably this matches the behavior of the
RecursivelyDeleteTriviallyDeadInstructions() function variant which
accepts a single value rather than vector.
2021-12-02 11:58:39 +01:00
Nikita Popov bc61e5e90b [GlobalOpt] Add test for PR39751 (NFC)
This has been fixed by D114889, as noted in the comments.
2021-12-02 09:17:33 +01:00
Nikita Popov 8d1759c404 [GlobalOpt] Simplify CleanupConstantGlobalUsers()
This bases the CleanupConstantGlobalUsers() implementation around
the ConstantFoldLoadFromConst() API. The general approach is that
we discover all users while looking through casts, and then
constant fold loads and drop stores and memintrinsics.

This avoids special cases and limitations in the previous
implementation, which is also incompatible with opaque pointers.
The result is a bit more powerful than before, because we now use
more general load folding logic which can for example look through
pointer bitcasts between different sizes. This is where the test
changes come from, as we now fold more loads and can thus remove
more globals.

Differential Revision: https://reviews.llvm.org/D114889
2021-12-01 21:06:25 +01:00