OSchip
/
llvm-project
mirror of https://github.com/THU-DSP-LAB/llvm-project.git
13
0
Fork 6
Code Issues Packages Projects Releases Wiki Activity
llvm-project/llvm/test/Transforms/LoopVectorize/AArch64
History
Philip Reames f79214d1e1 [LV] Support predicated div/rem operations via safe-divisor select idiom 
This patch adds support for vectorizing conditionally executed div/rem operations via a variant of widening. The existing support for predicated divrem in the vectorizer requires scalarization which we can't do for scalable vectors.

The basic idea is that we can always divide (take remainder) by 1 without executing UB. As such, we can use the active lane mask to conditional select either the actual divisor for active lanes, or a constant one for inactive lanes. We already account for the cost of the active lane mask, so the only additional cost is a splat of one and the vector select. This is one of several possible approaches to this problem; see the review thread for discussion on some of the others.  This one was chosen mostly because it was straight forward, and none of the others seemed oviously better.

I enabled the new code only for scalable vectors. We could also legally enable it for fixed vectors as well, but I haven't thought through the cost tradeoffs between widening and scalarization enough to know if that's profitable. This will be explored in future patches.

Differential Revision: https://reviews.llvm.org/D130164
 		2022-08-24 10:07:59 -07:00
..
Oz-and-forced-vectorize.ll Revert "[NFCI] Regenerate SROA/LoopVectorize test checks" 2022-04-04 01:15:30 +02:00
aarch64-predication.ll
…
aarch64-unroll.ll Revert "[NFCI] Regenerate SROA/LoopVectorize test checks" 2022-04-04 01:15:30 +02:00
arbitrary-induction-step.ll
…
arm64-unroll.ll Revert "[NFCI] Regenerate SROA/LoopVectorize test checks" 2022-04-04 01:15:30 +02:00
backedge-overflow.ll
…
deterministic-type-shrinkage.ll
…
eliminate-tail-predication.ll Revert "[NFCI] Regenerate SROA/LoopVectorize test checks" 2022-04-04 01:15:30 +02:00
extend-vectorization-factor-for-unprofitable-memops.ll Revert "Revert "[AArch64] Set maximum VF with shouldMaximizeVectorBandwidth"" 2022-05-23 16:15:45 +01:00
extractvalue-no-scalarization-required.ll [InstCombine] Tighten up known library function signature tests (PR #56463) 2022-08-10 14:15:46 -06:00
first-order-recurrence.ll Revert "[NFCI] Regenerate SROA/LoopVectorize test checks" 2022-04-04 01:15:30 +02:00
fixed-order-recurrence.ll [AArch64][X86] Add some fixed-order-recurrence tests to check the costmodel of fixed order recurrences. NFC 2022-08-24 08:18:01 +01:00
gather-cost.ll Revert "[NFCI] Regenerate SROA/LoopVectorize test checks" 2022-04-04 01:15:30 +02:00
gather-do-not-vectorize-addressing.ll [NFC][LoopVectorize] Explicitly disable tail-folding on some SVE tests 2022-07-21 15:23:00 +01:00
i1-reg-usage.ll [NFC][LoopVectorize] Explicitly disable tail-folding on some SVE tests 2022-07-21 15:23:00 +01:00
induction-trunc.ll
…
interleaved-store-of-first-order-recurrence.ll Revert "[NFCI] Regenerate SROA/LoopVectorize test checks" 2022-04-04 01:15:30 +02:00
interleaved-vs-scalar.ll [AArch64][LV] AArch64 does not prefer vectorized addressing 2022-06-17 18:32:50 +08:00
interleaved_cost.ll [NFC][LV] Use different braces in debug output 2022-03-07 19:32:37 +03:00
intrinsiccost.ll Revert "[NFCI] Regenerate SROA/LoopVectorize test checks" 2022-04-04 01:15:30 +02:00
lit.local.cfg
…
loop-vectorization-factors.ll Revert "Revert "[AArch64] Set maximum VF with shouldMaximizeVectorBandwidth"" 2022-05-23 16:15:45 +01:00
loopvectorize_pr33804_double.ll
…
masked-call.ll [NFC][LoopVectorize] Precommit masked vector function call tests 2022-08-23 09:47:10 +01:00
masked-op-cost.ll Revert "[NFCI] Regenerate SROA/LoopVectorize test checks" 2022-04-04 01:15:30 +02:00
max-vf-for-interleaved.ll
…
maximize-bandwidth-invalidate.ll [LV] Invalidate widening decisions after maximizing vector bandwidth 2022-03-31 09:19:31 +01:00
no_vector_instructions.ll
…
nontemporal-load-store.ll [AArch64] Add tests to check for loop vectorization of non temporal loads 2022-08-16 09:40:51 +01:00
outer_loop_test1_no_explicit_vect_width.ll
…
pr31900.ll
…
pr33053.ll Revert "[NFCI] Regenerate SROA/LoopVectorize test checks" 2022-04-04 01:15:30 +02:00
pr36032.ll
…
pr46950-load-cast-context-crash.ll
…
predication_costs.ll
…
reduction-small-size.ll Revert "Revert "[AArch64] Set maximum VF with shouldMaximizeVectorBandwidth"" 2022-05-23 16:15:45 +01:00
runtime-check-size-based-threshold.ll [LV] Vectorize cases with larger number of RT checks, execute only if profitable. 2022-07-04 15:11:39 +01:00
scalable-alloca.ll Revert "[NFCI] Regenerate SROA/LoopVectorize test checks" 2022-04-04 01:15:30 +02:00
scalable-avoid-scalarization.ll [ValueTracking] Enable -branch-on-poison-as-ub by default 2022-06-01 10:46:06 +02:00
scalable-call.ll [NFC][LoopVectorize] Explicitly disable tail-folding on some SVE tests 2022-07-21 15:23:00 +01:00
scalable-predicate-instruction.ll [LV] Support predicated div/rem operations via safe-divisor select idiom 2022-08-24 10:07:59 -07:00
scalable-reduction-inloop-cond.ll [NFC][LoopVectorize] Explicitly disable tail-folding on some SVE tests 2022-07-21 15:23:00 +01:00
scalable-reductions-tf.ll [AArch64][NFC] Simplify loop vectoriser tail-folding tests 2022-07-18 13:37:29 +01:00
scalable-reductions.ll [SVE][LoopVectorize][NFC] Tidy up some tests 2022-08-19 15:12:58 +01:00
scalable-strict-fadd.ll [NFC][LoopVectorize] Explicitly disable tail-folding on some SVE tests 2022-07-21 15:23:00 +01:00
scalable-vectorization-cost-tuning.ll Revert "Revert "[AArch64] Set maximum VF with shouldMaximizeVectorBandwidth"" 2022-05-23 16:15:45 +01:00
scalable-vectorization.ll Revert "Revert "[AArch64] Set maximum VF with shouldMaximizeVectorBandwidth"" 2022-05-23 16:15:45 +01:00
scalable-vf-hint.ll [NFC][LV] Use different braces in debug output 2022-03-07 19:32:37 +03:00
scalar_interleave.ll
…
scalarize-store-with-predication.ll [NFC][LoopVectorize] Explicitly disable tail-folding on some SVE tests 2022-07-21 15:23:00 +01:00
sdiv-pow2.ll
…
select-costs.ll Revert "[NFCI] Regenerate SROA/LoopVectorize test checks" 2022-04-04 01:15:30 +02:00
smallest-and-widest-types.ll [NFC][LV] Use different braces in debug output 2022-03-07 19:32:37 +03:00
strict-fadd-cost.ll
…
strict-fadd-vf1.ll Revert "[NFCI] Regenerate SROA/LoopVectorize test checks" 2022-04-04 01:15:30 +02:00
strict-fadd.ll [LoopVectorize] Ensure the VPReductionRecipe is placed after all it's inputs 2022-07-13 09:29:25 +01:00
sve-basic-vec.ll [NFC][LoopVectorize] Explicitly disable tail-folding on some SVE tests 2022-07-21 15:23:00 +01:00
sve-cond-inv-loads.ll [NFC][LoopVectorize] Explicitly disable tail-folding on some SVE tests 2022-07-21 15:23:00 +01:00
sve-epilog-vect-inloop-reductions.ll [LV] Do not LoopSimplify/LCSSA after generating main vector loop. 2022-05-20 09:58:40 +01:00
sve-epilog-vect-reductions.ll [LV] Do not LoopSimplify/LCSSA after generating main vector loop. 2022-05-20 09:58:40 +01:00
sve-epilog-vect-strict-reductions.ll [LV] Do not LoopSimplify/LCSSA after generating main vector loop. 2022-05-20 09:58:40 +01:00
sve-epilog-vect.ll [NFC][LoopVectorize] Explicitly disable tail-folding on some SVE tests 2022-07-21 15:23:00 +01:00
sve-extract-last-veclane.ll Revert "[NFCI] Regenerate SROA/LoopVectorize test checks" 2022-04-04 01:15:30 +02:00
sve-fneg.ll [NFC][LoopVectorize] Explicitly disable tail-folding on some SVE tests 2022-07-21 15:23:00 +01:00
sve-gather-scatter-cost.ll [NFC][LoopVectorize] Explicitly disable tail-folding on some SVE tests 2022-07-21 15:23:00 +01:00
sve-gather-scatter.ll [NFC][LoopVectorize] Explicitly disable tail-folding on some SVE tests 2022-07-21 15:23:00 +01:00
sve-illegal-type.ll [NFC][LoopVectorize] Explicitly disable tail-folding on some SVE tests 2022-07-21 15:23:00 +01:00
sve-inductions-unusual-types.ll [SVE][LoopVectorize][NFC] Tidy up some tests 2022-08-19 15:12:58 +01:00
sve-inductions.ll [NFC][LoopVectorize] Explicitly disable tail-folding on some SVE tests 2022-07-21 15:23:00 +01:00
sve-inv-loads.ll [NFC][LoopVectorize] Explicitly disable tail-folding on some SVE tests 2022-07-21 15:23:00 +01:00
sve-inv-store.ll [LV][SVE] Autogen a test for ease of update 2022-07-21 13:12:53 -07:00
sve-large-strides.ll [NFC][LoopVectorize] Explicitly disable tail-folding on some SVE tests 2022-07-21 15:23:00 +01:00
sve-live-out-pointer-induction.ll [LV] Widen ptr-inductions with scalar uses for scalable VFs. 2022-05-22 16:24:13 +01:00
sve-low-trip-count.ll [AArch64][LoopVectorize] Introduce trip count minimal value threshold to ignore tail-folding. 2022-08-09 22:10:17 +01:00
sve-masked-loadstore.ll [NFC][LoopVectorize] Explicitly disable tail-folding on some SVE tests 2022-07-21 15:23:00 +01:00
sve-runtime-check-size-based-threshold.ll [NFC][LoopVectorize] Explicitly disable tail-folding on some SVE tests 2022-07-21 15:23:00 +01:00
sve-scalable-load-in-loop.ll
…
sve-select-cmp.ll [NFC][LoopVectorize] Explicitly disable tail-folding on some SVE tests 2022-07-21 15:23:00 +01:00
sve-strict-fadd-cost.ll [SVE][LoopVectorize][NFC] Tidy up some tests 2022-08-19 15:12:58 +01:00
sve-tail-folding-cost.ll [AArch64][NFC] Simplify loop vectoriser tail-folding tests 2022-07-18 13:37:29 +01:00
sve-tail-folding-forced.ll [LoopVectorize] Add option to use active lane mask for loop control flow 2022-07-11 13:46:55 +01:00
sve-tail-folding-option.ll [AArch64][LoopVectorize] Disable tail-folding for SVE when loop has interleaved accesses 2022-08-02 09:52:33 +01:00
sve-tail-folding-optsize.ll [LoopVectorize] Add option to use active lane mask for loop control flow 2022-07-11 13:46:55 +01:00
sve-tail-folding-reductions.ll [LoopVectorize][AArch64] Add TTI hook preferPredicatedReductionSelect 2022-07-20 09:33:29 +01:00
sve-tail-folding-unroll.ll [AArch64][NFC] Simplify loop vectoriser tail-folding tests 2022-07-18 13:37:29 +01:00
sve-tail-folding.ll [LV] Support predicated div/rem operations via safe-divisor select idiom 2022-08-24 10:07:59 -07:00
sve-type-conv.ll Revert "[NFCI] Regenerate SROA/LoopVectorize test checks" 2022-04-04 01:15:30 +02:00
sve-vector-reverse-mask4.ll [NFC][LoopVectorize] Explicitly disable tail-folding on some SVE tests 2022-07-21 15:23:00 +01:00
sve-vector-reverse.ll [SVE][LoopVectorize][NFC] Tidy up some tests 2022-08-19 15:12:58 +01:00
sve-widen-extractvalue.ll Revert "[NFCI] Regenerate SROA/LoopVectorize test checks" 2022-04-04 01:15:30 +02:00
sve-widen-gep.ll [SVE][LoopVectorize][NFC] Tidy up some tests 2022-08-19 15:12:58 +01:00
sve-widen-phi.ll [SVE][LoopVectorize][NFC] Tidy up some tests 2022-08-19 15:12:58 +01:00
tail-fold-uniform-memops.ll [LoopVectorize] Add option to use active lane mask for loop control flow 2022-07-11 13:46:55 +01:00
type-shrinkage-insertelt.ll
…
unsafe-vf-hint-remark.ll
…
veclib-calls-libsystem-darwin.ll Revert "[NFCI] Regenerate SROA/LoopVectorize test checks" 2022-04-04 01:15:30 +02:00
vector-reverse-mask4.ll [SVE][LoopVectorize][NFC] Tidy up some tests 2022-08-19 15:12:58 +01:00
vector-reverse.ll [NFC][LoopVectorize] Explicitly disable tail-folding on some SVE tests 2022-07-21 15:23:00 +01:00