llvm-project/llvm/test/Transforms/LoopVectorize
Sander de Smalen 86729538bd [LV] Let selectVectorizationFactor reason directly on VectorizationFactor.
Rather than maintaining two separate values, a `float` for the per-lane
cost and a Width for the VF, maintain a single VectorizationFactor which
comprises the two and also removes the need for converting an integer value
to float.

This simplifies the query when asking if one VF is more profitable than
another when we want to extend this for scalable vectors (which may
require additional options to determine if e.g. a scalable VF of the
some cost, is more profitable than a fixed VF of the same cost).

The patch isn't entirely NFC because it also fixes an issue in
selectEpilogueVectorizationFactor, where the cost passed to ProfitableVFs
no longer truncates the floating-point cost from `float` to `unsigned` to
then perform the calculation on the truncated cost. It now does
a cost comparison with the correct precision.

Reviewed By: dmgreen

Differential Revision: https://reviews.llvm.org/D100121
2021-04-20 09:54:45 +01:00
..
AArch64 [NFC] Add tests for scalable vectorization of loops with in-order reductions 2021-04-19 11:15:55 +01:00
AMDGPU [Vectorizers][TTI] remove option to bypass creation of vector reduction intrinsics 2021-02-12 08:13:50 -05:00
ARM [LV] Logical and/or select costs 2021-04-08 10:39:47 +01:00
Hexagon [LV] Fix crash when computing max VF too early 2021-02-01 12:14:59 +00:00
PowerPC [LV] Add 'fast' flag to test to make sure it will be vectorized. 2021-03-23 15:32:23 +00:00
RISCV [RISCV] Add basic cost modelling for fixed vector gather/scatter. 2021-03-24 11:14:14 -07:00
SystemZ [SLP,LV] Use poison constant vector for shufflevector/initial insertelement 2021-01-06 11:22:50 +09:00
VE [VE][TTI] don't advertise vregs/vops 2020-11-06 11:12:10 +01:00
X86 [LV] Let selectVectorizationFactor reason directly on VectorizationFactor. 2021-04-20 09:54:45 +01:00
XCore
12-12-11-if-conv.ll
2012-10-20-infloop.ll
2012-10-22-isconsec.ll
2016-07-27-loop-vec.ll
alias-set-with-uncomputable-bounds.ll [LAA] Do not set CanDoRT to false for AS that do not need RT checks. 2020-06-14 20:55:59 +01:00
align.ll
assume.ll [LoopVectorize] auto-generate complete checks; NFC 2021-04-01 11:55:41 -04:00
bsd_regex.ll
bzip_reverse_loops.ll
calloc.ll [BasicAA] Rename deprecated -basicaa to -basic-aa 2020-06-26 20:41:37 -07:00
cast-induction.ll
check-prof-info.ll Revert "Use uint64_t for branch weights instead of uint32_t" 2020-10-31 00:25:32 -07:00
conditional-assignment.ll
consec_no_gep.ll
consecutive-ptr-uniforms.ll [SLP,LV] Use poison constant vector for shufflevector/initial insertelement 2021-01-06 11:22:50 +09:00
control-flow.ll Reapply "[LV] Vectorize (some) early and multiple exit loops"" w/fix for builder 2020-12-28 10:13:28 -08:00
cpp-new-array.ll
dbg.value.ll Migrate function attribute "no-frame-pointer-elim"="false" to "frame-pointer"="none" as cleanups after D56351 2019-12-24 16:27:51 -08:00
dead_instructions.ll [LV] Keep Primary Induction alive when folding tail by masking 2020-11-24 15:12:54 +02:00
debugloc.ll [Vectorizers][TTI] remove option to bypass creation of vector reduction intrinsics 2021-02-12 08:13:50 -05:00
demanded-bits-of-pointer-instruction.ll
diag-missing-instr-debug-loc.ll Migrate function attribute "no-frame-pointer-elim" to "frame-pointer"="all" as cleanups after D56351 2019-12-24 15:57:33 -08:00
diag-with-hotness-info-2.ll Migrate function attribute "no-frame-pointer-elim" to "frame-pointer"="all" as cleanups after D56351 2019-12-24 15:57:33 -08:00
diag-with-hotness-info.ll Migrate function attribute "no-frame-pointer-elim" to "frame-pointer"="all" as cleanups after D56351 2019-12-24 15:57:33 -08:00
disable_nonforced.ll
disable_nonforced_enable.ll
discriminator.ll
dont-fold-tail-for-const-TC.ll [IR] Let IRBuilder's CreateVectorSplat/CreateShuffleVector use poison as placeholder 2020-12-30 04:21:04 +09:00
dont-fold-tail-for-divisible-TC.ll [SCEV] Apply loop guards to divisibility tests 2021-02-02 08:09:39 +02:00
ee-crash.ll
exact.ll
explicit_outer_detection.ll
explicit_outer_nonuniform_inner.ll Require "target datalayout" to be at the beginning of an IR file. 2020-04-20 11:55:49 -07:00
explicit_outer_uniform_diverg_branch.ll Require "target datalayout" to be at the beginning of an IR file. 2020-04-20 11:55:49 -07:00
extract-last-veclane.ll [SVE][LoopVectorize] Add support for extracting the last lane of a scalable vector 2021-03-05 09:57:56 +00:00
fcmp-vectorize.ll Recommit #2: "[LV] Induction Variable does not remain scalar under tail-folding." 2020-05-13 13:50:09 +01:00
first-order-recurrence-complex.ll [NFCI] SCEVExpander: emit intrinsics for integral {u,s}{min,max} SCEV expressions 2021-03-06 21:52:46 +03:00
first-order-recurrence-multiply-recurrences.ll Recommit f0c2a5a "[LV] Generalize conditions for sinking instrs for first order recurrences." 2019-11-24 21:21:55 +00:00
first-order-recurrence.ll [NFCI] SCEVExpander: emit intrinsics for integral {u,s}{min,max} SCEV expressions 2021-03-06 21:52:46 +03:00
fix-reduction-dbg.ll [Vectorizers][TTI] remove option to bypass creation of vector reduction intrinsics 2021-02-12 08:13:50 -05:00
flags.ll [Vectorizers][TTI] remove option to bypass creation of vector reduction intrinsics 2021-02-12 08:13:50 -05:00
float-induction.ll [LoopVectorize] relax FMF constraint for FP induction 2021-03-18 08:11:22 -04:00
float-minmax-instruction-flag.ll [LoopVectorizer] Require no-signed-zeros-fp-math=true for fmin/fmax 2021-02-15 13:47:05 +00:00
float-reduction.ll Enable support for floating-point division reductions 2020-11-23 20:00:58 -05:00
fneg.ll
followup.ll [LV] Collect dead induction truncates 2020-10-08 08:28:58 +01:00
funcall.ll
gcc-examples.ll [BasicAA] Rename deprecated -basicaa to -basic-aa 2020-06-26 20:41:37 -07:00
gep_with_bitcast.ll
global_alias.ll
hints-trans.ll [SimplifyCFG] TryToSimplifyUncondBranchFromEmptyBlock() already knows how to preserve DomTree 2020-12-17 01:03:49 +03:00
hoist-loads.ll
i8-induction.ll
icmp-uniforms.ll [VPlan] Include name when printing after 93a9d2de8f. 2021-03-23 09:50:14 +00:00
if-conv-crash.ll
if-conversion-edgemasks.ll
if-conversion-nest.ll [LoopVectorize] Fix VPRecipeBuilder::createEdgeMask to correctly generate the mask 2021-02-14 21:12:34 +09:00
if-conversion-reduction.ll
if-conversion.ll
if-pred-non-void.ll [LoopVectorize] Fix VPRecipeBuilder::createEdgeMask to correctly generate the mask 2021-02-14 21:12:34 +09:00
if-pred-not-when-safe.ll [SimplifyCFG] TryToSimplifyUncondBranchFromEmptyBlock() already knows how to preserve DomTree 2020-12-17 01:03:49 +03:00
if-pred-stores.ll [NFCI] SCEVExpander: emit intrinsics for integral {u,s}{min,max} SCEV expressions 2021-03-06 21:52:46 +03:00
if-reduction.ll [LoopVectorize] Fix VPRecipeBuilder::createEdgeMask to correctly generate the mask 2021-02-14 21:12:34 +09:00
incorrect-dom-info.ll Migrate function attribute "no-frame-pointer-elim" to "frame-pointer"="all" as cleanups after D56351 2019-12-24 15:57:33 -08:00
increment.ll
induction-step.ll [IR] Let IRBuilder's CreateVectorSplat/CreateShuffleVector use poison as placeholder 2020-12-30 04:21:04 +09:00
induction.ll [Vectorizers][TTI] remove option to bypass creation of vector reduction intrinsics 2021-02-12 08:13:50 -05:00
induction_plus.ll
infiniteloop.ll
int_sideeffect.ll
interleaved-accesses-1.ll [IR] Let IRBuilder's CreateVectorSplat/CreateShuffleVector use poison as placeholder 2020-12-30 04:21:04 +09:00
interleaved-accesses-2.ll
interleaved-accesses-3.ll
interleaved-accesses-alias.ll
interleaved-accesses-masked-group.ll
interleaved-accesses-pred-stores.ll [IR] Let IRBuilder's CreateVectorSplat/CreateShuffleVector use poison as placeholder 2020-12-30 04:21:04 +09:00
interleaved-accesses-uniform-load.ll [LV] Apply sink-after & interleave-groups as VPlan transformations (NFCI) 2019-11-09 20:52:25 +02:00
interleaved-accesses.ll [InstCombine] (X | Op01C) + Op1C --> X + (Op01C + Op1C) iff the or is actually an add 2021-04-11 18:08:08 +03:00
interleaved-acess-with-remarks.ll
intrinsic.ll [LV] Add abs/smin/smax/umin/umax intrinsics to isTriviallyVectorizable 2020-07-29 10:23:07 -07:00
invariant-store-vectorization.ll [NFCI] SCEVExpander: emit intrinsics for integral {u,s}{min,max} SCEV expressions 2021-03-06 21:52:46 +03:00
irregular_type.ll [LoopVectorize] Refine hasIrregularType predicate 2021-03-17 17:03:47 +01:00
iv_outside_user.ll
lcssa-crash.ll
libcall-remark.ll
lifetime.ll
loop-form.ll [NFCI] SCEVExpander: emit intrinsics for integral {u,s}{min,max} SCEV expressions 2021-03-06 21:52:46 +03:00
loop-legality-checks.ll Reapply "[LV] Vectorize (some) early and multiple exit loops"" w/fix for builder 2020-12-28 10:13:28 -08:00
loop-scalars.ll [test] Fix Transforms/LoopVectorize under NPM 2020-12-07 21:48:21 -08:00
loop-vect-memdep.ll
loop-vect-option.ll
memdep-fold-tail.ll [LV] Fallback strategies if tail-folding fails 2020-08-26 16:55:25 +01:00
memdep.ll
metadata-unroll.ll
metadata-width.ll [LV] Legalize scalable VF hints 2021-01-08 10:49:44 +00:00
metadata.ll
middle-block-dbg.ll
miniters.ll
minmax_reduction.ll [LoopVectorizer] Require no-signed-zeros-fp-math=true for fmin/fmax 2021-02-15 13:47:05 +00:00
mixed-precision-remarks.ll [LV] Add analysis remark for mixed precision conversions 2021-02-17 21:37:08 -05:00
multi-use-reduction-bug.ll
multiple-address-spaces.ll [BasicAA] Rename deprecated -basicaa to -basic-aa 2020-06-26 20:41:37 -07:00
multiple-exits-versioning.ll [LV] Add test cases with multiple exits which require versioning. 2021-01-20 11:48:48 +00:00
multiple-strides-vectorization.ll [IR] Let IRBuilder's CreateVectorSplat/CreateShuffleVector use poison as placeholder 2020-12-30 04:21:04 +09:00
no-interleave-up-front.ll
no_array_bounds.ll
no_array_bounds_scalable.ll [SVE] Add support for scalable vectors with vectorize.scalable.enable loop attribute 2020-12-02 13:23:43 +00:00
no_idiv_reduction.ll
no_int_induction.ll [LV] Strip wrap flags from vectorized reductions 2019-12-20 14:48:53 +02:00
no_outside_user.ll
no_switch.ll
no_switch_disable_vectorization.ll
noalias-md-licm.ll Rename scoped-noalias -> scoped-noalias-aa 2020-07-24 12:14:27 -07:00
noalias-md.ll Rename scoped-noalias -> scoped-noalias-aa 2020-07-24 12:14:27 -07:00
noalias-scope-decl.ll [noalias.decl] Look through llvm.experimental.noalias.scope.decl 2021-01-19 20:09:42 +01:00
nofloat-report.ll
nofloat.ll
non-const-n.ll
nontemporal.ll
novect-lcssa-cfg-invalidation.ll [NewPM] Don't print 'Invalidating all non-preserved analyses' 2020-07-30 19:40:29 -07:00
nsw-crash.ll
nuw.ll [LV] Strip wrap flags from vectorized reductions 2019-12-20 14:48:53 +02:00
opt.ll
optimal-epilog-vectorization-limitations.ll [LV] Legalize scalable VF hints 2021-01-08 10:49:44 +00:00
optimal-epilog-vectorization-liveout.ll [LV] Epilogue Vectorization with Optimal Control Flow (Recommit) 2020-12-02 10:09:56 -05:00
optimal-epilog-vectorization-scalable.ll [LV] Legalize scalable VF hints 2021-01-08 10:49:44 +00:00
optimal-epilog-vectorization.ll [NewPM][opt] Run the "default" AA pipeline by default 2021-01-21 21:08:54 -08:00
optsize.ll [IR] Let IRBuilder's CreateVectorSplat/CreateShuffleVector use poison as placeholder 2020-12-30 04:21:04 +09:00
outer-loop-vec-phi-predecessor-order.ll [VPlan] Manage pairs of incoming (VPValue, VPBB) in VPWidenPHIRecipe. 2021-02-22 09:44:25 +00:00
outer_loop_test1.ll [VPlan] Manage pairs of incoming (VPValue, VPBB) in VPWidenPHIRecipe. 2021-02-22 09:44:25 +00:00
outer_loop_test2.ll [VPlan] Manage pairs of incoming (VPValue, VPBB) in VPWidenPHIRecipe. 2021-02-22 09:44:25 +00:00
partial-lcssa.ll
phi-cost.ll [test] Fix Transforms/LoopVectorize under NPM 2020-12-07 21:48:21 -08:00
phi-hang.ll
pointer-induction.ll Reland [SCEV] Improve modelling for (null) pointer constants 2021-03-13 16:05:34 +03:00
pr25281.ll
pr28541.ll
pr30654-phiscev-sext-trunc.ll
pr30806-phi-scev.ll
pr30806.ll
pr31098.ll
pr31190.ll
pr32859.ll
pr33706.ll
pr34681.ll [test, LoopVectorize] Fix use of var defined in CHECK-NOT 2021-04-09 10:01:57 +01:00
pr35743.ll
pr35773.ll Recommit #2: "[LV] Induction Variable does not remain scalar under tail-folding." 2020-05-13 13:50:09 +01:00
pr36311.ll
pr36983.ll
pr37248.ll
pr37515.ll
pr38697.ll
pr38800.ll
pr39099.ll
pr39417-optsize-scevchecks.ll [SLP,LV] Use poison constant vector for shufflevector/initial insertelement 2021-01-06 11:22:50 +09:00
pr43166-fold-tail-by-masking.ll
pr44488-predication.ll [SLP,LV] Use poison constant vector for shufflevector/initial insertelement 2021-01-06 11:22:50 +09:00
pr45259.ll [LoopVectorize] Fix crash on "getNoopOrZeroExtend cannot truncate!" (PR45259) 2020-03-30 10:14:14 -07:00
pr45525.ll [LV] Fix PR45525: Incorrect assert in blend recipe 2020-04-15 10:39:07 +03:00
pr45679-fold-tail-by-masking.ll [NFC][LV][TEST]: extend pr45679-fold-tail-by-masking.ll with -force-vector-width=1 -force-vector-interleave=4 2020-06-09 18:30:56 +00:00
pr46525-expander-insertpoint.ll [IR] Let IRBuilder's CreateVectorSplat/CreateShuffleVector use poison as placeholder 2020-12-30 04:21:04 +09:00
pr47343-expander-lcssa-after-cfg-update.ll [LV] Generate RT checks up-front and remove them if required. 2021-03-01 10:48:04 +00:00
pr48832.ll [LoopVectorize] Fix VPRecipeBuilder::createEdgeMask to correctly generate the mask 2021-02-14 21:12:34 +09:00
preserve-dbg-loc-and-loop-metadata.ll
ptr-induction.ll
ptr_loops.ll [BasicAA] Rename deprecated -basicaa to -basic-aa 2020-06-26 20:41:37 -07:00
read-only.ll
reduction-inloop-pred.ll [LoopVectorize] Fix VPRecipeBuilder::createEdgeMask to correctly generate the mask 2021-02-14 21:12:34 +09:00
reduction-inloop-uf4.ll [Vectorizers][TTI] remove option to bypass creation of vector reduction intrinsics 2021-02-12 08:13:50 -05:00
reduction-inloop.ll [LoopVectorize] Fix VPRecipeBuilder::createEdgeMask to correctly generate the mask 2021-02-14 21:12:34 +09:00
reduction-order.ll [LV] Tail folded inloop reductions. 2020-10-11 16:58:34 +01:00
reduction-predselect.ll [Vectorizers][TTI] remove option to bypass creation of vector reduction intrinsics 2021-02-12 08:13:50 -05:00
reduction-ptr.ll [Analysis][LoopVectorize] do not form reductions of pointers 2021-02-19 14:01:57 -05:00
reduction-small-size.ll [LV] Strip wrap flags from vectorized reductions 2019-12-20 14:48:53 +02:00
reduction.ll [Vectorizers][TTI] remove option to bypass creation of vector reduction intrinsics 2021-02-12 08:13:50 -05:00
remarks-multi-exit-loops.ll [LV] Fix crash when generating remarks with multi-exit loops. 2021-01-01 13:54:41 +00:00
remove_metadata.ll
reverse_induction.ll
reverse_iter.ll
runtime-check-address-space.ll [BasicAA] Rename deprecated -basicaa to -basic-aa 2020-06-26 20:41:37 -07:00
runtime-check-needed-but-empty.ll Recommit #2: "[LV] Induction Variable does not remain scalar under tail-folding." 2020-05-13 13:50:09 +01:00
runtime-check-pointer-element-type.ll [LAA] Use DL to get element size for bound computation. 2020-10-07 18:57:07 +01:00
runtime-check-readonly-address-space.ll Migrate function attribute "no-frame-pointer-elim" to "frame-pointer"="all" as cleanups after D56351 2019-12-24 15:57:33 -08:00
runtime-check-readonly.ll
runtime-check.ll Revert "[InstCombine] add multi-use demanded bits fold for add with low-bit mask" 2020-11-16 14:48:09 -05:00
runtime-drop-crash.ll [LV] Generate RT checks up-front and remove them if required. 2021-03-01 10:48:04 +00:00
safegep.ll
same-base-access.ll
scalable-loop-unpredicated-body-scalar-tail.ll [LV] Legalize scalable VF hints 2021-01-08 10:49:44 +00:00
scalable-vf-hint.ll [LV] Legalize scalable VF hints 2021-01-08 10:49:44 +00:00
scalar-select.ll
scalar_after_vectorization.ll
scev-exitlim-crash.ll Migrate function attribute "no-frame-pointer-elim" to "frame-pointer"="all" as cleanups after D56351 2019-12-24 15:57:33 -08:00
select-reduction.ll [Vectorizers][TTI] remove option to bypass creation of vector reduction intrinsics 2021-02-12 08:13:50 -05:00
simple-unroll.ll
single-value-blend-phis.ll Recommit "[LV] Allow tryToCreateWidenRecipe to return a VPValue, use for blends." 2021-02-24 10:36:02 +00:00
skeleton-lcssa-crash.ll [LV] Generate RT checks up-front and remove them if required. 2021-03-01 10:48:04 +00:00
skip-iterations.ll
small-loop.ll
start-non-zero.ll
store-shuffle-bug.ll [BasicAA] Rename deprecated -basicaa to -basic-aa 2020-06-26 20:41:37 -07:00
struct_access.ll
tail-folding-counting-down.ll [LV] Fallback strategies if tail-folding fails 2020-08-26 16:55:25 +01:00
tail-folding-vectorization-factor-1.ll [LoopVectorizer] Don't create unused block masks for reductions. NFC 2020-07-30 14:28:08 +01:00
tbaa-nodep.ll [SimplifyCFG] TryToSimplifyUncondBranchFromEmptyBlock() already knows how to preserve DomTree 2020-12-17 01:03:49 +03:00
tripcount.ll Revert "Use uint64_t for branch weights instead of uint32_t" 2020-10-31 00:25:32 -07:00
undef-inst-bug.ll
uniform-blend.ll [LV] Fix name in CHECK pattern after fb3ca7076 2021-03-12 13:31:48 +00:00
unroll-novec-memcheck-metadata.ll
unroll.ll
unroll_novec.ll
unsafe-dep-remark.ll
unsafe-vf-hint-remark.ll [LV] Clamp VF hint when unsafe 2020-12-01 11:30:34 +00:00
unsized-pointee-crash.ll
use-scalar-epilogue-if-tp-fails.ll [LV] Fallback strategies if tail-folding fails 2020-08-26 16:55:25 +01:00
value-ptr-bug.ll
vect-phiscev-sext-trunc.ll
vect.omp.persistence.ll
vect.stats.ll
vector-geps.ll [IR] Let IRBuilder's CreateVectorSplat/CreateShuffleVector use poison as placeholder 2020-12-30 04:21:04 +09:00
vector-intrinsic-call-cost.ll Recommit #2: "[LV] Induction Variable does not remain scalar under tail-folding." 2020-05-13 13:50:09 +01:00
vectorize-once.ll [SimplifyCFG] TryToSimplifyUncondBranchFromEmptyBlock() already knows how to preserve DomTree 2020-12-17 01:03:49 +03:00
vectorizeVFone.ll [VFDatabase] Testsuite for scalar functions are vector functions with VF =1 2020-04-30 15:47:21 -04:00
version-mem-access.ll [BasicAA] Rename deprecated -basicaa to -basic-aa 2020-06-26 20:41:37 -07:00
vplan-dot-printing.ll [VPlan] Add plain text (not DOT's digraph) dumps 2021-03-19 10:50:12 -07:00
vplan-outer-loop-uncomputable-trip-count.ll [VPlan] Reject loops without computable backedge taken counts 2020-06-12 10:31:18 +01:00
vplan-printing.ll [VPlan] Include name when printing after 93a9d2de8f. 2021-03-23 09:50:14 +00:00
vplan-stress-test-no-explict-vf.ll
vplan-vectorize-inner-loop-reduction.ll [VPlan] Manage pairs of incoming (VPValue, VPBB) in VPWidenPHIRecipe. 2021-02-22 09:44:25 +00:00
vplan-widen-call-instruction.ll [VPlan] Support to widen call intructions in VPlan native path 2021-03-06 21:59:52 +00:00
vplan-widen-select-instruction.ll [VPlan] Support to widen select intructions in VPlan native path 2021-03-10 20:59:53 +00:00
vplan_hcfg_stress_test.ll
write-only.ll
zero-sized-pointee-crash.ll