5a197c06f3
LoopVectorizer: Add support for floating point reductions
...
llvm-svn: 171812
2013-01-07 23:13:00 +00:00
c60d7d96f5
LoopVectorizer: When we vectorizer and widen loops we process many elements at once. This is a good thing, except for
...
small loops. On small loops post-loop that handles scalars (and runs slower) can take more time to execute than the
rest of the loop. This patch disables widening of loops with a small static trip count.
llvm-svn: 171798
2013-01-07 21:54:51 +00:00
f19d515316
Fix a typo. Remove the duplicated test.
...
llvm-svn: 171584
2013-01-05 01:17:46 +00:00
e9f5bfd5e9
iLoopVectorize: Non commutative operators can be used as reduction variables as long as the reduction chain is used in the LHS.
...
PR14803.
llvm-svn: 171583
2013-01-05 01:15:47 +00:00
6d9dafe3ff
Force a fixed unroll count on the target independent tests.
...
This should fix clang-native-arm-cortex-a9. Thanks Renato.
llvm-svn: 171582
2013-01-05 00:58:48 +00:00
874f01e956
Do not vectorize loops with subtraction reductions
...
Since subtraction does not commute the loop vectorizer incorrectly vectorizes
reductions such as x = A[i] - x.
Disabling for now.
llvm-svn: 171537
2013-01-04 22:10:16 +00:00
e1d5c4b8b9
LoopVectorizer:
...
1. Add code to estimate register pressure.
2. Add code to select the unroll factor based on register pressure.
3. Add bits to TargetTransformInfo to provide the number of registers.
llvm-svn: 171469
2013-01-04 17:48:25 +00:00
d554a517c0
LoopVectorizer: Test the unrolling flag.
...
llvm-svn: 171446
2013-01-03 01:47:31 +00:00
4897392360
Avoid vectorization when the function has the "noimplicitflot" attribute.
...
llvm-svn: 171429
2013-01-02 23:54:43 +00:00
0b37f14371
LoopVectorizer: Fix a bug in the code that updates the loop exiting block.
...
LCSSA PHIs may have undef values. The vectorizer updates values that are used by outside users such as PHIs.
The bug happened because undefs are not loop values. This patch handles these PHIs.
PR14725
llvm-svn: 171251
2012-12-30 07:47:00 +00:00
5350cd314b
If all of the write objects are identified then we can vectorize the loop even if the read objects are unidentified.
...
PR14719.
llvm-svn: 171124
2012-12-26 23:30:53 +00:00
3f7c4f36ba
LoopVectorizer: Optimize the vectorization of consecutive memory access when the iteration step is -1
...
llvm-svn: 171114
2012-12-26 19:08:17 +00:00
b44f890133
LoopVectorize: Enable vectorization of the fmuladd intrinsic
...
llvm-svn: 171076
2012-12-25 23:21:29 +00:00
fb43258080
Fix typo "Makre" -> "Make".
...
llvm-svn: 171043
2012-12-24 19:55:47 +00:00
5f7c12cfbd
LoopVectorizer: When checking for vectorizable types, also check
...
the StoreInst operands.
PR14705.
llvm-svn: 171023
2012-12-24 09:14:18 +00:00
bd5d1d832a
LoopVectorizer: Fix an endless loop in the code that looks for reductions.
...
The bug was in the code that detects PHIs in if-then-else block sequence.
PR14701.
llvm-svn: 171008
2012-12-24 01:22:06 +00:00
cf9999d9d5
CostModel: Change the default target-independent implementation for finding
...
the cost of arithmetic functions. We now assume that the cost of arithmetic
operations that are marked as Legal or Promote is low, but ops that are
marked as custom are higher.
llvm-svn: 171002
2012-12-23 17:31:23 +00:00
2cade68025
Loop Vectorizer: Update the cost model of scatter/gather operations and make
...
them more expensive.
llvm-svn: 170995
2012-12-23 07:23:55 +00:00
e7785686a5
Fix a bug in the code that checks if we can vectorize loops while using dynamic
...
memory bound checks. Before the fix we were able to vectorize this loop from
the Livermore Loops benchmark:
for ( k=1 ; k<n ; k++ )
x[k] = x[k-1] + y[k];
llvm-svn: 170811
2012-12-21 00:07:35 +00:00
2ababf68d7
LoopVectorize: Fix a bug in the scalarization of instructions.
...
Before if-conversion we could check if a value is loop invariant
if it was declared inside the basic block. Now that loops have
multiple blocks this check is incorrect.
This fixes External/SPEC/CINT95/099_go/099_go
llvm-svn: 170756
2012-12-20 20:24:40 +00:00
ae0bb61053
Make TargetLowering::getTypeConversion more resilient against odd illegal MVTs.
...
- An MVT can become an EVT when being split (e.g. v2i8 -> v1i8, the latter doesn't exist)
- Return the scalar value when an MVT is scalarized (v1i64 -> i64)
Fixes PR14639ff.
llvm-svn: 170546
2012-12-19 14:34:28 +00:00
f0e5d2f032
LoopVectorize: Emit reductions as log2(vectorsize) shuffles + vector ops instead of scalar operations.
...
For example on x86 with SSE4.2 a <8 x i8> add reduction becomes
movdqa %xmm0, %xmm1
movhlps %xmm1, %xmm1 ## xmm1 = xmm1[1,1]
paddw %xmm0, %xmm1
pshufd $1, %xmm1, %xmm0 ## xmm0 = xmm1[1,0,0,0]
paddw %xmm1, %xmm0
phaddw %xmm0, %xmm0
pextrb $0, %xmm0, %edx
instead of
pextrb $2, %xmm0, %esi
pextrb $0, %xmm0, %edx
addb %sil, %dl
pextrb $4, %xmm0, %esi
addb %dl, %sil
pextrb $6, %xmm0, %edx
addb %sil, %dl
pextrb $8, %xmm0, %esi
addb %dl, %sil
pextrb $10, %xmm0, %edi
pextrb $14, %xmm0, %edx
addb %sil, %dil
pextrb $12, %xmm0, %esi
addb %dil, %sil
addb %sil, %dl
llvm-svn: 170439
2012-12-18 18:40:20 +00:00
36510f7194
Teach the cost model about the optimization in r169904: Truncation of induction variables costs the same as scalar trunc.
...
llvm-svn: 170051
2012-12-13 00:21:03 +00:00
d0bb22bba3
LoopVectorizer: Use the "optsize" attribute to decide if we are allowed to increase the function size.
...
llvm-svn: 170004
2012-12-12 19:29:45 +00:00
f707bf4ca3
PR14574. Fix a bug in the code that calculates the mask the converted PHIs in if-conversion.
...
llvm-svn: 169916
2012-12-11 21:30:14 +00:00
e266efb70b
Loop Vectorize: optimize the vectorization of trunc(induction_var). The truncation is now done on scalars.
...
llvm-svn: 169904
2012-12-11 18:58:10 +00:00
dbb3328194
Fix PR14565. Don't if-convert loops that have switch statements in them.
...
llvm-svn: 169813
2012-12-11 04:55:10 +00:00
7b5b55c195
Add support for reverse induction variables. For example:
...
while (i--)
sum+=A[i];
llvm-svn: 169752
2012-12-10 19:25:06 +00:00
2adb13c100
LoopVectorize: support vectorizing intrinsic calls
...
- added function to VectorTargetTransformInfo to query cost of intrinsics
- vectorize trivially vectorizable intrinsic calls such as sin, cos, log, etc.
Reviewed by: Nadav
llvm-svn: 169711
2012-12-09 20:42:17 +00:00
93fa5ef957
Fix a bug in vectorization of if-converted reduction variables. If the
...
reduction variable is not used outside the loop then we ran into an
endless loop. This change checks if we found the original PHI.
llvm-svn: 169324
2012-12-04 22:40:22 +00:00
a10b311aec
Add support for reduction variables when IF-conversion is enabled.
...
llvm-svn: 169288
2012-12-04 18:17:33 +00:00
628c2dba60
Add the last part that is needed for vectorization of if-converted code.
...
Added the code that actually performs the if-conversion during vectorization.
We can now vectorize this code:
for (int i=0; i<n; ++i) {
unsigned k = 0;
if (a[i] > b[i]) <------ IF inside the loop.
k = k * 5 + 3;
a[i] = k; <---- K is a phi node that becomes vector-select.
}
llvm-svn: 169217
2012-12-04 06:15:11 +00:00
ea3824f160
Add support for pointer induction variables even when there is no integer induction variable.
...
llvm-svn: 168558
2012-11-25 08:41:35 +00:00
c3c07e62e8
LoopVectorizer: Add initial support for pointer induction variables (for example: *dst++ = *src++).
...
At the moment we still require to have an integer induction variable (for example: i++).
llvm-svn: 168231
2012-11-17 00:27:03 +00:00
e6beec6765
Relax the restrictions on vector of pointer types, and vector getelementptr.
...
Previously in a vector of pointers, the pointer couldn't be any pointer type,
it had to be a pointer to an integer or floating point type. This is a hassle
for dragonegg because the GCC vectorizer happily produces vectors of pointers
where the pointer is a pointer to a struct or whatever. Vector getelementptr
was restricted to just one index, but now that vectors of pointers can have
any pointer type it is more natural to allow arbitrary vector getelementptrs.
There is however the issue of struct GEPs, where if each lane chose different
struct fields then from that point on each lane will be working down into
unrelated types. This seems like too much pain for too little gain, so when
you have a vector struct index all the elements are required to be the same.
llvm-svn: 167828
2012-11-13 12:59:33 +00:00
1cfef3e9ee
Add support for memory runtime check. When we can, we calculate array bounds.
...
If the arrays are found to be disjoint then we run the vectorized version of
the loop. If they are not, we run the scalar code.
llvm-svn: 167608
2012-11-09 07:09:44 +00:00
0914f0b262
Cost Model: add tables for some avx type-conversion hacks.
...
llvm-svn: 167480
2012-11-06 19:33:53 +00:00
ae79765676
Code Model: Improve the accuracy of the zext/sext/trunc vector cost estimation.
...
llvm-svn: 167412
2012-11-05 22:20:53 +00:00
7411623fd8
Implement the cost of abnormal x86 instruction lowering as a table.
...
llvm-svn: 167395
2012-11-05 19:32:46 +00:00
4cb8cdab5e
LoopVectorize: Preserve NSW, NUW and IsExact flags.
...
llvm-svn: 167174
2012-10-31 21:40:39 +00:00
6d7d39783d
Fix a bug in the cost calculation of vector casts. Detect situations where bitcasts cost zero.
...
llvm-svn: 167170
2012-10-31 20:52:26 +00:00
ce77ab0c24
LoopVectorize: Do not vectorize loops with tiny constant trip counts.
...
llvm-svn: 167101
2012-10-31 03:31:07 +00:00
ff7889196b
Add support for loops that don't start with Zero.
...
This is important for loops in the LAPACK test-suite.
These loops start at 1 because they are auto-converted from fortran.
llvm-svn: 167084
2012-10-31 00:45:26 +00:00
bc21aceb19
LoopVectorize: Add support for write-only loops when the write destination is a single pointer.
...
Speedup SciMark by 1%
llvm-svn: 167035
2012-10-30 18:36:45 +00:00
b3e8e688da
LoopVectorize: Fix a bug in the initialization of reduction variables. AND needs to start at all-one
...
while XOR, and OR need to start at zero.
llvm-svn: 167032
2012-10-30 18:12:36 +00:00
859366f93f
1. Fix a bug in getTypeConversion. When a *simple* type is split, we need to return the type of the split result.
...
2. Change the maximum vectorization width from 4 to 8.
3. A test for both.
llvm-svn: 166864
2012-10-27 04:11:32 +00:00
afae78edab
Refactor the VectorTargetTransformInfo interface.
...
Add getCostXXX calls for different families of opcodes, such as casts, arithmetic, cmp, etc.
Port the LoopVectorizer to the new API.
The LoopVectorizer now finds instructions which will remain uniform after vectorization. It uses this information when calculating the cost of these instructions.
llvm-svn: 166836
2012-10-26 23:49:28 +00:00
fcd1af344c
Move the target-specific tests, which require specific backends, to dirs that only run if the target is present.
...
llvm-svn: 166796
2012-10-26 18:52:01 +00:00
ede4fd4777
Fix the cost-model test.
...
llvm-svn: 166722
2012-10-25 21:42:50 +00:00
27d523580c
Add the cpu model to the test.
...
llvm-svn: 166718
2012-10-25 21:18:42 +00:00
5ffb049a55
Add support for additional reduction variables: AND, OR, XOR.
...
Patch by Paul Redmond <paul.redmond@intel.com>.
llvm-svn: 166649
2012-10-25 00:08:41 +00:00
4a87683a41
Implement a basic cost model for vector and scalar instructions.
...
llvm-svn: 166642
2012-10-24 23:47:38 +00:00
a721b21c64
LoopVectorizer: Add a basic cost model which uses the VTTI interface.
...
llvm-svn: 166620
2012-10-24 20:36:32 +00:00
5bed7b4fad
Use the AliasAnalysis isIdentifiedObj because it also understands mallocs and c++ news.
...
PR14158.
llvm-svn: 166491
2012-10-23 18:44:18 +00:00
1c7fc71e69
Don't crash if the load/store pointer is not a GEP.
...
Fix by Shivarama Rao <Shivarama.Rao@amd.com>
llvm-svn: 166427
2012-10-22 18:27:56 +00:00
03011f1393
Vectorizer: optimize the generation of selects. If the condition is uniform, generate a scalar-cond select (i1 as selector).
...
llvm-svn: 166409
2012-10-22 04:38:00 +00:00
fe88c67161
Fix a bug in the vectorization of wide load/store operations.
...
We used a SCEV to detect that A[X] is consecutive. We assumed that X was
the induction variable. But X can be any expression that uses the induction
for example: X = i + 2;
llvm-svn: 166388
2012-10-21 06:49:10 +00:00
c1679a95b6
Add support for reduction variables that do not start at zero.
...
This is important for nested-loop reductions such as :
In the innermost loop, the induction variable does not start with zero:
for (i = 0 .. n)
for (j = 0 .. m)
sum += ...
llvm-svn: 166387
2012-10-21 05:52:51 +00:00
7e1084d36c
Vectorizer: fix a bug in the classification of induction/reduction phis.
...
llvm-svn: 166384
2012-10-21 02:38:01 +00:00
e5dc57d4fb
Fix an infinite loop in the loop-vectorizer.
...
PR14134.
llvm-svn: 166379
2012-10-20 20:45:01 +00:00
d189b82a9b
Vectorize: teach cavVectorizeMemory to distinguish between A[i]+=x and A[B[i]]+=x.
...
If the pointer is consecutive then it is safe to read and write. If the pointer is non-loop-consecutive then
it is unsafe to vectorize it because we may hit an ordering issue.
llvm-svn: 166371
2012-10-20 08:26:33 +00:00
4f7f72702b
Vectorizer: Add support for loop reductions.
...
For example:
for (i=0; i<n; i++)
sum += A[i] + B[i] + i;
llvm-svn: 166351
2012-10-19 23:05:40 +00:00
ced93f3a05
vectorizer: Add support for reading and writing from the same memory location.
...
llvm-svn: 166255
2012-10-19 01:24:18 +00:00
b52f717411
Vectorizer: Add support for loops with an unknown count. For example:
...
for (i=0; i<n; i++){
a[i] = b[i+1] + c[i+3];
}
llvm-svn: 166165
2012-10-18 05:29:12 +00:00
6b94c2a09b
Add a loop vectorizer.
...
llvm-svn: 166112
2012-10-17 18:25:06 +00:00
Nadav Rotem