The pattern matching optimization of Polly detects and optimizes dense general
matrix-matrix multiplication. The generated code is close to high performance
implementations of matrix-matrix multiplications, which are contained in
manually tuned libraries. The described pattern matching optimization is
a particular case of tensor contraction optimization, which was
introduced in [1].
This patch generalizes the pattern matching to the case of tensor contractions
using the form of data dependencies and memory accesses produced by tensor
contractions [1].
Optimization of tensor contractions will be added in the next patch. Following
the ideas introduced in [2], it will logically represent tensor contraction
operands as matrix multiplication operands and use an approach for
optimization of matrix-matrix multiplications.
[1] - Gareev R., Grosser T., Kruse M. High-Performance Generalized Tensor
Operations: A Compiler-Oriented Approach // ACM Transactions on
Architecture and Code Optimization (TACO). 2018. Vol. 15, no. 3.
P. 34:1–34:27. DOI: 10.1145/3235029.
[2] - Matthews D. High-Performance Tensor Contraction without BLAS // SIAM
Journal on Scientific Computing. 2018. Vol. 40, no. 1. P. C 1—C 24.
DOI: 110.1137/16m108968x.
Reviewed By: Meinersbur
Differential Revision: https://reviews.llvm.org/D114336
The copy statements inserted by the matrix-multiplication optimization
introduce new dependencies between the copy statements and other
statements. As a result, the DependenceInfo must be recomputed.
Not recomputing them caused IslAstInfo to deduce that some loops are
parallel but cause race conditions when accessing the packed arrays.
As a result, matrix-matrix multiplication currently cannot be
parallelized.
Also see discussion at https://reviews.llvm.org/D125202
This make is obivious that a class was not intended to be derived from.
NPM analysis pass can unfortunately not marked as final because they are
derived from a llvm::Checker<T> template internally by the NPM.
Also normalize the use of classes/structs
* NPM passes are structs
* Legacy passes are classes
* structs that have methods and are not a visitor pattern are classes
* structs have public inheritance by default, remove "public" keyword
* Use typedef'ed type instead of inline forward declaration
The `opt -analyze` option only works with the legacy pass manager and might be removed in the future, as explained in llvm.org/PR53733. This patch introduced -polly-print-* passes that print what the pass would print with the `-analyze` option and replaces all uses of `-analyze` in the regression tests.
There are two exceptions: `CodeGen\single_loop_param_less_equal.ll` and `CodeGen\loop_with_condition_nested.ll` use `-analyze on the `-loops` pass which is not part of Polly.
Reviewed By: aeubanks
Differential Revision: https://reviews.llvm.org/D120782
A prevectorized loop may contain multiple statements, in which case
isl_schedule_node_band_sink will sink the vector band to multiple
leaves. Instead of statically assuming a specific tree structure after
sinking, add a SIMD marker to all inner bands.
Fixes llvm.org/PR52637
This is part of an effort to reduce the differences between the custom C++ bindings used right now by polly in lib/External/isl/include/isl/isl-noxceptions.h and the official isl C++ interface.
In the official interface the type `isl::size` cannot be casted to an unsigned without previously having checked if it contains a valid value with the function `isl::size::is_error()`.
For this reason two helping functions have been added:
- `IslAssert`: assert that no errors are present in debug builds and just disables the mandatory error check in non-debug builds
- `unisgnedFromIslSIze`: cast the `isl::size` object to `unsigned`
Changes made:
- Add the functions `IslAssert` and `unsignedFromIslSize`
- Add the utility function `rangeIslSize()`
- Retype `MaxDisjunctsInDomain` from `int` to `unsigned`
- Retype `RunTimeChecksMaxAccessDisjuncts` from `int` to `unsigned`
- Retype `MaxDimensionsInAccessRange` from `int` to `unsigned`
- Replaced some usages of `isl_size` to `unsigned` since we aim not to use `isl_size` anymore
- `isl-noexceptions.h` has been generated by e704f73c88
No functional change intended.
Reviewed By: Meinersbur
Differential Revision: https://reviews.llvm.org/D113101
When the option -polly-loopfusion-greedy is set, the ScheduleOptimizer
tries to aggressively fuse any band it can and does not violate any
dependences.
As part if the implementation, the functionalty for copying a band
into an new schedule was extracted out of the ScheduleTreeRewriter.
This is a simple version without the possibility to define distribute
points or followup-transformations. However, it is the first
transformation that has to check whether the transformation is correct.
It interprets the same metadata as the LoopDistribute pass.
Re-apply after revert in c7bcd72a38 with
fix: Take isBand out of #ifndef NDEBUG since it now is used
unconditionally.
The name of the option is misleading and has been renamed by isl to
"serialize-sccs". Instead of also renaming the option, remove it.
The option is still accessible using
-polly-isl-arg=--no-schedule-serialize-sccs
This is a simple version without the possibility to define distribute
points or followup-transformations. However, it is the first
transformation that has to check whether the transformation is correct.
It interprets the same metadata as the LoopDistribute pass.
This is part of an effort to reduce the differences between the custom C++ bindings used right now by polly in `lib/External/isl/include/isl/isl-noxceptions.h` and the official isl C++ interface.
With this commit we are moving from the `polly-generator` branch to the `new-polly-generator` branch that is more mantainable and is based on the official C++ interface `cpp-checked.h`.
Changes made:
- There are now many sublcasses for `isl::ast_node` representing different isl types. Use `isl::ast_node_for`, `isl::ast_node_user`, `isl::ast_node_block` and `isl::ast_node_mark` where needed.
- There are now many sublcasses for `isl::schedule_node` representing different isl types. Use `isl::schedule_node_mark`, `isl::schedule_node_extension`, `isl::schedule_node_band` and `isl::schedule_node_filter` where needed.
- Replace the `isl::*::dump` with `dumpIslObj` since the isl dump method is not exposed in the C++ interface.
- `isl::schedule_node::get_child` has been renamed to `isl::schedule_node::child`
- `isl::pw_multi_aff::get_pw_aff` has been renamed to `isl::pw_multi_aff::at`
- The constructor `isl::union_map(isl::union_pw_multi_aff)` has been replaced with the static method `isl::union_map::from()`
- Replace usages of `isl::val::add_ui` with `isl::val::add`
- `isl::union_set_list::alloc` is now a constructor
- All the `isl_size` values are now wrapped inside the class `isl::size` use `isl::size::release` to get the internal `isl_size` value where needed.
- `isl-noexceptions.h` has been generated by 73f5ed1f4d
No functional change intended.
Reviewed By: Meinersbur
Differential Revision: https://reviews.llvm.org/D107225
This is part of an effort to reduce the differences between the custom C++ bindings used right now by polly in `lib/External/isl/include/isl/isl-noxceptions.h` and the official isl C++ interface.
Changes made:
- Use `isl::*::ctx()` instead of `isl::*::get_ctx()` (for example `isl::space::ctx()` instead of `isl::space::get_ctx()`)
- Add `isl::` namespace in front of isl types to avoid confusion (for example `isl::space::ctx` and `isl::ctx`
- `isl-noexceptions.h` has been generated by this b64e33c62d
Reviewed By: Meinersbur
Differential Revision: https://reviews.llvm.org/D105691
This is part of an effort to reduce the differences between the custom C++ bindings used right now by polly in `lib/External/isl/include/isl/isl-noxceptions.h` and the official isl C++ interface.
Changes made:
- Removing explicit operator bool() from all the classes in the isl C++ bindings.
- Replace each call to operator bool() to method `is_null()`.
- isl-noexceptions.h has been generated by this 27396daac5
Reviewed By: Meinersbur
Differential Revision: https://reviews.llvm.org/D103976
[Polly][Isl] Removing nullptr constructor from C++ bindings. NFC.
This is part of an effort to reduce the differences between the custom C++ bindings used right now by polly in `lib/External/isl/include/isl/isl-noxceptions.h` and the official isl C++ interface.
Changes made:
- Removed `std::nullptr_t` constructor from all the classes in the isl C++ bindings.
- `isl-noexceptions.h` has been generated by this a7e00bea38
Reviewed By: Meinersbur
Differential Revision: https://reviews.llvm.org/D103751
[Polly][Isl] Removing nullptr constructor from C++ bindings. NFC.
This is part of an effort to reduce the differences between the custom C++ bindings used right now by polly in `lib/External/isl/include/isl/isl-noxceptions.h` and the official isl C++ interface.
Changes made:
- Removed `std::nullptr_t` constructor from all the classes in the isl C++ bindings.
- `isl-noexceptions.h` has been generated by this a7e00bea38
Reviewed By: Meinersbur
Differential Revision: https://reviews.llvm.org/D103751
Functions shared between generalized matrix-multiplication optimization
and other post-reschedule optimizations (tiling, prevect) are moved into
the schedule tree transformation utility ScheduleTreeTransform.
This produced a compile error with GCC:
llvm-project/polly/lib/Transform/ScheduleOptimizer.cpp:1220:49: error: cannot convert ‘bool’ to ‘llvm::TargetTransformInfo::RegisterKind’
1220 | RegisterBitwidth = TTI->getRegisterBitWidth(true);
Make Polly look for unrolling metadata (https://llvm.org/docs/TransformMetadata.html#loop-unrolling) that is usually only interpreted by the LoopUnroll pass and apply it to the SCoP's schedule.
While not that useful by itself (there already is an unroll pass), it introduces mechanism to apply arbitrary loop transformation directives in arbitrary order to the schedule. Transformations are applied until no more directives are found. Since ISL's rescheduling would discard the manual transformations and it is assumed that when the user specifies the sequence of transformations, they do not want any other transformations to apply. Applying user-directed transformations can be controlled using the `-polly-pragma-based-opts` switch and is enabled by default.
This does not influence the SCoP detection heuristic. As a consequence, loop that do not fulfill SCoP requirements or the initial profitability heuristic will be ignored. `-polly-process-unprofitable` can be used to disable the latter.
Other than manually editing the IR, there is currently no way for the user to add loop transformations in an order other than the order in the default pipeline, or transformations other than the one supported by clang's LoopHint. See the `unroll_double.ll` test as example that clang currently is unable to emit. My own extension of `#pragma clang loop` allowing an arbitrary order and additional transformations is available here: https://github.com/meinersbur/llvm-project/tree/pragma-clang-loop. An effort to upstream this functionality as `#pragma clang transform` (because `#pragma clang loop` has an implicit transformation order defined by the loop pipeline) is D69088.
Additional transformations from my downstream pragma-clang-loop branch are tiling, interchange, reversal, unroll-and-jam, thread-parallelization and array packing. Unroll was chosen because it uses already-defined metadata and does not require correctness checks.
Reviewed By: sebastiankreutzer
Differential Revision: https://reviews.llvm.org/D97977
Regenerate the C++ wrapper header from the current isl version's
headers.
The most notable change is that some dimension sizes are represented by
an isl_size (instead of unsigned), which is a signed int. Additionally,
some function may return -1 in case of an error which already had been
fixed in the past. The C++ may no return -1 instead of UINT_MAX which
caused the problems.
Some types in Polly had been changed from unsigned to isl_size
(that were not already auto) and some loops/comparision had to be
changed to avoid unsigned/signed comparison warnings.
These are implementation details of the IslScheduleOptimizer pass
implementation and not use anywhere else. Hence, we can move them to the
cpp file and into an anonymous namespace.
Only getPartialTilePrefixes is, aside from the pass itself, used
externally (by the ScheduleOptimizerTest) and moved into the polly
namespace.
The schedule of a fused loop has one isl_space per statement, such that
a conversion to a isl_map fails. However, the prevectorization is
interested in the schedule space only: Converting to the non-union
representation only after extracting the schedule range fixes the problem.
This fixes llvm.org/PR46578
The member LastSchedule was never set, such that printScop would always
print "n/a" instead of the last schedule.
To ensure that the isl_ctx lives as least as long as the stored
schedule, also store a shared_ptr.
Also set the schedule tree output style to ISL_YAML_STYLE_BLOCK to avoid
printing everything on a single line.
`opt -polly-opt-isl -analyze` will be used in the next commit.
../polly/lib/Transform/ScheduleOptimizer.cpp:812:54: warning: comparison of integers of different signs: 'isl_size' (aka 'int') and 'const unsigned int' [-Wsign-compare]
isl_schedule_node_band_n_member(Node.get()) >
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^
This file lists every pass in LLVM, and is included by Pass.h, which is
very popular. Every time we add, remove, or rename a pass in LLVM, it
caused lots of recompilation.
I found this fact by looking at this table, which is sorted by the
number of times a file was changed over the last 100,000 git commits
multiplied by the number of object files that depend on it in the
current checkout:
recompiles touches affected_files header
342380 95 3604 llvm/include/llvm/ADT/STLExtras.h
314730 234 1345 llvm/include/llvm/InitializePasses.h
307036 118 2602 llvm/include/llvm/ADT/APInt.h
213049 59 3611 llvm/include/llvm/Support/MathExtras.h
170422 47 3626 llvm/include/llvm/Support/Compiler.h
162225 45 3605 llvm/include/llvm/ADT/Optional.h
158319 63 2513 llvm/include/llvm/ADT/Triple.h
140322 39 3598 llvm/include/llvm/ADT/StringRef.h
137647 59 2333 llvm/include/llvm/Support/Error.h
131619 73 1803 llvm/include/llvm/Support/FileSystem.h
Before this change, touching InitializePasses.h would cause 1345 files
to recompile. After this change, touching it only causes 550 compiles in
an incremental rebuild.
Reviewers: bkramer, asbirlea, bollu, jdoerfert
Differential Revision: https://reviews.llvm.org/D70211
lib/Transform/ScheduleOptimizer.cpp fails to compile on Solaris, both on the 9.x
branch (first noticed when running test-release.sh without -no-polly) and on trunk:
/var/llvm/llvm-9.0.0-rc4/rc4/llvm.src/tools/polly/lib/Transform/ScheduleOptimizer.cpp: In function ‘MicroKernelParamsTy getMicroKernelParams(const llvm::TargetTransformInfo*, polly::MatMulInfoTy)’:
/var/llvm/llvm-9.0.0-rc4/rc4/llvm.src/tools/polly/lib/Transform/ScheduleOptimizer.cpp:914:62: error: call of overloaded ‘sqrt(long unsigned int)’ is ambiguous
914 | ceil(sqrt(Nvec * LatencyVectorFma * ThroughputVectorFma) / Nvec) * Nvec;
| ^
In file included from /usr/gcc/9/lib/gcc/x86_64-pc-solaris2.11/9.1.0/include-fixed/math.h:24,
from /usr/gcc/9/include/c++/9.1.0/cmath:45,
from /var/llvm/llvm-9.0.0-rc4/rc4/llvm.src/include/llvm-c/DataTypes.h:28,
from /var/llvm/llvm-9.0.0-rc4/rc4/llvm.src/include/llvm/Support/DataTypes.h:16,
from /var/llvm/llvm-9.0.0-rc4/rc4/llvm.src/include/llvm/ADT/Hashing.h:47,
from /var/llvm/llvm-9.0.0-rc4/rc4/llvm.src/include/llvm/ADT/ArrayRef.h:12,
from /var/llvm/llvm-9.0.0-rc4/rc4/llvm.src/tools/polly/include/polly/ScheduleOptimizer.h:12,
from /var/llvm/llvm-9.0.0-rc4/rc4/llvm.src/tools/polly/lib/Transform/ScheduleOptimizer.cpp:48:
/usr/gcc/9/lib/gcc/x86_64-pc-solaris2.11/9.1.0/include-fixed/iso/math_iso.h:220:21: note: candidate: ‘long double std::sqrt(long double)’
220 | inline long double sqrt(long double __X) { return __sqrtl(__X); }
| ^~~~
/usr/gcc/9/lib/gcc/x86_64-pc-solaris2.11/9.1.0/include-fixed/iso/math_iso.h:186:15:
note: candidate: ‘float std::sqrt(float)’
186 | inline float sqrt(float __X) { return __sqrtf(__X); }
| ^~~~
/usr/gcc/9/lib/gcc/x86_64-pc-solaris2.11/9.1.0/include-fixed/iso/math_iso.h:74:15:
note: candidate: ‘double std::sqrt(double)’
74 | extern double sqrt __P((double));
| ^~~~
/var/llvm/llvm-9.0.0-rc4/rc4/llvm.src/tools/polly/lib/Transform/ScheduleOptimizer.cpp:915:67:
error: call of overloaded ‘ceil(long unsigned int)’ is ambiguous
915 | int Mr = ceil(Nvec * LatencyVectorFma * ThroughputVectorFma / Nr);
| ^
In file included from /usr/gcc/9/lib/gcc/x86_64-pc-solaris2.11/9.1.0/include-fixed/math.h:24,
from /usr/gcc/9/include/c++/9.1.0/cmath:45,
from /var/llvm/llvm-9.0.0-rc4/rc4/llvm.src/include/llvm-c/DataTypes.h:28,
from /var/llvm/llvm-9.0.0-rc4/rc4/llvm.src/include/llvm/Support/DataTypes.h:16,
from /var/llvm/llvm-9.0.0-rc4/rc4/llvm.src/include/llvm/ADT/Hashing.h:47,
from /var/llvm/llvm-9.0.0-rc4/rc4/llvm.src/include/llvm/ADT/ArrayRef.h:12,
from /var/llvm/llvm-9.0.0-rc4/rc4/llvm.src/tools/polly/include/polly/ScheduleOptimizer.h:12,
from /var/llvm/llvm-9.0.0-rc4/rc4/llvm.src/tools/polly/lib/Transform/ScheduleOptimizer.cpp:48:
/usr/gcc/9/lib/gcc/x86_64-pc-solaris2.11/9.1.0/include-fixed/iso/math_iso.h:196:21: note: candidate: ‘long double std::ceil(long double)’
196 | inline long double ceil(long double __X) { return __ceill(__X); }
| ^~~~
/usr/gcc/9/lib/gcc/x86_64-pc-solaris2.11/9.1.0/include-fixed/iso/math_iso.h:160:15:
note: candidate: ‘float std::ceil(float)’
160 | inline float ceil(float __X) { return __ceilf(__X); }
| ^~~~
/usr/gcc/9/lib/gcc/x86_64-pc-solaris2.11/9.1.0/include-fixed/iso/math_iso.h:76:15:
note: candidate: ‘double std::ceil(double)’
76 | extern double ceil __P((double));
| ^~~~
Fixed by adding casts to disambiguate, checked that it now compiles on both
amd64-pc-solaris2.11 and sparcv9-sun-solaris2.11 and on x86_64-pc-linux-gnu.
Differential Revision: https://reviews.llvm.org/D67442
llvm-svn: 371825
Extension nodes make schedule trees are less flexible: Many operations,
such as rescheduling, do not work on such schedule trees with extension.
As such, some functionality such as determining parallel loops in isl's
AST are disabled.
Currently, only the pattern-matching generalized matrix-matrix
multiplication optimization adds extension nodes (to add copy-in
statements).
This patch removes all extension nodes as the last step of the schedule
optimization by hoisting the extension node's added domain up to the
root domain node. All following passes can assume that schedule trees
work without restrictions, including the parallelism test. Mark the
outermost loop of the optimized matrix-matrix multiplication as parallel
such that -polly-parallel is able to parallelize that loop.
Differential Revision: https://reviews.llvm.org/D58202
llvm-svn: 362257
This removes unused includes (and forward declarations) as
suggested by include-what-you-use. If a transitive include of a removed
include is required to compile a file, I added the required header (or
forward declaration if suggested by include-what-you-use).
This should reduce compilation time and reduce the number of iterative
recompilations when a header was changed.
llvm-svn: 357209
to reflect the new license.
We understand that people may be surprised that we're moving the header
entirely to discuss the new license. We checked this carefully with the
Foundation's lawyer and we believe this is the correct approach.
Essentially, all code in the project is now made available by the LLVM
project under our new license, so you will see that the license headers
include that license only. Some of our contributors have contributed
code under our old license, and accordingly, we have retained a copy of
our old license notice in the top-level files in each project and
repository.
llvm-svn: 351636
The DEBUG() macro is very generic so it might clash with other projects.
The renaming was done as follows:
- git grep -l 'DEBUG' | xargs sed -i 's/\bDEBUG\s\?(/LLVM_DEBUG(/g'
- git diff -U0 master | ../clang/tools/clang-format/clang-format-diff.py -i -p1 -style LLVM
Differential Revision: https://reviews.llvm.org/D44978
llvm-svn: 332352
As part of this cleanup a couple of unnecessary isl::manage(obj.copy()) pattern
are eliminated as well.
We checked for all potential cleanups by scanning for:
"grep -R isl::manage\( lib/ | grep copy"
llvm-svn: 325558
Roman Gareev