442 lines
16 KiB
ReStructuredText
442 lines
16 KiB
ReStructuredText
.. _transformation-metadata:
|
|
|
|
============================
|
|
Code Transformation Metadata
|
|
============================
|
|
|
|
.. contents::
|
|
:local:
|
|
|
|
Overview
|
|
========
|
|
|
|
LLVM transformation passes can be controlled by attaching metadata to
|
|
the code to transform. By default, transformation passes use heuristics
|
|
to determine whether or not to perform transformations, and when doing
|
|
so, other details of how the transformations are applied (e.g., which
|
|
vectorization factor to select).
|
|
Unless the optimizer is otherwise directed, transformations are applied
|
|
conservatively. This conservatism generally allows the optimizer to
|
|
avoid unprofitable transformations, but in practice, this results in the
|
|
optimizer not applying transformations that would be highly profitable.
|
|
|
|
Frontends can give additional hints to LLVM passes on which
|
|
transformations they should apply. This can be additional knowledge that
|
|
cannot be derived from the emitted IR, or directives passed from the
|
|
user/programmer. OpenMP pragmas are an example of the latter.
|
|
|
|
If any such metadata is dropped from the program, the code's semantics
|
|
must not change.
|
|
|
|
Metadata on Loops
|
|
=================
|
|
|
|
Attributes can be attached to loops as described in :ref:`llvm.loop`.
|
|
Attributes can describe properties of the loop, disable transformations,
|
|
force specific transformations and set transformation options.
|
|
|
|
Because metadata nodes are immutable (with the exception of
|
|
``MDNode::replaceOperandWith`` which is dangerous to use on uniqued
|
|
metadata), in order to add or remove a loop attributes, a new ``MDNode``
|
|
must be created and assigned as the new ``llvm.loop`` metadata. Any
|
|
connection between the old ``MDNode`` and the loop is lost. The
|
|
``llvm.loop`` node is also used as LoopID (``Loop::getLoopID()``), i.e.
|
|
the loop effectively gets a new identifier. For instance,
|
|
``llvm.mem.parallel_loop_access`` references the LoopID. Therefore, if
|
|
the parallel access property is to be preserved after adding/removing
|
|
loop attributes, any ``llvm.mem.parallel_loop_access`` reference must be
|
|
updated to the new LoopID.
|
|
|
|
Transformation Metadata Structure
|
|
=================================
|
|
|
|
Some attributes describe code transformations (unrolling, vectorizing,
|
|
loop distribution, etc.). They can either be a hint to the optimizer
|
|
that a transformation might be beneficial, instruction to use a specific
|
|
option, , or convey a specific request from the user (such as
|
|
``#pragma clang loop`` or ``#pragma omp simd``).
|
|
|
|
If a transformation is forced but cannot be carried-out for any reason,
|
|
an optimization-missed warning must be emitted. Semantic information
|
|
such as a transformation being safe (e.g.
|
|
``llvm.mem.parallel_loop_access``) can be unused by the optimizer
|
|
without generating a warning.
|
|
|
|
Unless explicitly disabled, any optimization pass may heuristically
|
|
determine whether a transformation is beneficial and apply it. If
|
|
metadata for another transformation was specified, applying a different
|
|
transformation before it might be inadvertent due to being applied on a
|
|
different loop or the loop not existing anymore. To avoid having to
|
|
explicitly disable an unknown number of passes, the attribute
|
|
``llvm.loop.disable_nonforced`` disables all optional, high-level,
|
|
restructuring transformations.
|
|
|
|
The following example avoids the loop being altered before being
|
|
vectorized, for instance being unrolled.
|
|
|
|
.. code-block:: llvm
|
|
|
|
br i1 %exitcond, label %for.exit, label %for.header, !llvm.loop !0
|
|
...
|
|
!0 = distinct !{!0, !1, !2}
|
|
!1 = !{!"llvm.loop.vectorize.enable", i1 true}
|
|
!2 = !{!"llvm.loop.disable_nonforced"}
|
|
|
|
After a transformation is applied, follow-up attributes are set on the
|
|
transformed and/or new loop(s). This allows additional attributes
|
|
including followup-transformations to be specified. Specifying multiple
|
|
transformations in the same metadata node is possible for compatibility
|
|
reasons, but their execution order is undefined. For instance, when
|
|
``llvm.loop.vectorize.enable`` and ``llvm.loop.unroll.enable`` are
|
|
specified at the same time, unrolling may occur either before or after
|
|
vectorization.
|
|
|
|
As an example, the following instructs a loop to be vectorized and only
|
|
then unrolled.
|
|
|
|
.. code-block:: llvm
|
|
|
|
!0 = distinct !{!0, !1, !2, !3}
|
|
!1 = !{!"llvm.loop.vectorize.enable", i1 true}
|
|
!2 = !{!"llvm.loop.disable_nonforced"}
|
|
!3 = !{!"llvm.loop.vectorize.followup_vectorized", !{"llvm.loop.unroll.enable"}}
|
|
|
|
If, and only if, no followup is specified, the pass may add attributes itself.
|
|
For instance, the vectorizer adds a ``llvm.loop.isvectorized`` attribute and
|
|
all attributes from the original loop excluding its loop vectorizer
|
|
attributes. To avoid this, an empty followup attribute can be used, e.g.
|
|
|
|
.. code-block:: llvm
|
|
|
|
!3 = !{!"llvm.loop.vectorize.followup_vectorized"}
|
|
|
|
The followup attributes of a transformation that cannot be applied will
|
|
never be added to a loop and are therefore effectively ignored. This means
|
|
that any followup-transformation in such attributes requires that its
|
|
prior transformations are applied before the followup-transformation.
|
|
The user should receive a warning about the first transformation in the
|
|
transformation chain that could not be applied if it a forced
|
|
transformation. All following transformations are skipped.
|
|
|
|
Pass-Specific Transformation Metadata
|
|
=====================================
|
|
|
|
Transformation options are specific to each transformation. In the
|
|
following, we present the model for each LLVM loop optimization pass and
|
|
the metadata to influence them.
|
|
|
|
Loop Vectorization and Interleaving
|
|
-----------------------------------
|
|
|
|
Loop vectorization and interleaving is interpreted as a single
|
|
transformation. It is interpreted as forced if
|
|
``!{"llvm.loop.vectorize.enable", i1 true}`` is set.
|
|
|
|
Assuming the pre-vectorization loop is
|
|
|
|
.. code-block:: c
|
|
|
|
for (int i = 0; i < n; i+=1) // original loop
|
|
Stmt(i);
|
|
|
|
then the code after vectorization will be approximately (assuming an
|
|
SIMD width of 4):
|
|
|
|
.. code-block:: c
|
|
|
|
int i = 0;
|
|
if (rtc) {
|
|
for (; i + 3 < n; i+=4) // vectorized/interleaved loop
|
|
Stmt(i:i+3);
|
|
}
|
|
for (; i < n; i+=1) // epilogue loop
|
|
Stmt(i);
|
|
|
|
where ``rtc`` is a generated runtime check.
|
|
|
|
``llvm.loop.vectorize.followup_vectorized`` will set the attributes for
|
|
the vectorized loop. If not specified, ``llvm.loop.isvectorized`` is
|
|
combined with the original loop's attributes to avoid it being
|
|
vectorized multiple times.
|
|
|
|
``llvm.loop.vectorize.followup_epilogue`` will set the attributes for
|
|
the remainder loop. If not specified, it will have the original loop's
|
|
attributes combined with ``llvm.loop.isvectorized`` and
|
|
``llvm.loop.unroll.runtime.disable`` (unless the original loop already
|
|
has unroll metadata).
|
|
|
|
The attributes specified by ``llvm.loop.vectorize.followup_all`` are
|
|
added to both loops.
|
|
|
|
When using a follow-up attribute, it replaces any automatically deduced
|
|
attributes for the generated loop in question. Therefore it is
|
|
recommended to add ``llvm.loop.isvectorized`` to
|
|
``llvm.loop.vectorize.followup_all`` which avoids that the loop
|
|
vectorizer tries to optimize the loops again.
|
|
|
|
Loop Unrolling
|
|
--------------
|
|
|
|
Unrolling is interpreted as forced any ``!{!"llvm.loop.unroll.enable"}``
|
|
metadata or option (``llvm.loop.unroll.count``, ``llvm.loop.unroll.full``)
|
|
is present. Unrolling can be full unrolling, partial unrolling of a loop
|
|
with constant trip count or runtime unrolling of a loop with a trip
|
|
count unknown at compile-time.
|
|
|
|
If the loop has been unrolled fully, there is no followup-loop. For
|
|
partial/runtime unrolling, the original loop of
|
|
|
|
.. code-block:: c
|
|
|
|
for (int i = 0; i < n; i+=1) // original loop
|
|
Stmt(i);
|
|
|
|
is transformed into (using an unroll factor of 4):
|
|
|
|
.. code-block:: c
|
|
|
|
int i = 0;
|
|
for (; i + 3 < n; i+=4) { // unrolled loop
|
|
Stmt(i);
|
|
Stmt(i+1);
|
|
Stmt(i+2);
|
|
Stmt(i+3);
|
|
}
|
|
for (; i < n; i+=1) // remainder loop
|
|
Stmt(i);
|
|
|
|
``llvm.loop.unroll.followup_unrolled`` will set the loop attributes of
|
|
the unrolled loop. If not specified, the attributes of the original loop
|
|
without the ``llvm.loop.unroll.*`` attributes are copied and
|
|
``llvm.loop.unroll.disable`` added to it.
|
|
|
|
``llvm.loop.unroll.followup_remainder`` defines the attributes of the
|
|
remainder loop. If not specified the remainder loop will have no
|
|
attributes. The remainder loop might not be present due to being fully
|
|
unrolled in which case this attribute has no effect.
|
|
|
|
Attributes defined in ``llvm.loop.unroll.followup_all`` are added to the
|
|
unrolled and remainder loops.
|
|
|
|
To avoid that the partially unrolled loop is unrolled again, it is
|
|
recommended to add ``llvm.loop.unroll.disable`` to
|
|
``llvm.loop.unroll.followup_all``. If no follow-up attribute specified
|
|
for a generated loop, it is added automatically.
|
|
|
|
Unroll-And-Jam
|
|
--------------
|
|
|
|
Unroll-and-jam uses the following transformation model (here with an
|
|
unroll factor if 2). Currently, it does not support a fallback version
|
|
when the transformation is unsafe.
|
|
|
|
.. code-block:: c
|
|
|
|
for (int i = 0; i < n; i+=1) { // original outer loop
|
|
Fore(i);
|
|
for (int j = 0; j < m; j+=1) // original inner loop
|
|
SubLoop(i, j);
|
|
Aft(i);
|
|
}
|
|
|
|
.. code-block:: c
|
|
|
|
int i = 0;
|
|
for (; i + 1 < n; i+=2) { // unrolled outer loop
|
|
Fore(i);
|
|
Fore(i+1);
|
|
for (int j = 0; j < m; j+=1) { // unrolled inner loop
|
|
SubLoop(i, j);
|
|
SubLoop(i+1, j);
|
|
}
|
|
Aft(i);
|
|
Aft(i+1);
|
|
}
|
|
for (; i < n; i+=1) { // remainder outer loop
|
|
Fore(i);
|
|
for (int j = 0; j < m; j+=1) // remainder inner loop
|
|
SubLoop(i, j);
|
|
Aft(i);
|
|
}
|
|
|
|
``llvm.loop.unroll_and_jam.followup_outer`` will set the loop attributes
|
|
of the unrolled outer loop. If not specified, the attributes of the
|
|
original outer loop without the ``llvm.loop.unroll.*`` attributes are
|
|
copied and ``llvm.loop.unroll.disable`` added to it.
|
|
|
|
``llvm.loop.unroll_and_jam.followup_inner`` will set the loop attributes
|
|
of the unrolled inner loop. If not specified, the attributes of the
|
|
original inner loop are used unchanged.
|
|
|
|
``llvm.loop.unroll_and_jam.followup_remainder_outer`` sets the loop
|
|
attributes of the outer remainder loop. If not specified it will not
|
|
have any attributes. The remainder loop might not be present due to
|
|
being fully unrolled.
|
|
|
|
``llvm.loop.unroll_and_jam.followup_remainder_inner`` sets the loop
|
|
attributes of the inner remainder loop. If not specified it will have
|
|
the attributes of the original inner loop. It the outer remainder loop
|
|
is unrolled, the inner remainder loop might be present multiple times.
|
|
|
|
Attributes defined in ``llvm.loop.unroll_and_jam.followup_all`` are
|
|
added to all of the aforementioned output loops.
|
|
|
|
To avoid that the unrolled loop is unrolled again, it is
|
|
recommended to add ``llvm.loop.unroll.disable`` to
|
|
``llvm.loop.unroll_and_jam.followup_all``. It suppresses unroll-and-jam
|
|
as well as an additional inner loop unrolling. If no follow-up
|
|
attribute specified for a generated loop, it is added automatically.
|
|
|
|
Loop Distribution
|
|
-----------------
|
|
|
|
The LoopDistribution pass tries to separate vectorizable parts of a loop
|
|
from the non-vectorizable part (which otherwise would make the entire
|
|
loop non-vectorizable). Conceptually, it transforms a loop such as
|
|
|
|
.. code-block:: c
|
|
|
|
for (int i = 1; i < n; i+=1) { // original loop
|
|
A[i] = i;
|
|
B[i] = 2 + B[i];
|
|
C[i] = 3 + C[i - 1];
|
|
}
|
|
|
|
into the following code:
|
|
|
|
.. code-block:: c
|
|
|
|
if (rtc) {
|
|
for (int i = 1; i < n; i+=1) // coincident loop
|
|
A[i] = i;
|
|
for (int i = 1; i < n; i+=1) // coincident loop
|
|
B[i] = 2 + B[i];
|
|
for (int i = 1; i < n; i+=1) // sequential loop
|
|
C[i] = 3 + C[i - 1];
|
|
} else {
|
|
for (int i = 1; i < n; i+=1) { // fallback loop
|
|
A[i] = i;
|
|
B[i] = 2 + B[i];
|
|
C[i] = 3 + C[i - 1];
|
|
}
|
|
}
|
|
|
|
where ``rtc`` is a generated runtime check.
|
|
|
|
``llvm.loop.distribute.followup_coincident`` sets the loop attributes of
|
|
all loops without loop-carried dependencies (i.e. vectorizable loops).
|
|
There might be more than one such loops. If not defined, the loops will
|
|
inherit the original loop's attributes.
|
|
|
|
``llvm.loop.distribute.followup_sequential`` sets the loop attributes of the
|
|
loop with potentially unsafe dependencies. There should be at most one
|
|
such loop. If not defined, the loop will inherit the original loop's
|
|
attributes.
|
|
|
|
``llvm.loop.distribute.followup_fallback`` defines the loop attributes
|
|
for the fallback loop, which is a copy of the original loop for when
|
|
loop versioning is required. If undefined, the fallback loop inherits
|
|
all attributes from the original loop.
|
|
|
|
Attributes defined in ``llvm.loop.distribute.followup_all`` are added to
|
|
all of the aforementioned output loops.
|
|
|
|
It is recommended to add ``llvm.loop.disable_nonforced`` to
|
|
``llvm.loop.distribute.followup_fallback``. This avoids that the
|
|
fallback version (which is likely never executed) is further optimized
|
|
which would increase the code size.
|
|
|
|
Versioning LICM
|
|
---------------
|
|
|
|
The pass hoists code out of loops that are only loop-invariant when
|
|
dynamic conditions apply. For instance, it transforms the loop
|
|
|
|
.. code-block:: c
|
|
|
|
for (int i = 0; i < n; i+=1) // original loop
|
|
A[i] = B[0];
|
|
|
|
into:
|
|
|
|
.. code-block:: c
|
|
|
|
if (rtc) {
|
|
auto b = B[0];
|
|
for (int i = 0; i < n; i+=1) // versioned loop
|
|
A[i] = b;
|
|
} else {
|
|
for (int i = 0; i < n; i+=1) // unversioned loop
|
|
A[i] = B[0];
|
|
}
|
|
|
|
The runtime condition (``rtc``) checks that the array ``A`` and the
|
|
element `B[0]` do not alias.
|
|
|
|
Currently, this transformation does not support followup-attributes.
|
|
|
|
Loop Interchange
|
|
----------------
|
|
|
|
Currently, the ``LoopInterchange`` pass does not use any metadata.
|
|
|
|
Ambiguous Transformation Order
|
|
==============================
|
|
|
|
If there multiple transformations defined, the order in which they are
|
|
executed depends on the order in LLVM's pass pipeline, which is subject
|
|
to change. The default optimization pipeline (anything higher than
|
|
``-O0``) has the following order.
|
|
|
|
When using the legacy pass manager:
|
|
|
|
- LoopInterchange (if enabled)
|
|
- SimpleLoopUnroll/LoopFullUnroll (only performs full unrolling)
|
|
- VersioningLICM (if enabled)
|
|
- LoopDistribute
|
|
- LoopVectorizer
|
|
- LoopUnrollAndJam (if enabled)
|
|
- LoopUnroll (partial and runtime unrolling)
|
|
|
|
When using the legacy pass manager with LTO:
|
|
|
|
- LoopInterchange (if enabled)
|
|
- SimpleLoopUnroll/LoopFullUnroll (only performs full unrolling)
|
|
- LoopVectorizer
|
|
- LoopUnroll (partial and runtime unrolling)
|
|
|
|
When using the new pass manager:
|
|
|
|
- SimpleLoopUnroll/LoopFullUnroll (only performs full unrolling)
|
|
- LoopDistribute
|
|
- LoopVectorizer
|
|
- LoopUnrollAndJam (if enabled)
|
|
- LoopUnroll (partial and runtime unrolling)
|
|
|
|
Leftover Transformations
|
|
========================
|
|
|
|
Forced transformations that have not been applied after the last
|
|
transformation pass should be reported to the user. The transformation
|
|
passes themselves cannot be responsible for this reporting because they
|
|
might not be in the pipeline, there might be multiple passes able to
|
|
apply a transformation (e.g. ``LoopInterchange`` and Polly) or a
|
|
transformation attribute may be 'hidden' inside another passes' followup
|
|
attribute.
|
|
|
|
The pass ``-transform-warning`` (``WarnMissedTransformationsPass``)
|
|
emits such warnings. It should be placed after the last transformation
|
|
pass.
|
|
|
|
The current pass pipeline has a fixed order in which transformations
|
|
passes are executed. A transformation can be in the followup of a pass
|
|
that is executed later and thus leftover. For instance, a loop nest
|
|
cannot be distributed and then interchanged with the current pass
|
|
pipeline. The loop distribution will execute, but there is no loop
|
|
interchange pass following such that any loop interchange metadata will
|
|
be ignored. The ``-transform-warning`` should emit a warning in this
|
|
case.
|
|
|
|
Future versions of LLVM may fix this by executing transformations using
|
|
a dynamic ordering.
|