All in-tree targets pass pointer-sized ConstantSDNodes to the
method. This overload reduced amount of boilerplate code a bit. This
also makes getCALLSEQ_END consistent with getCALLSEQ_START, which
already takes uint64_ts.
This change completes the process of replacing OperandValueKind and OperandValueProperties which were previously passed independently in this API with a single container class which contains both.
This is the change which motivated the whole sequence which preceeded it. In an original spike version of this change, I'd noticed a nasty bug: I'd changed the signature without changing names, and as result, we silently passed additional information through a callsite which previously dropped the power-of-two fact. This might be harmless in most cases, but at least a couple clearly dependend for correctness on not passing that property through.
I did my best to split off prior changes which reduced the scope of this one, and which made it possible to use compiler assistance. For instance, every parameter which changes type in this change also changes name. This was intentional to make sure that every call site possible effected must show up in the diff. This let me audit each one closely.
D25618 added a method to verify the instruction predicates for an
emitted instruction, through verifyInstructionPredicates added into
<Target>MCCodeEmitter::encodeInstruction. This is a very useful idea,
but the implementation inside MCCodeEmitter made it only fire for object
files, not assembly which most of the llvm test suite uses.
This patch moves the code into the <Target>_MC::verifyInstructionPredicates
method, inside the InstrInfo. The allows it to be called from other
places, such as in this patch where it is called from the
<Target>AsmPrinter::emitInstruction methods which should trigger for
both assembly and object files. It can also be called from other places
such as verifyInstruction, but that is not done here (it tends to catch
errors earlier, but in reality just shows all the mir tests that have
incorrect feature predicates). The interface was also simplified
slightly, moving computeAvailableFeatures into the function so that it
does not need to be called externally.
The ARM, AMDGPU (but not R600), AVR, Mips and X86 backends all currently
show errors in the test-suite, so have been disabled with FIXME
comments.
Recommitted with some fixes for the leftover MCII variables in release
builds.
Differential Revision: https://reviews.llvm.org/D129506
When doing experiment in kernel, for kernel data structure sockptr_t
in CO-RE operation, I hit an assertion error. The sockptr_t definition
and usage look like below:
#pragma clang attribute push (__attribute__((preserve_access_index)), apply_to = record)
typedef struct {
union {
void *kernel;
void *user;
};
unsigned is_kernel : 1;
} sockptr_t;
#pragma clang attribute pop
int test(sockptr_t *arg) {
return arg->is_kernel;
}
The assertion error looks like
clang: ../lib/Target/BPF/BPFAbstractMemberAccess.cpp:878: llvm::Value*
{anonymous}::BPFAbstractMemberAccess::computeBaseAndAccessKey(llvm::CallInst*,
{anonymous}::BPFAbstractMemberAccess::CallInfo&, std::__cxx11::string&,
llvm::MDNode*&): Assertion `TypeName.size()' failed.
In this particular, the clang frontend attach the debuginfo metadata associated
with anon structure with the preserve_access_info IR intrinsic. But the first
debuginfo type has to be a named type so libbpf can have a sound start to
do CO-RE relocation.
Besides the above approach using pragma to push attribute, the below typedef/struct
definition can have preserve_access_index directly applying to the anon struct.
typedef struct {
union {
void *kernel;
void *user;
};
unsigned is_kernel : 1;
} __attribute__((preserve_access_index) sockptr_t;
This patch fixed the issue by preprocessing function argument/return types
and local variable types used by other CO-RE intrinsics. For any
typedef struct/union { ... } typedef_name
an association of <anon struct/union, typedef> is recorded to replace
the IR intrinsic metadata 'anon struct/union' to 'typedef'.
It is possible that two different 'typedef' types may have identical
anon struct/union type. For such a case, the association will be
<anon struct/union, nullptr> to indicate the invalid case.
Differential Revision: https://reviews.llvm.org/D129621
This reverts commit e2fb8c0f4b as it does
not build for Release builds, and some buildbots are giving more warning
than I saw locally. Reverting to fix those issues.
D25618 added a method to verify the instruction predicates for an
emitted instruction, through verifyInstructionPredicates added into
<Target>MCCodeEmitter::encodeInstruction. This is a very useful idea,
but the implementation inside MCCodeEmitter made it only fire for object
files, not assembly which most of the llvm test suite uses.
This patch moves the code into the <Target>_MC::verifyInstructionPredicates
method, inside the InstrInfo. The allows it to be called from other
places, such as in this patch where it is called from the
<Target>AsmPrinter::emitInstruction methods which should trigger for
both assembly and object files. It can also be called from other places
such as verifyInstruction, but that is not done here (it tends to catch
errors earlier, but in reality just shows all the mir tests that have
incorrect feature predicates). The interface was also simplified
slightly, moving computeAvailableFeatures into the function so that it
does not need to be called externally.
The ARM, AMDGPU (but not R600), AVR, Mips and X86 backends all currently
show errors in the test-suite, so have been disabled with FIXME
comments.
Differential Revision: https://reviews.llvm.org/D129506
Among others, BPF currently supports the type-exists CO-RE relocation
(e.g., see D83878 & D83242). Its intention, as the name tries to convey,
is to be used for checking existence of a type in a target.
While that check is useful and has its place, we would also like to
be able to perform stricter type queries: instead of just checking mere
existence, we want to make sure that members match up in composite
types, that enum variants are present, etc. We refer to this as "type
match".
This change proposes the addition of a new relocation variant/value that
we intend to use for establishing this match relation.
Differential Revision: https://reviews.llvm.org/D126838
Current BTF only supports 32-bit value. For example,
enum T { VAL = 0xffffFFFF00000008 };
the generated BTF looks like
.long 16 # BTF_KIND_ENUM(id = 4)
.long 100663297 # 0x6000001
.long 8
.long 18
.long 8
The encoded value is 8 which equals to (uint32_t)0xffffFFFF00000008
and this is incorrect.
This patch introduced BTF_KIND_ENUM64 which permits to encode
64-bit value. The format for each enumerator looks like:
.long name_offset
.long (uint32_t)value # lower-32 bit value
.long value >> 32 # high-32 bit value
We use two 32-bit values to represent a 64-bit value as current
BTF type subsection has 4-byte alignment and gaps are not permitted
in the subsection.
This patch also added support for kflag (the bit 31 of CommonType.Info)
such that kflag = 1 implies the value is signed and kflag = 0
implies the value is unsigned. The kernel UAPI enumerator definition is
struct btf_enum {
__u32 name_off;
__s32 val;
};
so kflag = 0 with unsigned value provides backward compatability.
With this patch, for
enum T { VAL = 0xffffFFFF00000008 };
the generated BTF looks like
.long 16 # BTF_KIND_ENUM64(id = 4)
.long 3187671053 # 0x13000001
.long 8
.long 18
.long 8 # 0x8
.long 4294967295 # 0xffffffff
and the enumerator value and signedness are encoded correctly.
Differential Revision: https://reviews.llvm.org/D124641
The name `MCFixedLenDisassembler.h` is out of date after D120958.
Rename it as `MCDecoderOps.h` to reflect the change.
Reviewed By: myhsu
Differential Revision: https://reviews.llvm.org/D124987
When a specific sequence of bytes is present in the file during
disassembly the disassembler fails with the following assertion:
...
0: 18 20 00 00 00 00 00 00 lea
... Assertion `idx < size()' failed.
...
llvm::SmallVectorTemplateCommon<...>::operator[](...) ...
llvm::MCInst::getOperand(unsigned int) ...
llvm::BPFInstPrinter::printOperand(...) ...
llvm::BPFInstPrinter::printInstruction() ...
llvm::BPFInstPrinter::printInst(...) ...
...
The byte sequence causing the error is (little endian):
18 20 00 00 00 00 00 00 00 00 00 00 00 00 00 00
The issue could be reproduced using the program bellow:
test.ir:
@G = constant
[16 x i8]
[i8 u0x18, i8 u0x20, i8 u0x00, i8 u0x00, i8 u0x00, i8 u0x00, i8 u0x00, i8 u0x00,
i8 u0x00, i8 u0x00, i8 u0x00, i8 u0x00, i8 u0x00, i8 u0x00, i8 u0x00, i8 u0x00],
section "foo", align 8
Compiled and disassembled as follows:
cat test.ir | llc -march=bpfel -filetype=obj -o - \
| llvm-objdump --arch=bpfel --section=foo -d -
This byte sequence corresponds to FI_ri instruction declared in the
BPFInstrInfo.td as follows:
def FI_ri
: TYPE_LD_ST<BPF_IMM.Value, BPF_DW.Value,
(outs GPR:$dst),
(ins MEMri:$addr),
"lea\t$dst, $addr",
[(set i64:$dst, FIri:$addr)]> {
// This is a tentative instruction, and will be replaced
// with MOV_rr and ADD_ri in PEI phase
let Inst{51-48} = 0;
let Inst{55-52} = 2;
let Inst{47-32} = 0;
let Inst{31-0} = 0;
let BPFClass = BPF_LD;
}
Notes:
- First byte (opcode) is formed as follows:
- BPF_IMM.Value is 0x00
- BPF_DW.Value is 0x18
- BPF_LD is 0x00
- Second byte (registers) is formed as follows:
- let Inst{55-52} = 2;
- let Inst{51-48} = 0;
The FI_ri instruction is always replaced by MOV_rr ADD_ri instructions
pair in the BPFRegisterInfo::eliminateFrameIndex method. Thus, this
instruction should be invisible to disassembler. This patch achieves
this by adding "isPseudo" flag for this instruction.
The bug was found by decompiling of one of the BPF tests from Linux
kernel (llvm-objdump -D tools/testing/selftests/bpf/bpf_iter_sockmap.o)
Differential Revision: https://reviews.llvm.org/D125185
LLVM BPF pass SimplifyPatchable is used to do necessary
code conversion for CO-RE operations. When studying bpf
selftest 'exhandler', I found a corner case not handled properly.
The following is the C code, modified from original 'exhandler'
code.
int g;
int test(struct t1 *p) {
struct t2 *q = p->q;
if (q)
return 0;
struct t3 *f = q->f;
if (!f) g = 5;
return 0;
}
For code:
struct t3 *f = q->f;
if (!f) ...
The IR before BPFMISimplifyPatchable pass looks like:
%5:gpr = LD_imm64 @"llvm.t2:0:8$0:1"
%6:gpr = LDD killed %5:gpr, 0
%7:gpr = LDD killed %6:gpr, 0
JNE_ri killed %7:gpr, 0, %bb.3
JMP %bb.2
Note that compiler knows q = 0 based dataflow and value analysis.
The correct generated code after the pass should be
%5:gpr = LD_imm64 @"llvm.t2:0:8$0:1"
%7:gpr = LDD killed %5:gpr, 0
JNE_ri killed %7:gpr, 0, %bb.3
JMP %bb.2
But the current implementation did further optimization for the
above code and generates
%5:gpr = LD_imm64 @"llvm.t2:0:8$0:1"
JNE_ri killed %5:gpr, 0, %bb.3
JMP %bb.2
which is incorrect.
This patch added a cache to remember those load insns not associated
with CO-RE offset value and will skip these load insns during
transformation.
Differential Revision: https://reviews.llvm.org/D123883
Opaque pointer [1] is enabled as the default with commit [2].
Andrii found that current __builtin_preserve_enum_value() can only handle
non opaque pointer code pattern and will segfault with latest
llvm main branch where opaque-pointer is enabled by default.
This patch added the opaque pointer support.
Besides llvm selftests, also verified with bpf-next bpf selftests.
[1] https://llvm.org/docs/OpaquePointers.html
[2] https://reviews.llvm.org/D123122
Differential Revision: https://reviews.llvm.org/D123800
Delyan Kratunov reported an issue where __builtin_memcmp is
not inlined into simple load/compare instructions.
This is a known issue. In the current state, __builtin_memcmp
will be converted to memcmp call which won't work for
bpf programs.
This patch added support for expanding __builtin_memcmp with
actual loads and compares up to currently maximum 128 total loads.
The implementation is identical to PowerPC.
Differential Revision: https://reviews.llvm.org/D122676
All LLVM backends use MCDisassembler as a base class for their
instruction decoders. Use "const MCDisassembler *" for the decoder
instead of "const void *". Remove unnecessary static casts.
Reviewed By: skan
Differential Revision: https://reviews.llvm.org/D122245
The way the alignment is determined here is not compatible with
opaque pointers -- we'd have to thread it through using either
align or elementtype attributes.
However, as far as I can tell this alignment is actually never
used for this particular intrinsic, so I've dropped the assignment
entirely and converted RecordAlignment to MaybeAlign, so we get an
assertion failure if it does end up being used.
Differential Revision: https://reviews.llvm.org/D122379
When investigating an issue with bcc tool inject.py, I found
a verifier failure with latest clang. The portion of code
can be illustrated as below:
struct pid_struct {
u64 curr_call;
u64 conds_met;
u64 stack[2];
};
struct pid_struct *bpf_map_lookup_elem();
int foo() {
struct pid_struct *p = bpf_map_lookup_elem();
if (!p) return 0;
p->curr_call--;
if (p->conds_met < 1 || p->conds_met >= 3)
return 0;
if (p->stack[p->conds_met - 1] == p->curr_call)
p->conds_met--;
...
}
The verifier failure looks like:
...
8: (79) r1 = *(u64 *)(r0 +0)
R0_w=map_value(id=0,off=0,ks=4,vs=32,imm=0) R10=fp0 fp-8=mmmm????
9: (07) r1 += -1
10: (7b) *(u64 *)(r0 +0) = r1
R0_w=map_value(id=0,off=0,ks=4,vs=32,imm=0) R1_w=inv(id=0) R10=fp0 fp-8=mmmm????
11: (79) r2 = *(u64 *)(r0 +8)
R0_w=map_value(id=0,off=0,ks=4,vs=32,imm=0) R1_w=inv(id=0) R10=fp0 fp-8=mmmm????
12: (bf) r3 = r2
13: (07) r3 += -3
14: (b7) r4 = -2
15: (2d) if r4 > r3 goto pc+13
R0=map_value(id=0,off=0,ks=4,vs=32,imm=0) R1=inv(id=0) R2=inv(id=2)
R3=inv(id=0,umin_value=18446744073709551614,var_off=(0xffffffff00000000; 0xffffffff))
R4=inv-2 R10=fp0 fp-8=mmmm????
16: (07) r2 += -1
17: (bf) r3 = r2
18: (67) r3 <<= 3
19: (bf) r4 = r0
20: (0f) r4 += r3
math between map_value pointer and register with unbounded min value is not allowed
Here the compiler optimized "p->conds_met < 1 || p->conds_met >= 3" to
r2 = p->conds_met
r3 = r2
r3 += -3
r4 = -2
if (r3 < r4) return 0
r2 += -1
r3 = r2
...
In the above, r3 is initially equal to r2, but is modified used by the comparison.
But later on r2 is used again. This caused verification failure.
BPF backend has a pass, AdjustOpt, to prevent such transformation, but only
focused on signed integers since typical bpf helper returns signed integers.
To fix this case, let us handle unsigned integers as well.
Differential Revision: https://reviews.llvm.org/D121937
When checking a bcc issue related to bcc tool inject.py,
I found a bug in BPFAdjustOpt pass for icmp transformation,
caused by typo's. For the following condition:
Cond2Op != ICmpInst::ICMP_SLT && Cond1Op != ICmpInst::ICMP_SLE
it should be
Cond2Op != ICmpInst::ICMP_SLT && Cond2Op != ICmpInst::ICMP_SLE
This patch fixed the problem and a test case is added.
Differential Revision: https://reviews.llvm.org/D121883
Jussi Maki reported a fatal error like below for a bitfield
CO-RE relocation:
fatal error: error in backend: Unsupported field expression for
llvm.bpf.preserve.field.info, requiring too big alignment
The failure is related to kernel struct thread_struct. The following
is a simplied example.
Suppose we have below structure:
struct t2 {
int a[8];
} __attribute__((aligned(64))) __attribute__((preserve_access_index));
struct t1 {
int f1:1;
int f2:2;
struct t2 f3;
} __attribute__((preserve_access_index));
Note that struct t2 has aligned 64, which is used sometimes in the
kernel to enforce cache line alignment.
The above struct will be encoded into BTF and the following is what
C code looks like and the struct will appear in the file like vmlinux.h.
struct t2 {
int a[8];
long: 64;
long: 64;
long: 64;
long: 64;
} __attribute__((preserve_access_index));
struct t1 {
int f1: 1;
int f2: 2;
long: 61;
long: 64;
long: 64;
long: 64;
long: 64;
long: 64;
long: 64;
long: 64;
struct t2 f3;
} __attribute__((preserve_access_index));
Note that after
origin_source -> BTF -> new_source
transition, the new source has the same memory layout as the old one
but the alignment interpretation inside the compiler could be different.
The bpf program will use the later explicitly padded structure as in
vmlinux.h.
In the above case, the compiler internal ABI alignment for new struct t1
is 16 while it is 4 for old struct t1. I didn't do a thorough investigation
why the ABI alignment is 16 and I suspect it is related to anonymous padding
in the above.
Current BPF bitfield CO-RE handling requires alignment <= 8 so proper
bitfield operatin can be performed. Therefore, alignment 16 will cause
a compiler fatal error.
To fix the ABI alignment >=16, let us check whether the bitfield
can be held within a 8-byte-aligned range. If this is the case,
we can use alignment 8. Otherwise, a fatal error will be reported.
Differential Revision: https://reviews.llvm.org/D121821
Most notably, Pass.h is no longer included by TargetMachine.h
before: 1063570306
after: 1063332844
Differential Revision: https://reviews.llvm.org/D121168
In BPF backend, BTF type generation may skip
some debuginfo types if they are the pointee
type of a struct member. For example,
struct task_struct {
...
struct mm_struct *mm;
...
};
BPF backend may generate a forward decl for
'struct mm_struct' instead of full type if
there are no other usage of 'struct mm_struct'.
The reason is to avoid bringing too much unneeded types
in BTF.
Alexei found a pruning bug where we may miss
some full type generation. The following is an illustrating
example:
struct t1 { ... }
struct t2 { struct t1 *p; };
struct t2 g;
void foo(struct t1 *arg) { ... }
In the above case, we will have partial debuginfo chain like below:
struct t2 -> member p
\ -> ptr -> struct t1
/
foo -> argument arg
During traversing
struct t2 -> member p -> ptr -> struct t1
The corresponding BTF types are generated except 'struct t1' which
will be in FixUp stage. Later, when traversing
foo -> argument arg -> ptr -> struct t1
The 'ptr' BTF type has been generated and currently implementation
ignores 'pointer' type hence 'struct t1' is not generated.
This patch fixed the issue not just for the above case, but for
general case with multiple derived types, e.g.,
struct t2 -> member p
\ -> const -> ptr -> volatile -> struct t1
/
foo -> argument arg
Differential Revision: https://reviews.llvm.org/D119986
Kumar Kartikeya Dwivedi reported a bug ([1]) where BTF_KIND_TYPE_TAG types
are not generated.
Currently, BPF backend only generates BTF types which are used by
the program, e.g., global variables, functions and some builtin functions.
For example, suppose we have
struct task_struct {
...
struct task_group *sched_task_group;
struct mm_struct *mm;
...
pid_t pid;
pid_t tgid;
...
}
If BPF program intends to access task_struct->pid and task_struct->tgid,
there really no need to generate BTF types for struct task_group
and mm_struct.
In BPF backend, during BTF generation, when generating BTF for struct
task_struct, if types for task_group and mm_struct have not been generated
yet, a Fixup structure will be created, which will be reexamined later
to instantiate into either a full type or a forward type.
In current implementation, if we have something like
struct foo {
struct bar __tag1 *f;
};
and when generating types for struct foo, struct bar type
has not been generated, the __tag1 will be lost during later
Fixup instantiation. This patch fixed this issue by properly
handling btf_type_tag's during Fixup instantiation stage.
[1] https://lore.kernel.org/bpf/20220210232411.pmhzj7v5uptqby7r@apollo.legion/
Differential Revision: https://reviews.llvm.org/D119799
As usual with that header cleanup series, some implicit dependencies now need to
be explicit:
llvm/MC/MCParser/MCAsmParser.h no longer includes llvm/MC/MCParser/MCAsmLexer.h
Preprocessed lines to build llvm on my setup:
after: 1068185081
before: 1068324320
So no compile time benefit to expect, but we still get the looser coupling
between files which is great.
Discourse thread: https://discourse.llvm.org/t/include-what-you-use-include-cleanup
Differential Revision: https://reviews.llvm.org/D119359
There's a few relevant forward declarations in there that may require downstream
adding explicit includes:
llvm/MC/MCContext.h no longer includes llvm/BinaryFormat/ELF.h, llvm/MC/MCSubtargetInfo.h, llvm/MC/MCTargetOptions.h
llvm/MC/MCObjectStreamer.h no longer include llvm/MC/MCAssembler.h
llvm/MC/MCAssembler.h no longer includes llvm/MC/MCFixup.h, llvm/MC/MCFragment.h
Counting preprocessed lines required to rebuild llvm-project on my setup:
before: 1052436830
after: 1049293745
Which is significant and backs up the change in addition to the usual benefits of
decreasing coupling between headers and compilation units.
Discourse thread: https://discourse.llvm.org/t/include-what-you-use-include-cleanup
Differential Revision: https://reviews.llvm.org/D119244
This header is very large (3M Lines once expended) and was included in location
where dwarf-specific information were not needed.
More specifically, this commit suppresses the dependencies on
llvm/BinaryFormat/Dwarf.h in two headers: llvm/IR/IRBuilder.h and
llvm/IR/DebugInfoMetadata.h. As these headers (esp. the former) are widely used,
this has a decent impact on number of preprocessed lines generated during
compilation of LLVM, as showcased below.
This is achieved by moving some definitions back to the .cpp file, no
performance impact implied[0].
As a consequence of that patch, downstream user may need to manually some extra
files:
llvm/IR/IRBuilder.h no longer includes llvm/BinaryFormat/Dwarf.h
llvm/IR/DebugInfoMetadata.h no longer includes llvm/BinaryFormat/Dwarf.h
In some situations, codes maybe relying on the fact that
llvm/BinaryFormat/Dwarf.h was including llvm/ADT/Triple.h, this hidden
dependency now needs to be explicit.
$ clang++ -E -Iinclude -I../llvm/include ../llvm/lib/Transforms/Scalar/*.cpp -std=c++14 -fno-rtti -fno-exceptions | wc -l
after: 10978519
before: 11245451
Related Discourse thread: https://llvm.discourse.group/t/include-what-you-use-include-cleanup
[0] https://llvm-compile-time-tracker.com/compare.php?from=fa7145dfbf94cb93b1c3e610582c495cb806569b&to=995d3e326ee1d9489145e20762c65465a9caeab4&stat=instructions
Differential Revision: https://reviews.llvm.org/D118781
Instead use either Type::getPointerElementType() or
Type::getNonOpaquePointerElementType().
This is part of D117885, in preparation for deprecating the API.
This reverts commit fd4808887e.
This patch causes gcc to issue a lot of warnings like:
warning: base class ‘class llvm::MCParsedAsmOperand’ should be
explicitly initialized in the copy constructor [-Wextra]
Matteo Croce reported a bpf backend fatal error in
https://github.com/llvm/llvm-project/issues/52779
A simplified case looks like:
$ cat bug.c
extern int do_smth(int);
int test() {
return __builtin_btf_type_id(*(typeof(do_smth) *)do_smth, 1);
}
$ clang -target bpf -O2 -g -c bug.c
fatal error: error in backend: Empty type name for BTF_TYPE_ID_REMOTE reloc
...
The reason for the fatal error is that the relocation is against
a DISubroutineType like type 13 below:
!10 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
!11 = !{}
!12 = !DILocation(line: 3, column: 10, scope: !7)
!13 = !DISubroutineType(types: !14)
!14 = !{!10, !10}
The DISubroutineType doesn't have a name and there is no way for
downstream bpfloader/kernel to do proper relocation for it.
But we can improve error message to be more specific for this case.
The patch improved the error message to be:
fatal error: error in backend: SubroutineType not supported for BTF_TYPE_ID_REMOTE reloc
Differential Revision: https://reviews.llvm.org/D116063
Sergei Barannikov