This patch fixes https://github.com/google/sanitizers/issues/703
On a Graviton-A1 aarch64 machine with 48-bit VMA,
the time spent in LSan and ASan reduced from 2.5s to 0.01s when running
clang -fsanitize=leak compiler-rt/test/lsan/TestCases/sanity_check_pure_c.c && time ./a.out
clang -fsanitize=address compiler-rt/test/lsan/TestCases/sanity_check_pure_c.c && time ./a.out
With this patch, LSan and ASan create both the 32 and 64 allocators and select
at run time between the two allocators following a global variable that is
initialized at init time to whether the allocator64 can be used in the virtual
address space.
Differential Revision: https://reviews.llvm.org/D60243
llvm-svn: 369441
SanitizerCommon.PthreadDestructorIterations currently FAILs on Solaris:
[ RUN ] SanitizerCommon.PthreadDestructorIterations
/vol/llvm/src/compiler-rt/local/lib/sanitizer_common/tests/sanitizer_posix_test.cc:58: Failure
Value of: destructor_executed
Actual: true
Expected: false
[ FAILED ] SanitizerCommon.PthreadDestructorIterations (1 ms)
It turns out that destructor is called 4 times after the first call to SpawnThread, but
5 times after the second. While PTHREAD_DESTRUCTOR_ITERATIONS is 4 in
<limits.h>, the Solaris pthread_key_create(3C) man page documents
If, after all the destructors have been called for all keys with non-
null values, there are still some keys with non-null values, the
process will be repeated. POSIX requires that this process be executed
at least PTHREAD_DESTRUCTOR_ITERATIONS times. Solaris calls the
destructors repeatedly until all values with associated destructors are
NULL. Destructors that set new values can cause an infinite loop.
The patch adjusts the test case to allow for this.
Tested on x86_64-pc-solaris2.11.
Differential Revision: https://reviews.llvm.org/D65055
llvm-svn: 367705
A bot complains:
/b/sanitizer-x86_64-linux-autoconf/build/llvm/projects/compiler-rt/lib/sanitizer_common/tests/malloc_stress_transfer_test.cpp:2: Streams are highly discouraged. [readability/streams] [3]
/b/sanitizer-x86_64-linux-autoconf/build/llvm/projects/compiler-rt/lib/sanitizer_common/tests/sanitizer_libc_test.cpp:11: Streams are highly discouraged. [readability/streams] [3]
lib/CMakeFiles/SanitizerLintCheck.dir/build.make:57: recipe for target 'lib/CMakeFiles/SanitizerLintCheck' failed
I do not know why this apparently wasn't a problem when the files
had extension .cc.
llvm-svn: 367493
See https://reviews.llvm.org/D58620 for discussion, and for the commands
I ran. In addition I also ran
for f in $(svn diff | diffstat | grep .cc | cut -f 2 -d ' '); do rg $(basename $f) . ; done
and manually updated references to renamed files found by that.
llvm-svn: 367467
See https://reviews.llvm.org/D58620 for discussion, and for the commands
I ran. In addition I also ran
for f in $(svn diff | diffstat | grep .cc | cut -f 2 -d ' '); do rg $f . ; done
and manually updated (many) references to renamed files found by that.
llvm-svn: 367463
They need to have same AddressSpaceView and MapUnmapCallback.
Reviewers: eugenis
Subscribers: kubamracek, #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D61168
llvm-svn: 359719
Originally this code as added for 64-bit platform and was never changed.
Add static_assert to make sure that we have correct map on all platforms.
llvm-svn: 359269
Replace remaining uses of old Unwind API in unit tests.
Allows us to remove the old API and WillUseFastUnwind can be made
private.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D58754
llvm-svn: 355242
This reverts revision 354601 and disables ReadBinaryNameCached check on
Windows since Windows has no working ReadBinaryName.
Differential Revision: https://reviews.llvm.org/D58788
llvm-svn: 355129
FastUnwindStack -> UnwindFast
SlowUnwindStack -> UnwindSlow
Stack is redundant, verb should come first.
SlowUnwindStackWithContext(uptr pc, void *context, u32 max_depth) ->
SlowUnwindStack
WithContext is redundant, since it is a required parameter.
Reviewers: vitalybuka
Differential Revision: https://reviews.llvm.org/D58551
llvm-svn: 354696
There is an ambiguity between ::SizeClassMap (the typedef declared near
the start of this file) and __sanitizer::SizeClassMap (found by the
'using namespace __sanitizer;' near the start of this file).
Historically a Clang bug has meant that the error was not diagnosed, but
soon Clang will start diagnosing it. Explicitly qualify this use of
SizeClassMap so that it finds __sanitizer::SizeClassMap rather than
being ill-formed due to ambiguity.
llvm-svn: 354174
Replace bool workerthread flag with ThreadType enum.
This change is preparation for fiber support.
[dvyukov: fixed build of sanitizer_thread_registry_test.cc]
Author: yuri (Yuri Per)
Reviewed in: https://reviews.llvm.org/D57839
Context: https://reviews.llvm.org/D54889
llvm-svn: 353390
to reflect the new license.
We understand that people may be surprised that we're moving the header
entirely to discuss the new license. We checked this carefully with the
Foundation's lawyer and we believe this is the correct approach.
Essentially, all code in the project is now made available by the LLVM
project under our new license, so you will see that the license headers
include that license only. Some of our contributors have contributed
code under our old license, and accordingly, we have retained a copy of
our old license notice in the top-level files in each project and
repository.
llvm-svn: 351636
Summary:
This is a follow up patch to r349138.
This patch makes a `AddressSpaceView` a type declaration in the
allocator parameters used by `SizeClassAllocator64`. For ASan, LSan, and
the unit tests the AP64 declarations have been made templated so that
`AddressSpaceView` can be changed at compile time. For the other
sanitizers we just hard-code `LocalAddressSpaceView` because we have no
plans to use these allocators in an out-of-process manner.
rdar://problem/45284065
Reviewers: kcc, dvyukov, vitalybuka, cryptoad, eugenis, kubamracek, george.karpenkov
Subscribers: #sanitizers, llvm-commits
Differential Revision: https://reviews.llvm.org/D55764
llvm-svn: 349954
Summary:
This is a follow up patch to r346956 for the `SizeClassAllocator32`
allocator.
This patch makes `AddressSpaceView` a template parameter both to the
`ByteMap` implementations (but makes `LocalAddressSpaceView` the
default), some `AP32` implementations and is used in `SizeClassAllocator32`.
The actual changes to `ByteMap` implementations and
`SizeClassAllocator32` are very simple. However the patch is large
because it requires changing all the `AP32` definitions, and users of
those definitions.
For ASan and LSan we make `AP32` and `ByteMap` templateds type that take
a single `AddressSpaceView` argument. This has been done because we will
instantiate the allocator with a type that isn't `LocalAddressSpaceView`
in the future patches. For the allocators used in the other sanitizers
(i.e. HWAsan, MSan, Scudo, and TSan) use of `LocalAddressSpaceView` is
hard coded because we do not intend to instantiate the allocators with
any other type.
In the cases where untemplated types have become templated on a single
`AddressSpaceView` parameter (e.g. `PrimaryAllocator`) their name has
been changed to have a `ASVT` suffix (Address Space View Type) to
indicate they are templated. The only exception to this are the `AP32`
types due to the desire to keep the type name as short as possible.
In order to check that template is instantiated in the correct a way a
`static_assert(...)` has been added that checks that the
`AddressSpaceView` type used by `Params::ByteMap::AddressSpaceView` matches
the `Params::AddressSpaceView`. This uses the new `sanitizer_type_traits.h`
header.
rdar://problem/45284065
Reviewers: kcc, dvyukov, vitalybuka, cryptoad, eugenis, kubamracek, george.karpenkov
Subscribers: mgorny, llvm-commits, #sanitizers
Differential Revision: https://reviews.llvm.org/D54904
llvm-svn: 349138
Summary:
In particular we implement the `is_same<T,U>` templated type. This is
useful for doing compile-time comparison of types in `static_assert`s.
The plan is to use this in another patch (
https://reviews.llvm.org/D54904 ).
Reviewers: kcc, dvyukov, vitalybuka, cryptoad, eugenis, kubamracek, george.karpenkov
Subscribers: mgorny, #sanitizers, llvm-commits
Differential Revision: https://reviews.llvm.org/D54951
llvm-svn: 349077
Summary:
Previously we weren't testing this function in the unit tests.
Reviewers: kcc, cryptoad, dvyukov, eugenis, kubamracek
Subscribers: #sanitizers, llvm-commits
Differential Revision: https://reviews.llvm.org/D54861
llvm-svn: 348260
Summary:
The default values used for Space/Size for the new SizeClassMap do not work
with Android. The Compact map appears to be in the same boat.
Disable the test on Android for now to turn the bots green, but there is no
reason Compact & Dense should not have an Android test.
Added a FIXME, I will revisit this soon.
Reviewers: eugenis
Subscribers: srhines, kubamracek, delcypher, #sanitizers, llvm-commits
Differential Revision: https://reviews.llvm.org/D52623
llvm-svn: 343252
Summary:
_Note_: I am not attached to the name `DenseSizeClassMap`, so if someone has a
better idea, feel free to suggest it.
The current pre-defined `SizeClassMap` hold a decent amount of cached entries,
either in cheer number of, or in amount of memory cached.
Empirical testing shows that more compact per-class arrays (whose sizes are
directly correlated to the number of cached entries) are beneficial to
performances, particularly in highly threaded environments.
The new proposed `SizeClassMap` has the following properties:
```
c00 => s: 0 diff: +0 00% l 0 cached: 0 0; id 0
c01 => s: 16 diff: +16 00% l 4 cached: 8 128; id 1
c02 => s: 32 diff: +16 100% l 5 cached: 8 256; id 2
c03 => s: 48 diff: +16 50% l 5 cached: 8 384; id 3
c04 => s: 64 diff: +16 33% l 6 cached: 8 512; id 4
c05 => s: 80 diff: +16 25% l 6 cached: 8 640; id 5
c06 => s: 96 diff: +16 20% l 6 cached: 8 768; id 6
c07 => s: 112 diff: +16 16% l 6 cached: 8 896; id 7
c08 => s: 128 diff: +16 14% l 7 cached: 8 1024; id 8
c09 => s: 144 diff: +16 12% l 7 cached: 7 1008; id 9
c10 => s: 160 diff: +16 11% l 7 cached: 6 960; id 10
c11 => s: 176 diff: +16 10% l 7 cached: 5 880; id 11
c12 => s: 192 diff: +16 09% l 7 cached: 5 960; id 12
c13 => s: 208 diff: +16 08% l 7 cached: 4 832; id 13
c14 => s: 224 diff: +16 07% l 7 cached: 4 896; id 14
c15 => s: 240 diff: +16 07% l 7 cached: 4 960; id 15
c16 => s: 256 diff: +16 06% l 8 cached: 4 1024; id 16
c17 => s: 320 diff: +64 25% l 8 cached: 3 960; id 49
c18 => s: 384 diff: +64 20% l 8 cached: 2 768; id 50
c19 => s: 448 diff: +64 16% l 8 cached: 2 896; id 51
c20 => s: 512 diff: +64 14% l 9 cached: 2 1024; id 48
c21 => s: 640 diff: +128 25% l 9 cached: 1 640; id 49
c22 => s: 768 diff: +128 20% l 9 cached: 1 768; id 50
c23 => s: 896 diff: +128 16% l 9 cached: 1 896; id 51
c24 => s: 1024 diff: +128 14% l 10 cached: 1 1024; id 48
c25 => s: 1280 diff: +256 25% l 10 cached: 1 1280; id 49
c26 => s: 1536 diff: +256 20% l 10 cached: 1 1536; id 50
c27 => s: 1792 diff: +256 16% l 10 cached: 1 1792; id 51
c28 => s: 2048 diff: +256 14% l 11 cached: 1 2048; id 48
c29 => s: 2560 diff: +512 25% l 11 cached: 1 2560; id 49
c30 => s: 3072 diff: +512 20% l 11 cached: 1 3072; id 50
c31 => s: 3584 diff: +512 16% l 11 cached: 1 3584; id 51
c32 => s: 4096 diff: +512 14% l 12 cached: 1 4096; id 48
c33 => s: 5120 diff: +1024 25% l 12 cached: 1 5120; id 49
c34 => s: 6144 diff: +1024 20% l 12 cached: 1 6144; id 50
c35 => s: 7168 diff: +1024 16% l 12 cached: 1 7168; id 51
c36 => s: 8192 diff: +1024 14% l 13 cached: 1 8192; id 48
c37 => s: 10240 diff: +2048 25% l 13 cached: 1 10240; id 49
c38 => s: 12288 diff: +2048 20% l 13 cached: 1 12288; id 50
c39 => s: 14336 diff: +2048 16% l 13 cached: 1 14336; id 51
c40 => s: 16384 diff: +2048 14% l 14 cached: 1 16384; id 48
c41 => s: 20480 diff: +4096 25% l 14 cached: 1 20480; id 49
c42 => s: 24576 diff: +4096 20% l 14 cached: 1 24576; id 50
c43 => s: 28672 diff: +4096 16% l 14 cached: 1 28672; id 51
c44 => s: 32768 diff: +4096 14% l 15 cached: 1 32768; id 48
c45 => s: 40960 diff: +8192 25% l 15 cached: 1 40960; id 49
c46 => s: 49152 diff: +8192 20% l 15 cached: 1 49152; id 50
c47 => s: 57344 diff: +8192 16% l 15 cached: 1 57344; id 51
c48 => s: 65536 diff: +8192 14% l 16 cached: 1 65536; id 48
c49 => s: 81920 diff: +16384 25% l 16 cached: 1 81920; id 49
c50 => s: 98304 diff: +16384 20% l 16 cached: 1 98304; id 50
c51 => s: 114688 diff: +16384 16% l 16 cached: 1 114688; id 51
c52 => s: 131072 diff: +16384 14% l 17 cached: 1 131072; id 48
c53 => s: 64 diff: +0 00% l 0 cached: 8 512; id 4
Total cached: 864928 (152/432)
```
It holds a bit less of 1MB of cached entries at most, and the cache fits in a
page.
The plan is to use this map by default for Scudo once we make sure that there
is no unforeseen impact for any of current use case.
Benchmarks give the most increase in performance (with Scudo) when looking at
highly threaded/contentious environments. For example, rcp2-benchmark
experiences a 10K QPS increase (~3%), and a decrease of 50MB for the max RSS
(~10%). On platforms like Android where we only have a couple of caches,
performance remain similar.
Reviewers: eugenis, kcc
Reviewed By: eugenis
Subscribers: kubamracek, delcypher, #sanitizers, llvm-commits
Differential Revision: https://reviews.llvm.org/D52371
llvm-svn: 343246
Summary:
Display a list of recent stack frames (not a stack trace!) when
tag-mismatch is detected on a stack address.
The implementation uses alignment tricks to get both the address of
the history buffer, and the base address of the shadow with a single
8-byte load. See the comment in hwasan_thread_list.h for more
details.
Developed in collaboration with Kostya Serebryany.
Reviewers: kcc
Subscribers: srhines, kubamracek, mgorny, hiraditya, jfb, llvm-commits
Differential Revision: https://reviews.llvm.org/D52249
llvm-svn: 342923
Summary:
Display a list of recent stack frames (not a stack trace!) when
tag-mismatch is detected on a stack address.
The implementation uses alignment tricks to get both the address of
the history buffer, and the base address of the shadow with a single
8-byte load. See the comment in hwasan_thread_list.h for more
details.
Developed in collaboration with Kostya Serebryany.
Reviewers: kcc
Subscribers: srhines, kubamracek, mgorny, hiraditya, jfb, llvm-commits
Differential Revision: https://reviews.llvm.org/D52249
llvm-svn: 342921
The test doesn't pass on Windows, where sizeof(long) == 4 also
on 64-bit, and so it isn't a multiple of sizeof(void*).
This also reverts the follow-up r340886.
> Summary: a constrained RingBuffer optimized for fast push
>
> Reviewers: eugenis
>
> Reviewed By: eugenis
>
> Subscribers: kubamracek, mgorny, delcypher, #sanitizers, llvm-commits
>
> Differential Revision: https://reviews.llvm.org/D51196
llvm-svn: 340924
- GetRandom and GetnumberOfCPUs using sys call for the former.
- enabling unit tests for the other oses.
Reviewers: kubamracek
Reviewed By: kubamracek
Differential Revision: https://reviews.llvm.org/D50937
llvm-svn: 340621
The variable name `SANITIZER_HEADERS` is already used for the list of
public headers in `include/CMakeLists.txt`. Although the previous
implementation worked it's probably best to avoid shadowing global
variables to avoid confusion.
Differential Revision: https://reviews.llvm.org/D49176
llvm-svn: 336904
Summary: Otherwise if the file existed and was larger than the write size before the OpenFile call, the file will not be truncated and contain garbage in trailing bytes.
Reviewers: glider, kcc, vitalybuka
Subscribers: kubamracek, delcypher, llvm-commits, #sanitizers
Differential Revision: https://reviews.llvm.org/D48250
llvm-svn: 334881
Summary:
Added unit-test.
Fixed behavior of max_len argument.
Call read syscall with all available buffer, not just a page.
Reviewers: eugenis
Subscribers: kubamracek, mgorny, llvm-commits
Differential Revision: https://reviews.llvm.org/D46618
llvm-svn: 334130
Summary:
The `TestOnlyInit` function of `{Flat,TwoLevel}ByteMap` seems to be a misnomer
since the function is used outside of tests as well, namely in
`SizeClassAllocator32::Init`. Rename it to `Init` and update the callers.
Reviewers: alekseyshl, vitalybuka
Reviewed By: vitalybuka
Subscribers: kubamracek, delcypher, #sanitizers, llvm-commits
Differential Revision: https://reviews.llvm.org/D46408
llvm-svn: 331662
Summary:
The following functions are only used in tests: `SetEnv`,
`SanitizerSetThreadName`, `SanitizerGetThreadName`. I don't think they are
going to be used in the future, and I propose to get rid of them, and associated
tests and include.
Reviewers: alekseyshl, eugenis, vitalybuka
Reviewed By: vitalybuka
Subscribers: dvyukov, vitalybuka, kubamracek, delcypher, llvm-commits, #sanitizers
Differential Revision: https://reviews.llvm.org/D45838
llvm-svn: 330724
Summary:
Example:
Printf("%-5s", "123");
should yield:
'123 '
In case Printf's requested string field width is larger than the string
argument length, the resulting string should be padded up to the requested
width.
For the simplicity sake, implementing left-justified (right padding) only.
Reviewers: eugenis
Subscribers: kubamracek, delcypher, #sanitizers, llvm-commits
Differential Revision: https://reviews.llvm.org/D45906
llvm-svn: 330643
Summary:
Example:
Printf("%.*s", 5, "123");
should yield:
'123 '
In case Printf's requested string precision is larger than the string
argument, the resulting string should be padded up to the requested
precision.
For the simplicity sake, implementing right padding only.
Reviewers: eugenis
Subscribers: kubamracek, delcypher, #sanitizers, llvm-commits
Differential Revision: https://reviews.llvm.org/D45844
llvm-svn: 330458
Summary:
Host symbolizer & stacktraces related code in their own RT:
`RTSanitizerCommonSymbolizer`, which is "libcdep" by nature. Symbolizer &
stacktraces specific code that used to live in common files is moved to a new
file `sanitizer_symbolizer_report.cc` as is.
The purpose of this is the enforce a separation between code that relies on
symbolization and code that doesn't. This saves the inclusion of spurious code
due to the interface functions with default visibility, and the extra data
associated.
The following sanitizers makefiles were modified & tested locally:
- dfsan: doesn't require the new symbolizer RT
- esan: requires it
- hwasan: requires it
- lsan: requires it
- msan: requires it
- safestack: doesn't require it
- xray: doesn't require it
- tsan: requires it
- ubsan: requires it
- ubsan_minimal: doesn't require it
- scudo: requires it (but not for Fuchsia that has a minimal runtime)
This was tested locally on Linux, Android, Fuchsia.
Reviewers: alekseyshl, eugenis, dberris, kubamracek, vitalybuka, dvyukov, mcgrathr
Reviewed By: alekseyshl, vitalybuka
Subscribers: srhines, kubamracek, mgorny, krytarowski, delcypher, llvm-commits, #sanitizers
Differential Revision: https://reviews.llvm.org/D45457
llvm-svn: 330131
Summary:
r327219 added wrappers to std::sort which randomly shuffle the container before sorting.
This will help in uncovering non-determinism caused due to undefined sorting
order of objects having the same key.
To make use of that infrastructure we need to invoke llvm::sort instead of std::sort.
Reviewers: kcc, rsmith, RKSimon, eugenis
Reviewed By: RKSimon
Subscribers: efriedma, kubamracek, dberris, #sanitizers, llvm-commits
Differential Revision: https://reviews.llvm.org/D44360
llvm-svn: 327929
Summary:
Make common allocator agnostic to failure handling modes and move the
decision up to the particular sanitizer's allocator, where the context
is available (call stack, parameters, return nullptr/crash mode etc.)
It simplifies the common allocator and allows the particular sanitizer's
allocator to generate more specific and detailed error reports (which
will be implemented later).
The behavior is largely the same, except one case, the violation of the
common allocator's check for "size + alignment" overflow is now reportied
as OOM instead of "bad request". It feels like a worthy tradeoff and
"size + alignment" is huge in this case anyway (thus, can be interpreted
as not enough memory to satisfy the request). There's also a Report()
statement added there.
Reviewers: eugenis
Subscribers: kubamracek, llvm-commits, #sanitizers
Differential Revision: https://reviews.llvm.org/D42198
llvm-svn: 322784
Summary:
This patch (on top of the previous two (https://reviews.llvm.org/D40898 and
https://reviews.llvm.org/D40899) complete the compiler-rt side of the the Solaris
sanitizer port.
It contains the following sets of changes:
* For the time being, the port is for 32-bit x86 only, so reject the various tests on
x86_64.
* When compiling as C++, <setjmp.h> resp. <iso/setjmp_iso.h> only declares
_setjmp and _longjmp inside namespace std.
* MAP_FILE is a Windows feature. While e.g. Linux <sys/mman.h> provides a
no-op compat define, Solaris does not.
* test/asan/TestCases/Posix/coverage.cc was initially failing like this:
/vol/gcc/src/llvm/llvm/local/projects/compiler-rt/lib/sanitizer_common/scripts/sancov.py: 4 files merged; 2 PCs total
rm: cannot remove '/var/gcc/llvm/local/projects/compiler-rt/test/asan/I386SunOSConfig/TestCases/Posix/Output/coverage': Invalid argument
Further digging revealed that the rm was trying to remove the running test's working
directory which failed as observed. cd'ing out of the dir before let the test pass.
* Two tests needed a declaration of alloca. I've now copied the existing code from
test/asan/TestCases/alloca_constant_size.cc, but it may be more profitable and
maintainable to have a common testsuite header where such code is collected.
* Similarly, Solaris' printf %p format doesn't include the leading 0x.
* In test/asan/TestCases/malloc-no-intercept.c, I had to undef __EXTENSIONS__
(predefined by clang for no apparent reason) to avoid conflicting declarations
for memalign.
* test/ubsan/TestCases/Float/cast-overflow.cpp has different platform dependent
ways to define BYTE_ORDER and friends. Why not just use __BYTE_ORDER__ and
friends as predefined by clang and gcc?
Patch by Rainer Orth.
Reviewers: kcc, alekseyshl
Reviewed By: alekseyshl
Subscribers: srhines, kubamracek, mgorny, krytarowski, fedor.sergeev, JDevlieghere, llvm-commits, #sanitizers
Tags: #sanitizers
Differential Revision: https://reviews.llvm.org/D40900
llvm-svn: 322635
Summary:
The low-fat STL-like vector container will be reused in MSan.
It is needed to implement an atexit(3) interceptor on NetBSD/amd64 in MSan.
Sponsored by <The NetBSD Foundation>
Reviewers: joerg, dvyukov, eugenis, vitalybuka, kcc
Reviewed By: dvyukov
Subscribers: kubamracek, mgorny, llvm-commits, #sanitizers
Tags: #sanitizers
Differential Revision: https://reviews.llvm.org/D40726
llvm-svn: 319650
Summary:
Fixed version of https://reviews.llvm.org/D38437 (fixes Win/Fuchsia failures).
Creating a new revision, since the old one was getting a bit old/crowded.
In Fuchsia, MmapNoAccess/MmapFixedOrDie are implemented using a global
VMAR, which means that MmapNoAccess can only be called once. This works
for the sanitizer allocator but *not* for the Scudo allocator.
Hence, this changeset introduces a new ReservedAddressRange object to
serve as the new API for these calls. In this changeset, the object
still calls into the old Mmap implementations.
The next changeset two changesets will convert the sanitizer and scudo
allocators to use the new APIs, respectively. (ReservedAddressRange will
replace the SecondaryHeader in Scudo.)
Finally, a last changeset will update the Fuchsia implementation.
Reviewers: alekseyshl, cryptoad, phosek
Reviewed By: alekseyshl, cryptoad
Subscribers: kubamracek
Differential Revision: https://reviews.llvm.org/D39072
llvm-svn: 316934
The SanitizerCommon.ReservedAddressRangeUnmap test fails on Windows:
FAIL: SanitizerCommon-Unit :: ./Sanitizer-x86_64-Test.exe/SanitizerCommon.ReservedAddressRangeUnmap (34003 of 35554)
******************** TEST 'SanitizerCommon-Unit :: ./Sanitizer-x86_64-Test.exe/SanitizerCommon.ReservedAddressRangeUnmap' FAILED ********************
Note: Google Test filter = SanitizerCommon.ReservedAddressRangeUnmap
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from SanitizerCommon
[ RUN ] SanitizerCommon.ReservedAddressRangeUnmap
==3780==ERROR: SanitizerTool failed to deallocate 0x1000 (4096) bytes at address 0x0000000c3000 (error code: 487)
==3780==Sanitizer CHECK failed: E:\b\build\slave\win_upload_clang\build\src\third_party\llvm\projects\compiler-rt\lib\sanitizer_common\sanitizer_win.cc:129 (("unable to unmap" && 0)) != (0) (0, 0)
********************
Testing: 0 .. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90..
Testing Time: 299.76s
********************
Failing Tests (1):
SanitizerCommon-Unit :: ./Sanitizer-x86_64-Test.exe/SanitizerCommon.ReservedAddressRangeUnmap
> In Fuchsia, MmapNoAccess/MmapFixedOrDie are implemented using a global
> VMAR, which means that MmapNoAccess can only be called once. This works
> for the sanitizer allocator but *not* for the Scudo allocator.
>
> Hence, this changeset introduces a new ReservedAddressRange object to
> serve as the new API for these calls. In this changeset, the object
> still calls into the old Mmap implementations.
>
> The next changeset two changesets will convert the sanitizer and scudo
> allocators to use the new APIs, respectively. (ReservedAddressRange will
> replace the SecondaryHeader in Scudo.)
>
> Finally, a last changeset will update the Fuchsia implementation.
>
> Patch by Julia Hansbrough
>
> Differential Revision: https://reviews.llvm.org/D38437
llvm-svn: 315553
In Fuchsia, MmapNoAccess/MmapFixedOrDie are implemented using a global
VMAR, which means that MmapNoAccess can only be called once. This works
for the sanitizer allocator but *not* for the Scudo allocator.
Hence, this changeset introduces a new ReservedAddressRange object to
serve as the new API for these calls. In this changeset, the object
still calls into the old Mmap implementations.
The next changeset two changesets will convert the sanitizer and scudo
allocators to use the new APIs, respectively. (ReservedAddressRange will
replace the SecondaryHeader in Scudo.)
Finally, a last changeset will update the Fuchsia implementation.
Patch by Julia Hansbrough
Differential Revision: https://reviews.llvm.org/D38437
llvm-svn: 315533
In Fuchsia, MmapNoAccess/MmapFixedOrDie are implemented using a global
VMAR, which means that MmapNoAccess can only be called once. This works
for the sanitizer allocator but *not* for the Scudo allocator.
Hence, this changeset introduces a new ReservedAddressRange object to
serve as the new API for these calls. In this changeset, the object
still calls into the old Mmap implementations.
The next changeset two changesets will convert the sanitizer and scudo
allocators to use the new APIs, respectively. (ReservedAddressRange will
replace the SecondaryHeader in Scudo.)
Finally, a last changeset will update the Fuchsia implementation.
Patch by Julia Hansbrough
Differential Revision: https://reviews.llvm.org/D38759
llvm-svn: 315493
Summary:
The current implementation of the allocator returning freed memory
back to OS (controlled by allocator_release_to_os_interval_ms flag)
requires sorting of the free chunks list, which has two major issues,
first, when free list grows to millions of chunks, sorting, even the
fastest one, is just too slow, and second, sorting chunks in place
is unacceptable for Scudo allocator as it makes allocations more
predictable and less secure.
The proposed approach is linear in complexity (altough requires quite
a bit more temporary memory). The idea is to count the number of free
chunks on each memory page and release pages containing free chunks
only. It requires one iteration over the free list of chunks and one
iteration over the array of page counters. The obvious disadvantage
is the allocation of the array of the counters, but even in the worst
case we support (4T allocator space, 64 buckets, 16 bytes bucket size,
full free list, which leads to 2 bytes per page counter and ~17M page
counters), requires just about 34Mb of the intermediate buffer (comparing
to ~64Gb of actually allocated chunks) and usually it stays under 100K
and released after each use. It is expected to be a relatively rare event,
releasing memory back to OS, keeping the buffer between those runs
and added complexity of the bookkeeping seems unnesessary here (it can
always be improved later, though, never say never).
The most interesting problem here is how to calculate the number of chunks
falling into each memory page in the bucket. Skipping all the details,
there are three cases when the number of chunks per page is constant:
1) P >= C, P % C == 0 --> N = P / C
2) C > P , C % P == 0 --> N = 1
3) C <= P, P % C != 0 && C % (P % C) == 0 --> N = P / C + 1
where P is page size, C is chunk size and N is the number of chunks per
page and the rest of the cases, where the number of chunks per page is
calculated on the go, during the page counter array iteration.
Among the rest, there are still cases where N can be deduced from the
page index, but they require not that much less calculations per page
than the current "brute force" way and 2/3 of the buckets fall into
the first three categories anyway, so, for the sake of simplicity,
it was decided to stick to those two variations. It can always be
refined and improved later, should we see that brute force way slows
us down unacceptably.
Reviewers: eugenis, cryptoad, dvyukov
Subscribers: kubamracek, mehdi_amini, llvm-commits
Differential Revision: https://reviews.llvm.org/D38245
llvm-svn: 314311
Summary:
Currently `TransferBatch` are located within the same memory regions as
"regular" chunks. This is not ideal for security: they make for an interesting
target to overwrite, and are not protected by the frontend (namely, Scudo).
To solve this, we re-introduce `kUseSeparateSizeClassForBatch` for the 32-bit
Primary allowing for `TransferBatch` to end up in their own memory region.
Currently only Scudo would use this new feature, the default behavior remains
unchanged. The separate `kBatchClassID` was used for a brief period of time
previously but removed when the 64-bit ended up using the "free array".
Reviewers: alekseyshl, kcc, eugenis
Reviewed By: alekseyshl
Subscribers: llvm-commits, kubamracek
Differential Revision: https://reviews.llvm.org/D37082
llvm-svn: 311891
into a function.
Most CMake configuration under compiler-rt/lib/*/tests have
almost-the-same-but-not-quite functions of the form add_X_[unit]tests
for compiling and running the tests.
Much of the logic is duplicated with minor variations across different
sub-folders.
This can harm productivity for multiple reasons:
For newcomers, resulting CMake files are very large, hard to understand,
and hide the intention of the code.
Changes for enabling certain architectures end up being unnecessarily
large, as they get duplicated across multiple folders.
Adding new sub-projects requires more effort than it should, as a
developer has to again copy-n-paste the configuration, and it's not even
clear from which sub-project it should be copy-n-pasted.
With this change the logic of compile-and-generate-a-set-of-tests is
extracted into a function, which hopefully makes writing and reading
CMake much easier.
Differential Revision: https://reviews.llvm.org/D36116
llvm-svn: 310971
Summary:
On platforms with `getrandom`, the system call defaults to blocking. This
becomes an issue in the very early stage of the boot for Scudo, when the RNG
source is not set-up yet: the syscall will block and we'll stall.
Introduce a parameter to specify that the function should not block, defaulting
to blocking as the underlying syscall does.
Update Scudo to use the non-blocking version.
Reviewers: alekseyshl
Reviewed By: alekseyshl
Subscribers: llvm-commits, kubamracek
Differential Revision: https://reviews.llvm.org/D36399
llvm-svn: 310839
Summary:
NetBSD ships with printf_l(3) like FreeBSD.
NetBSD does not ship with memalign, pvalloc, malloc with "usable size"
and is the same here as Darwin, Android, FreeBSD and Windows.
Sponsored by <The NetBSD Foundation>
Reviewers: joerg, vitalybuka, kcc, fjricci, filcab
Reviewed By: vitalybuka
Subscribers: srhines, llvm-commits, emaste, kubamracek, #sanitizers
Tags: #sanitizers
Differential Revision: https://reviews.llvm.org/D36373
llvm-svn: 310248
This patch addresses two issues:
Most of the time, hacks with `if/else` in order to get support for
multi-configuration builds are superfluous.
The variable `CMAKE_CFG_INTDIR` was created precisely for this purpose: it
expands to `.` on all single-configuration builds, and to a configuration
name otherwise.
The `if/else` hacks for the library name generation should also not be
done, as CMake has `TARGET_FILE` generator expression precisely for this
purpose, as it expands to the exact filename of the resulting target.
Differential Revision: https://reviews.llvm.org/D35952
llvm-svn: 309341
This patch addresses two issues:
Most of the time, hacks with `if/else` in order to get support for
multi-configuration builds are superfluous.
The variable `CMAKE_CFG_INTDIR` was created precisely for this purpose: it
expands to `.` on all single-configuration builds, and to a configuration
name otherwise.
The `if/else` hacks for the library name generation should also not be
done, as CMake has `TARGET_FILE` generator expression precisely for this
purpose, as it expands to the exact filename of the resulting target.
Differential Revision: https://reviews.llvm.org/D35952
llvm-svn: 309306
Summary:
This is a pure refactoring change. It just moves code that is
related to filesystem operations from sanitizer_common.{cc,h} to
sanitizer_file.{cc,h}. This makes it cleaner to disable the
filesystem-related code for a new port that doesn't want it.
Submitted on behalf of Roland McGrath.
Reviewers: kcc, eugenis, alekseyshl
Reviewed By: alekseyshl
Subscribers: vitalybuka, llvm-commits, kubamracek, mgorny, phosek
Tags: #sanitizers
Differential Revision: https://reviews.llvm.org/D35591
llvm-svn: 308819
This is a pure refactoring change. It just moves code that is
related to filesystem operations from sanitizer_common.{cc,h} to
sanitizer_file.{cc,h}. This makes it cleaner to disable the
filesystem-related code for a new port that doesn't want it.
Commiting for mcgrathr.
Reviewers: alekseyshl
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D35591
llvm-svn: 308640
Summary:
Make SizeClassAllocator32 return nullptr when it encounters OOM, which
allows the entire sanitizer's allocator to follow allocator_may_return_null=1
policy, even for small allocations (LargeMmapAllocator is already fixed
by D34243).
Will add a test for OOM in primary allocator later, when
SizeClassAllocator64 can gracefully handle OOM too.
Reviewers: eugenis
Subscribers: kubamracek, llvm-commits
Differential Revision: https://reviews.llvm.org/D34433
llvm-svn: 305972
Summary:
AFAICT compiler-rt doesn't have a function that would return 'good' random
bytes to seed a PRNG. Currently, the `SizeClassAllocator64` uses addresses
returned by `mmap` to seed its PRNG, which is not ideal, and
`SizeClassAllocator32` doesn't benefit from the entropy offered by its 64-bit
counterpart address space, so right now it has nothing. This function aims at
solving this, allowing to implement good 32-bit chunk randomization. Scudo also
has a function that does this for Cookie purposes, which would go away in a
later CL once this lands.
This function will try the `getrandom` syscall if available, and fallback to
`/dev/urandom` if not.
Unfortunately, I do not have a way to implement and test a Mac and Windows
version, so those are unimplemented as of now. Note that `kRandomShuffleChunks`
is only used on Linux for now.
Reviewers: alekseyshl
Reviewed By: alekseyshl
Subscribers: zturner, rnk, llvm-commits, kubamracek
Differential Revision: https://reviews.llvm.org/D34412
llvm-svn: 305922
Summary:
Move cached allocator_may_return_null flag to sanitizer_allocator.cc and
provide API to consolidate and unify the behavior of all specific allocators.
Make all sanitizers using CombinedAllocator to follow
AllocatorReturnNullOrDieOnOOM() rules to behave the same way when OOM
happens.
When OOM happens, turn allocator_out_of_memory flag on regardless of
allocator_may_return_null flag value (it used to not to be set when
allocator_may_return_null == true).
release_to_os_interval_ms and rss_limit_exceeded will likely be moved to
sanitizer_allocator.cc too (later).
Reviewers: eugenis
Subscribers: srhines, kubamracek, llvm-commits
Differential Revision: https://reviews.llvm.org/D34310
llvm-svn: 305858
Summary:
This broke thread_local_quarantine_pthread_join.cc on some architectures, due
to the overhead of the stashed regions. Reverting while figuring out the best
way to deal with it.
Reviewers: alekseyshl
Reviewed By: alekseyshl
Subscribers: llvm-commits, kubamracek
Differential Revision: https://reviews.llvm.org/D34213
llvm-svn: 305404
Summary:
The reasoning behind this change is explained in D33454, which unfortunately
broke the Windows version (due to the platform not supporting partial unmapping
of a memory region).
This new approach changes `MmapAlignedOrDie` to allow for the specification of
a `padding_chunk`. If non-null, and the initial allocation is aligned, this
padding chunk will hold the address of the extra memory (of `alignment` bytes).
This allows `AllocateRegion` to get 2 regions if the memory is aligned
properly, and thus help reduce fragmentation (and saves on unmapping
operations). As with the initial D33454, we use a stash in the 32-bit Primary
to hold those extra regions and return them on the fast-path.
The Windows version of `MmapAlignedOrDie` will always return a 0
`padding_chunk` if one was requested.
Reviewers: alekseyshl, dvyukov, kcc
Reviewed By: alekseyshl
Subscribers: llvm-commits, kubamracek
Differential Revision: https://reviews.llvm.org/D34152
llvm-svn: 305391
Summary:
allow_user_segv_handler had confusing name did not allow to control behavior for
signals separately.
Reviewers: eugenis, alekseyshl, kcc
Subscribers: llvm-commits, dberris, kubamracek
Differential Revision: https://reviews.llvm.org/D33371
llvm-svn: 303941
Summary: We are going to make it tri-state and remove allow_user_segv_handler.
Reviewers: eugenis, alekseys, kcc
Subscribers: kubamracek, dberris, llvm-commits
Differential Revision: https://reviews.llvm.org/D33159
llvm-svn: 303464
The sanitizer library unit tests for libc can get a different definition
of 'struct stat' to what the sanitizer library is built with for certain
targets.
For MIPS the size element of 'struct stat' is after a macro guarded
explicit padding element.
This patch resolves any possible inconsistency by adding the same
_FILE_OFFSET_BITS=64 and _LARGE_SOURCE with the same
conditions as the sanitizer library to the build flags for the unit tests.
This resolves a recurring build failure on the MIPS buildbots due to
'struct stat' defintion differences.
Reviewers: slthakur
Differential Revision: https://reviews.llvm.org/D33131
llvm-svn: 303350
Summary:
With rL279771, SizeClassAllocator64 was changed to accept only one template
instead of 5, for the following reasons: "First, this will make the mangled
names shorter. Second, this will make adding more parameters simpler". This
patch mirrors that work for SizeClassAllocator32.
This is in preparation for introducing the randomization of chunks in the
32-bit SizeClassAllocator in a later patch.
Reviewers: kcc, alekseyshl, dvyukov
Reviewed By: alekseyshl
Subscribers: llvm-commits, kubamracek
Differential Revision: https://reviews.llvm.org/D33141
llvm-svn: 303071
macOS
Summary:
In https://bugs.freebsd.org/215125 I was notified that some configure
scripts attempt to test for the Linux-specific `mallinfo` and `mallopt`
functions by compiling and linking small programs which references the
functions, and observing whether that results in errors.
FreeBSD and macOS do not have the `mallinfo` and `mallopt` functions, so
normally these tests would fail, but when sanitizers are enabled, they
incorrectly succeed, because the sanitizers define interceptors for
these functions. This also applies to some other malloc-related
functions, such as `memalign`, `pvalloc` and `cfree`.
Fix this by not intercepting `mallinfo`, `mallopt`, `memalign`,
`pvalloc` and `cfree` for FreeBSD and macOS, in all sanitizers.
Also delete the non-functional `cfree` wrapper for Windows, to fix the
test cases on that platform.
Reviewers: emaste, kcc, rnk
Subscribers: timurrrr, eugenis, hans, joerg, llvm-commits, kubamracek
Differential Revision: https://reviews.llvm.org/D27654
llvm-svn: 293536
This reverts r293337, which breaks tests on Windows:
malloc-no-intercept-499eb7.o : error LNK2019: unresolved external symbol _mallinfo referenced in function _main
llvm-svn: 293346
Summary:
In https://bugs.freebsd.org/215125 I was notified that some configure
scripts attempt to test for the Linux-specific `mallinfo` and `mallopt`
functions by compiling and linking small programs which references the
functions, and observing whether that results in errors.
FreeBSD and macOS do not have the `mallinfo` and `mallopt` functions, so
normally these tests would fail, but when sanitizers are enabled, they
incorrectly succeed, because the sanitizers define interceptors for
these functions. This also applies to some other malloc-related
functions, such as `memalign`, `pvalloc` and `cfree`.
Fix this by not intercepting `mallinfo`, `mallopt`, `memalign`,
`pvalloc` and `cfree` for FreeBSD and macOS, in all sanitizers.
Reviewers: emaste, kcc
Subscribers: hans, joerg, llvm-commits, kubamracek
Differential Revision: https://reviews.llvm.org/D27654
llvm-svn: 293337
Summary:
There are cases when thread local quarantine drains almost empty
quarantine batches into the global quarantine. The current approach leaves
them almost empty, which might create a huge memory overhead (each batch
is 4K/8K, depends on bitness).
Reviewers: eugenis
Subscribers: kubabrecka, llvm-commits
Differential Revision: https://reviews.llvm.org/D28068
llvm-svn: 292525
Summary:
The build system was inconsistent in its naming conventions for
link flags. This patch changes all uses of LINKFLAGS to LINK_FLAGS,
for consistency with cmake's LINK_FLAGS property.
This patch should make it easier to search the source code for
uses of link flags, as well as providing the benefit of improved
style and consistency.
Reviewers: compnerd, beanz
Subscribers: kubabrecka, llvm-commits, mgorny
Differential Revision: https://reviews.llvm.org/D28506
llvm-svn: 291539
Summary:
By default, darwin requires a definition for weak interface functions at
link time. Adding the '-U' link flag with each weak function allows these
weak interface functions to be used without definitions, which mirrors
behavior on linux and windows.
Reviewers: compnerd, eugenis
Subscribers: kubabrecka, mgorny, llvm-commits
Differential Revision: https://reviews.llvm.org/D28203
llvm-svn: 291417
This patch starts passing architecture information about a module to llvm-symbolizer and into text reports. This fixes the longstanding x86_64/x86_64h mismatch issue on Darwin.
Differential Revision: https://reviews.llvm.org/D27390
llvm-svn: 291287
When we enumerate loaded modules, we only track the module name and base address, which then has several problems on macOS. Dylibs and executables often have several architecture slices and not storing which architecture/UUID is actually loaded creates problems with symbolication: A file path + offset isn't enough to correctly symbolicate, since the offset can be valid in multiple slices. This is especially common for Haswell+ X86_64 machines, where x86_64h slices are preferred, but if one is not available, a regular x86_64 is loaded instead. But the same issue exists for i386 vs. x86_64 as well.
This patch adds tracking of arch and UUID for each LoadedModule. At this point, this information isn't used in reports, but this is the first step. The goal is to correctly identify which slice is loaded in symbolication, and also to output this information in reports so that we can tell which exact slices were loaded in post-mortem analysis.
Differential Revision: https://reviews.llvm.org/D26632
llvm-svn: 288537
Summary:
In order to avoid starting a separate thread to return unused memory to
the system (the thread interferes with process startup on Android,
Zygota waits for all threads to exit before fork, but this thread never
exits), try to return it right after free.
Reviewers: eugenis
Subscribers: cryptoad, filcab, danalbert, kubabrecka, llvm-commits
Patch by Aleksey Shlyapnikov.
Differential Revision: https://reviews.llvm.org/D27003
llvm-svn: 288091
Summary: The new name better corresponds to its logic.
Reviewers: kcc
Subscribers: kubabrecka
Differential Revision: https://reviews.llvm.org/D26821
llvm-svn: 287377
Looks like we are missing these flags only in tsan and sanitizer-common.
This results in linker warnings in some settings as it can cause the Unit
tests to be built with a different SDK version than that was used to build
the runtime. For example, we are not setting the minimal deployment target
on the tests but are setting the minimal deployment target for the sanitizer
library, which leads to the following warning on some bots: ld: warning:
object file (sanitizer_posix_test.cc.i386.o) was built for newer OSX version
(10.12) than being linked (10.11).
Differential Revision: https://reviews.llvm.org/D25860https://reviews.llvm.org/D25352
llvm-svn: 285255
The definitions in sanitizer_common may conflict with definitions from system headers because:
The runtime includes the system headers after the project headers (as per LLVM coding guidelines).
lib/sanitizer_common/sanitizer_internal_defs.h pollutes the namespace of everything defined after it, which is all/most of the sanitizer .h and .cc files and the included system headers with: using namespace __sanitizer; // NOLINT
This patch solves the problem by introducing the namespace only within the sanitizer namespaces as proposed by Dmitry.
Differential Revision: https://reviews.llvm.org/D21947
llvm-svn: 281657
These got out of sync and the tests were failing for me locally. We
assume a 47 bit address space in ASan, so we should do the same in the
tests.
llvm-svn: 281622
Summary:
The sanitizer allocators can works with a dynamic address space
(i.e. specified with ~0ULL).
Unfortunately, the code was broken on GetMetadata and GetChunkIdx.
The current patch is moving the Win64 memory test to a dynamic
address space. There is a migration to move every concept to a
dynamic address space on windows.
To have a better coverage, the unittest are now testing
dynamic address space on other platforms too.
Reviewers: rnk, kcc
Subscribers: kubabrecka, dberris, llvm-commits, chrisha
Differential Revision: https://reviews.llvm.org/D23170
llvm-svn: 277745
Summary:
Due to a QoI issuse in FreeBSD's libcxxrt-based demangler, one sanitizer
symbolizer test consistently appears to fail:
Value of: DemangleSwiftAndCXX("foo")
Actual: "float"
Expected: "foo"
This is because libcxxrt's __cxa_demangle() incorrectly demangles the "foo"
identifier to "float". It should return an error instead.
For now, XFAIL this particular test for FreeBSD, until we can fix libcxxrt
properly (which might take some time to coordinate with upstream).
Reviewers: rnk, zaks.anna, emaste
Subscribers: emaste, llvm-commits, kubabrecka
Differential Revision: https://reviews.llvm.org/D23001
llvm-svn: 277297
This test attempts to allocate 100 512MB aligned pages of memory. This
is implemented in the usual way by allocating size + alignment bytes and
aligning the result. As a result, this test allocates 51.2GB of memory.
Windows allocates swap for all memory allocated, and our bots do not
have this much swap available.
Avoid the failure by using a more reasonable alignment, like 16MB, as we
do on 32-bit.
llvm-svn: 276779
Add a %stdcxx11 lit substitution for -std=c++11. Windows defaults to
-std=c++14 when VS 2015 is used because the STL requires it. Harcoding
-std=c++11 in the ASan tests actually downgrades the C++ standard level,
leading to test failures.
Relax a FileCheck pattern in use-after-scope-types.cc.
Disable the sanitizer_common OOM tests. They fail on bots with low swap,
and cause other concurrently running tests to OOM.
llvm-svn: 276454
Vitaly Buka