I had left this as a TODO, but it turns out it wasn't complicated.
By specifying `MAP_RESIZABLE`, it allows us to keep the VMO which we
can then use for release purposes.
`releasePagesToOS` also had to be called the "proper" way, as Fuchsia
requires the `Offset` field to be correct. This has no impact on
non-Fuchsia platforms.
Differential Revision: https://reviews.llvm.org/D86800
Summary:
Partners have requested the ability to configure more parts of Scudo
at runtime, notably the Secondary cache options (maximum number of
blocks cached, maximum size) as well as the TSD registry options
(the maximum number of TSDs in use).
This CL adds a few more Scudo specific `mallopt` parameters that are
passed down to the various subcomponents of the Combined allocator.
- `M_CACHE_COUNT_MAX`: sets the maximum number of Secondary cached items
- `M_CACHE_SIZE_MAX`: sets the maximum size of a cacheable item in the Secondary
- `M_TSDS_COUNT_MAX`: sets the maximum number of TSDs that can be used (Shared Registry only)
Regarding the TSDs maximum count, this is a one way option, only
allowing to increase the count.
In order to allow for this, I rearranged the code to have some `setOption`
member function to the relevant classes, using the `scudo::Option` class
enum to determine what is to be set.
This also fixes an issue where a static variable (`Ready`) was used in
templated functions without being set back to `false` every time.
Reviewers: pcc, eugenis, hctim, cferris
Subscribers: jfb, llvm-commits, #sanitizers
Tags: #sanitizers
Differential Revision: https://reviews.llvm.org/D84667
This guarantees that we will detect a buffer overflow or underflow
that overwrites an adjacent block. This spatial guarantee is similar
to the temporal guarantee that we provide for immediate use-after-free.
Enabling odd/even tags involves a tradeoff between use-after-free
detection and buffer overflow detection. Odd/even tags make it more
likely for buffer overflows to be detected by increasing the size of
the guaranteed "red zone" around the allocation, but on the other
hand use-after-free is less likely to be detected because the tag
space for any particular chunk is cut in half. Therefore we introduce
a tuning setting to control whether odd/even tags are enabled.
Differential Revision: https://reviews.llvm.org/D84361
Note: Resubmission with frame pointers force-enabled to fix builds with
-DCOMPILER_RT_BUILD_BUILTINS=False
Summary:
Splits the unwinder into a non-segv (for allocation/deallocation traces) and a
segv unwinder. This ensures that implementations can select an accurate, slower
unwinder in the segv handler (if they choose to use the GWP-ASan provided one).
This is important as fast frame-pointer unwinders (like the sanitizer unwinder)
don't like unwinding through signal handlers.
Reviewers: morehouse, cryptoad
Reviewed By: morehouse, cryptoad
Subscribers: cryptoad, mgorny, eugenis, pcc, #sanitizers
Tags: #sanitizers
Differential Revision: https://reviews.llvm.org/D83994
It was causing tests to fail in -DCOMPILER_RT_BUILD_BUILTINS=OFF builds:
GwpAsan-Unittest :: ./GwpAsan-x86_64-Test/BacktraceGuardedPoolAllocator.DoubleFree
GwpAsan-Unittest :: ./GwpAsan-x86_64-Test/BacktraceGuardedPoolAllocator.UseAfterFree
see comment on the code review.
> Summary:
> Splits the unwinder into a non-segv (for allocation/deallocation traces) and a
> segv unwinder. This ensures that implementations can select an accurate, slower
> unwinder in the segv handler (if they choose to use the GWP-ASan provided one).
> This is important as fast frame-pointer unwinders (like the sanitizer unwinder)
> don't like unwinding through signal handlers.
>
> Reviewers: morehouse, cryptoad
>
> Reviewed By: morehouse, cryptoad
>
> Subscribers: cryptoad, mgorny, eugenis, pcc, #sanitizers
>
> Tags: #sanitizers
>
> Differential Revision: https://reviews.llvm.org/D83994
This reverts commit 502f0cc0e3.
Summary:
Splits the unwinder into a non-segv (for allocation/deallocation traces) and a
segv unwinder. This ensures that implementations can select an accurate, slower
unwinder in the segv handler (if they choose to use the GWP-ASan provided one).
This is important as fast frame-pointer unwinders (like the sanitizer unwinder)
don't like unwinding through signal handlers.
Reviewers: morehouse, cryptoad
Reviewed By: morehouse, cryptoad
Subscribers: cryptoad, mgorny, eugenis, pcc, #sanitizers
Tags: #sanitizers
Differential Revision: https://reviews.llvm.org/D83994
Summary:
When enabling some malloc debug features on Android, multiple 32 bit
regions become exhausted, and the allocations fail. Allow allocations
to keep trying each bigger class in the Primary until it finds a fit.
In addition, some Android tests running on 32 bit fail sometimes due
to a running out of space in two regions, and then fail the allocation.
Reviewers: cryptoad
Reviewed By: cryptoad
Subscribers: #sanitizers, llvm-commits
Tags: #sanitizers
Differential Revision: https://reviews.llvm.org/D82070
Summary:
Implement pattern initialization of memory (excluding the secondary
allocator because it already has predictable memory contents).
Expose both zero and pattern initialization through the C API.
Reviewers: pcc, cryptoad
Subscribers: #sanitizers, llvm-commits
Tags: #sanitizers
Differential Revision: https://reviews.llvm.org/D79133
Summary:
If this is called before the malloc call in a thread (or in the whole
program), the lazy initialization of the allocation can overwrite
Options.
Reviewers: pcc, cryptoad
Subscribers: #sanitizers, llvm-commits
Tags: #sanitizers
Differential Revision: https://reviews.llvm.org/D79130
Introduce a function __scudo_get_error_info() that may be called to interpret
a crash resulting from a memory error, potentially in another process,
given information extracted from the crashing process. The crash may be
interpreted as a use-after-free, buffer overflow or buffer underflow.
Also introduce a feature to optionally record a stack trace for each
allocation and deallocation. If this feature is enabled, a stack trace for
the allocation and (if applicable) the deallocation will also be available
via __scudo_get_error_info().
Differential Revision: https://reviews.llvm.org/D77283
For MTE error reporting we will need to expose interfaces for crash handlers
to use to interpret scudo headers and metadata. The intent is that these
interfaces will live in scudo/interface.h.
Move the existing interface.h into an include/scudo directory and make it
independent of the internal headers, so that we will be able to add the
interfaces there.
Differential Revision: https://reviews.llvm.org/D76648
Summary:
We introduced a way to fallback to the immediately larger size class for
the Primary in the event a region was full, but in the event of the largest
size class, we would just fail.
This change allows to fallback to the Secondary when the last region of
the Primary is full. We also expand the trick to all platforms as opposed
to being Android only, and update the test to cover the new case.
Reviewers: hctim, cferris, eugenis, morehouse, pcc
Subscribers: #sanitizers, llvm-commits
Tags: #sanitizers
Differential Revision: https://reviews.llvm.org/D76430
Summary:
This patch includes several changes to reduce the overall footprint
of the allocator:
- for realloc'd chunks: only keep the same chunk when lowering the size
if the delta is within a page worth of bytes;
- when draining a cache: drain the beginning, not the end; we add pointers
at the end, so that meant we were draining the most recently added
pointers;
- change the release code to account for an freed up last page: when
scanning the pages, we were looking for pages fully covered by blocks;
in the event of the last page, if it's only partially covered, we
wouldn't mark it as releasable - even what follows the last chunk is
all 0s. So now mark the rest of the page as releasable, and adapt the
test;
- add a missing `setReleaseToOsIntervalMs` to the cacheless secondary;
- adjust the Android classes based on more captures thanks to pcc@'s
tool.
Reviewers: pcc, cferris, hctim, eugenis
Subscribers: #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D75142
Summary:
Add a method to set the release to OS value as the system runs,
and allow this to be set differently in the primary and the secondary.
Also, add a default value to use for primary and secondary. This
allows Android to have a default that is different for
primary/secondary.
Update mallopt to support setting the release to OS value.
Reviewers: pcc, cryptoad
Reviewed By: cryptoad
Subscribers: cryptoad, jfb, #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D74448
Summary:
Due to Unity, we had to reduce our region sizes, but in some rare
situations, some programs (mostly tests AFAICT) manage to fill up
a region for a given size class.
So this adds a workaround for that attempts to allocate the block
from the immediately larger size class, wasting some memory but
allowing the application to keep going.
Reviewers: pcc, eugenis, cferris, hctim, morehouse
Subscribers: #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D74567
Summary:
Forewarning: This patch looks big in #LOC changed. I promise it's not that bad, it just moves a lot of content from one file to another. I've gone ahead and left inline comments on Phabricator for sections where this has happened.
This patch:
1. Introduces the crash handler API (crash_handler_api.h).
2. Moves information required for out-of-process crash handling into an AllocatorState. This is a trivially-copied POD struct that designed to be recovered from a deceased process, and used by the crash handler to create a GWP-ASan report (along with the other trivially-copied Metadata struct).
3. Implements the crash handler API using the AllocatorState and Metadata.
4. Adds tests for the crash handler.
5. Reimplements the (now optionally linked by the supporting allocator) in-process crash handler (i.e. the segv handler) using the new crash handler API.
6. Minor updates Scudo & Scudo Standalone to fix compatibility.
7. Changed capitalisation of errors (e.g. /s/Use after free/Use After Free).
Reviewers: cryptoad, eugenis, jfb
Reviewed By: eugenis
Subscribers: merge_guards_bot, pcc, jfb, dexonsmith, mgorny, cryptoad, #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D73557
Summary:
The Secondary's cache needs to be released when the Combined's
`releaseToOS` function is called (via `M_PURGE`) for example,
which this CL adds.
Additionally, if doing a forced release, we'll release the
transfer batch class as well since now we can do that.
There is a couple of other house keeping changes as well:
- read the page size only once in the Secondary Cache `store`
- remove the interval check for `CanRelease`: we are going to
make that configurable via `mallopt` so this needs not be
set in stone there.
Reviewers: cferris, hctim, pcc, eugenis
Subscribers: #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D73730
Summary:
This CL changes multiple things to improve performance (notably on
Android).We introduce a cache class for the Secondary that is taking
care of this mechanism now.
The changes:
- change the Secondary "freelist" to an array. By keeping free secondary
blocks linked together through their headers, we were keeping a page
per block, which isn't great. Also we know touch less pages when
walking the new "freelist".
- fix an issue with the freelist getting full: if the pattern is an ever
increasing size malloc then free, the freelist would fill up and
entries would not be used. So now we empty the list if we get to many
"full" events;
- use the global release to os interval option for the secondary: it
was too costly to release all the time, particularly for pattern that
are malloc(X)/free(X)/malloc(X). Now the release will only occur
after the selected interval, when going through the deallocate path;
- allow release of the `BatchClassId` class: it is releasable, we just
have to make sure we don't mark the batches containing batches
pointers as free.
- change the default release interval to 1s for Android to match the
current Bionic allocator configuration. A patch is coming up to allow
changing it through `mallopt`.
- lower the smallest class that can be released to `PageSize/64`.
Reviewers: cferris, pcc, eugenis, morehouse, hctim
Subscribers: phosek, #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D73507
Summary:
This is an Android-specific interface for iterating over all live
allocations in a memory range.
Reviewers: hctim, cferris
Subscribers: mgorny, mgrang, #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D73305
Summary:
* Implement enable() and disable() in GWP-ASan.
* Setup atfork handler.
* Improve test harness sanity and re-enable GWP-ASan in Scudo.
Scudo_standalone disables embedded GWP-ASan as necessary around fork().
Standalone GWP-ASan sets the atfork handler in init() if asked to. This
requires a working malloc(), therefore GWP-ASan initialization in Scudo
is delayed to the post-init callback.
Test harness changes are about setting up a single global instance of
the GWP-ASan allocator so that pthread_atfork() does not create
dangling pointers.
Test case shamelessly stolen from D72470.
Reviewers: cryptoad, hctim, jfb
Subscribers: mgorny, jfb, #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D73294
When the hardware and operating system support the ARM Memory Tagging
Extension, tag primary allocation granules with a random tag. The granules
either side of the allocation are tagged with tag 0, which is normally
excluded from the set of tags that may be selected randomly. Memory is
also retagged with a random tag when it is freed, and we opportunistically
reuse the new tag when the block is reused to reduce overhead. This causes
linear buffer overflows to be caught deterministically and non-linear buffer
overflows and use-after-free to be caught probabilistically.
This feature is currently only enabled for the Android allocator
and depends on an experimental Linux kernel branch available here:
https://github.com/pcc/linux/tree/android-experimental-mte
All code that depends on the kernel branch is hidden behind a macro,
ANDROID_EXPERIMENTAL_MTE. This is the same macro that is used by the Android
platform and may only be defined in non-production configurations. When the
userspace interface is finalized the code will be updated to use the stable
interface and all #ifdef ANDROID_EXPERIMENTAL_MTE will be removed.
Differential Revision: https://reviews.llvm.org/D70762
Summary:
fork() wasn't well (or at all) supported in Scudo. This materialized
in deadlocks in children.
In order to properly support fork, we will lock the allocator pre-fork
and unlock it post-fork in parent and child. This is done via a
`pthread_atfork` call installing the necessary handlers.
A couple of things suck here: this function allocates - so this has to
be done post initialization as our init path is not reentrance, and it
doesn't allow for an extra pointer - so we can't pass the allocator we
are currently working with.
In order to work around this, I added a post-init template parameter
that gets executed once the allocator is initialized for the current
thread. Its job for the C wrappers is to install the atfork handlers.
I reorganized a bit the impacted area and added some tests, courtesy
of cferris@ that were deadlocking prior to this fix.
Subscribers: jfb, #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D72470
Summary:
In order to implement `malloc_{enable|disable}` we were just disabling
(or really locking) the Primary and the Secondary. That meant that
allocations could still be serviced from the TSD as long as the cache
wouldn't have to be filled from the Primary.
This wasn't working out for Android tests, so this change implements
registry disabling (eg: locking) so that `getTSDAndLock` doesn't
return a TSD if the allocator is disabled. This also means that the
Primary doesn't have to be disabled in this situation.
For the Shared Registry, we loop through all the TSDs and lock them.
For the Exclusive Registry, we add a `Disabled` boolean to the Registry
that forces `getTSDAndLock` to use the Fallback TSD instead of the
thread local one. Disabling the Registry is then done by locking the
Fallback TSD and setting the boolean in question (I don't think this
needed an atomic variable but I might be wrong).
I clang-formatted the whole thing as usual hence the couple of extra
whiteline changes in this CL.
Reviewers: cferris, pcc, hctim, morehouse, eugenis
Subscribers: jfb, #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D71719
Summary:
Adds GWP-ASan to Scudo standalone. Default parameters are pulled across from the
GWP-ASan build. No backtrace support as of yet.
Reviewers: cryptoad, eugenis, pcc
Reviewed By: cryptoad
Subscribers: merge_guards_bot, mgorny, #sanitizers, llvm-commits, cferris, vlad.tsyrklevich, pcc
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D71229
Instead of testing whether the pointer is aligned, just align it
unconditionally and compare it to the original pointer.
This moves the computation of UserPtr up to before we start preparing the
header, so that the memory tagging code will be able to read the original
header containing the bounds of the previous allocation before it gets
potentially clobbered by the pointer realignment code.
Differential Revision: https://reviews.llvm.org/D71292
With tag-on-free we will need to get the chunk of a deallocated block. Change
getChunkFromBlock() so that it doesn't check that the chunk is allocated,
and move the check into the caller, so that it can be reused for this purpose.
Differential Revision: https://reviews.llvm.org/D71291
Summary:
In order to be compliant with tcmalloc's extension ownership
determination function, we have to expose a function that will
say if a chunk was allocated by us.
As to whether or not this has security consequences: someone
able to call this function repeatedly could use it to determine
secrets (cookie) or craft a valid header. So this should not be
exposed directly to untrusted user input.
Add related tests.
Additionally clang-format caught a few things to change.
Reviewers: hctim, pcc, cferris, eugenis, vitalybuka
Subscribers: JDevlieghere, jfb, #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D70908
The macros INLINE and COMPILER_CHECK always expand to the same thing (inline
and static_assert respectively). Both expansions are standards compliant C++
and are used consistently in the rest of LLVM, so let's improve consistency
with the rest of LLVM by replacing them with the expansions.
Differential Revision: https://reviews.llvm.org/D70793
Summary:
cferris@ found an issue where calling `releaseToOS` prior to any other
heap operation would lead to a crash, due to the allocator not being
properly initialized (it was discovered via `mallopt`).
The fix is to call `initThreadMaybe` prior to calling `releaseToOS` for
the Primary.
Add a test that crashes prior to fix.
Reviewers: hctim, cferris, pcc, eugenis
Subscribers: #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D70552
Summary:
A few small improvements and optimizations:
- when refilling the free list, push back the last batch and return
the front one: this allows to keep the allocations towards the front
of the region;
- instead of using 48 entries in the shuffle array, use a multiple of
`MaxNumCached`;
- make the maximum number of batches to create on refil a constant;
ultimately it should be configurable, but that's for later;
- `initCache` doesn't need to zero out the cache, it's already done.
- it turns out that when using `||` or `&&`, the compiler is adamant
on adding a short circuit for every part of the expression. Which
ends up making somewhat annoying asm with lots of test and
conditional jump. I am changing that to bitwise `|` or `&` in two
place so that the generated code looks better. Added comments since
it might feel weird to people.
This yields to some small performance gains overall, nothing drastic
though.
Reviewers: hctim, morehouse, cferris, eugenis
Subscribers: #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D70452
Summary:
`SCUDO_DEBUG` was not enabled for unit tests, meaning the `DCHECK`s
were never tripped. While turning this on, I discovered that a few
of those not-exercised checks were actually wrong. This CL addresses
those incorrect checks.
Not that to work in tests `CHECK_IMPL` has to explicitely use the
`scudo` namespace. Also changes a C cast to a C++ cast.
Reviewers: hctim, pcc, cferris, eugenis, vitalybuka
Subscribers: mgorny, #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D70276
Summary:
cferris@ found an issue due to the new Secondary free list behavior
and unfortunately it's completely my fault. The issue is twofold:
- I lost track of the (major) fact that the Combined assumes that
all chunks returned by the Secondary are zero'd out apprioriately
when dealing with `ZeroContents`. With the introduction of the
freelist, it's no longer the case as there can be a small portion
of memory between the header and the next page boundary that is
left untouched (the rest is zero'd via release). So the next time
that block is returned, it's not fully zero'd out.
- There was no test that would exercise that behavior :(
There are several ways to fix this, the one I chose makes the most
sense to me: we pass `ZeroContents` to the Secondary's `allocate`
and it zero's out the block if requested and it's coming from the
freelist. The prevents an extraneous `memset` in case the block
comes from `map`. Another possbility could have been to `memset`
in `deallocate`, but it's probably overzealous as all secondary
blocks don't need to be zero'd out.
Add a test that would have found the issue prior to fix.
Reviewers: morehouse, hctim, cferris, pcc, eugenis, vitalybuka
Subscribers: #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D69675
Summary:
The secondary allocator is slow, because we map and unmap each block
on allocation and deallocation.
While I really like the security benefits of such a behavior, this
yields very disappointing performance numbers on Android for larger
allocation benchmarks.
So this change adds a free list to the secondary, that will hold
recently deallocated chunks, and (currently) release the extraneous
memory. This allows to save on some memory mapping operations on
allocation and deallocation. I do not think that this lowers the
security of the secondary, but can increase the memory footprint a
little bit (RSS & VA).
The maximum number of blocks the free list can hold is templatable,
`0U` meaning that we fallback to the old behavior. The higher that
number, the higher the extra memory footprint.
I added default configurations for all our platforms, but they are
likely to change in the near future based on needs and feedback.
Reviewers: hctim, morehouse, cferris, pcc, eugenis, vitalybuka
Subscribers: mgorny, #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D69570
Summary:
Following up on D68471, this CL introduces some `getStats` APIs to
gather statistics in char buffers (`ScopedString` really) instead of
printing them out right away. Ultimately `printStats` will just
output the buffer, but that allows us to potentially do some work
on the intermediate buffer, and can be used for a `mallocz` type
of functionality. This allows us to pretty much get rid of all the
`Printf` calls around, but I am keeping the function in for
debugging purposes.
This changes the existing tests to use the new APIs when required.
I will add new tests as suggested in D68471 in another CL.
Reviewers: morehouse, hctim, vitalybuka, eugenis, cferris
Reviewed By: morehouse
Subscribers: delcypher, #sanitizers, llvm-commits
Tags: #llvm, #sanitizers
Differential Revision: https://reviews.llvm.org/D68653
llvm-svn: 374173
Summary:
This changes a few things to improve memory footprint and performances
on Android, and fixes a test compilation error:
- add `stdlib.h` to `wrappers_c_test.cc` to address
https://bugs.llvm.org/show_bug.cgi?id=42810
- change Android size class maps, based on benchmarks, to improve
performances and lower the Svelte memory footprint. Also change the
32-bit region size for said configuration
- change the `reallocate` logic to reallocate in place for sizes larger
than the original chunk size, when they still fit in the same block.
This addresses patterns from `memory_replay` dumps like the following:
```
202: realloc 0xb48fd000 0xb4930650 12352
202: realloc 0xb48fd000 0xb48fd000 12420
202: realloc 0xb48fd000 0xb48fd000 12492
202: realloc 0xb48fd000 0xb48fd000 12564
202: realloc 0xb48fd000 0xb48fd000 12636
202: realloc 0xb48fd000 0xb48fd000 12708
202: realloc 0xb48fd000 0xb48fd000 12780
202: realloc 0xb48fd000 0xb48fd000 12852
202: realloc 0xb48fd000 0xb48fd000 12924
202: realloc 0xb48fd000 0xb48fd000 12996
202: realloc 0xb48fd000 0xb48fd000 13068
202: realloc 0xb48fd000 0xb48fd000 13140
202: realloc 0xb48fd000 0xb48fd000 13212
202: realloc 0xb48fd000 0xb48fd000 13284
202: realloc 0xb48fd000 0xb48fd000 13356
202: realloc 0xb48fd000 0xb48fd000 13428
202: realloc 0xb48fd000 0xb48fd000 13500
202: realloc 0xb48fd000 0xb48fd000 13572
202: realloc 0xb48fd000 0xb48fd000 13644
202: realloc 0xb48fd000 0xb48fd000 13716
202: realloc 0xb48fd000 0xb48fd000 13788
...
```
In this situation we were deallocating the old chunk, and
allocating a new one for every single one of those, but now we can
keep the same chunk (we just updated the header), which saves some
heap operations.
Reviewers: hctim, morehouse, vitalybuka, eugenis, cferris, rengolin
Reviewed By: morehouse
Subscribers: srhines, delcypher, #sanitizers, llvm-commits
Tags: #llvm, #sanitizers
Differential Revision: https://reviews.llvm.org/D67293
llvm-svn: 371628
Summary:
cferris's Bionic tests found an issue in Scudo's `malloc_iterate`.
We were inclusive of both boundaries, which resulted in a `Block` that
was located on said boundary to be possibly accounted for twice, or
just being accounted for while iterating on regions that are not ours
(usually the unmapped ones in between Primary regions).
The fix is to exclude the upper boundary in `iterateOverChunks`, and
add a regression test.
This additionally corrects a typo in a comment, and change the 64-bit
Primary iteration function to not assume that `BatchClassId` is 0.
Reviewers: cferris, morehouse, hctim, vitalybuka, eugenis
Reviewed By: hctim
Subscribers: delcypher, #sanitizers, llvm-commits
Tags: #llvm, #sanitizers
Differential Revision: https://reviews.llvm.org/D66231
llvm-svn: 369400
Summary:
This introduces a bunch of small optimizations with the purpose of
making the fastpath tighter:
- tag more conditions as `LIKELY`/`UNLIKELY`: as a rule of thumb we
consider that every operation related to the secondary is unlikely
- attempt to reduce the number of potentially extraneous instructions
- reorganize the `Chunk` header to not straddle a word boundary and
use more appropriate types
Note that some `LIKELY`/`UNLIKELY` impact might be less obvious as
they are in slow paths (for example in `secondary.cc`), but at this
point I am throwing a pretty wide net, and it's consistant and doesn't
hurt.
This was mosly done for the benfit of Android, but other platforms
benefit from it too. An aarch64 Android benchmark gives:
- before:
```
BM_youtube/min_time:15.000/repeats:4/manual_time_mean 445244 us 659385 us 4
BM_youtube/min_time:15.000/repeats:4/manual_time_median 445007 us 658970 us 4
BM_youtube/min_time:15.000/repeats:4/manual_time_stddev 885 us 1332 us 4
```
- after:
```
BM_youtube/min_time:15.000/repeats:4/manual_time_mean 415697 us 621925 us 4
BM_youtube/min_time:15.000/repeats:4/manual_time_median 415913 us 622061 us 4
BM_youtube/min_time:15.000/repeats:4/manual_time_stddev 990 us 1163 us 4
```
Additional since `-Werror=conversion` is enabled on some platforms we
are built on, enable it upstream to catch things early: a few sign
conversions had slept through and needed additional casting.
Reviewers: hctim, morehouse, eugenis, vitalybuka
Reviewed By: vitalybuka
Subscribers: srhines, mgorny, javed.absar, kristof.beyls, delcypher, #sanitizers, llvm-commits
Tags: #llvm, #sanitizers
Differential Revision: https://reviews.llvm.org/D64664
llvm-svn: 366918
Summary:
A few corrections:
- rename `TransferBatch::MaxCached` to `getMaxCached` to conform with
the style guide;
- move `getBlockBegin` from `Chunk::` to `Allocator::`: I believe it
was a fallacy to have this be a `Chunk` method, as chunks'
relationship to backend blocks are up to the frontend allocator. It
makes more sense now, particularly with regard to the offset. Update
the associated chunk test as the method isn't available there
anymore;
- add a forgotten `\n` to a log string;
- for `releaseToOs`, instead of starting at `1`, start at `0` and
`continue` on `BatchClassId`: in the end it's identical but doesn't
assume a particular class id for batches;
- change a `CHECK` to a `reportOutOfMemory`: it's a clearer message
Reviewers: hctim, morehouse, eugenis, vitalybuka
Reviewed By: hctim
Subscribers: delcypher, #sanitizers, llvm-commits
Tags: #llvm, #sanitizers
Differential Revision: https://reviews.llvm.org/D64570
llvm-svn: 365816
Summary:
The Combined allocator hold together all the other components, and
provides a memory allocator interface based on various template
parameters. This will be in turn used by "wrappers" that will provide
the standard C and C++ memory allocation functions, but can be
used as is as well.
This doesn't depart significantly from the current Scudo implementation
except for a few details:
- Quarantine batches are now protected by a header a well;
- an Allocator instance has its own TSD registry, as opposed to a
static one for everybody;
- a function to iterate over busy chunks has been added, for Android
purposes;
This also adds the associated tests, and a few default configurations
for several platforms, that will likely be further tuned later on.
Reviewers: morehouse, hctim, eugenis, vitalybuka
Reviewed By: morehouse
Subscribers: srhines, mgorny, delcypher, jfb, #sanitizers, llvm-commits
Tags: #llvm, #sanitizers
Differential Revision: https://reviews.llvm.org/D63231
llvm-svn: 363569
Kostya Kortchinsky