forked from OSchip/llvm-project
8 Commits
| Author | SHA1 | Message | Date |
|---|---|---|---|
Kostya Kortchinsky
|
bd5ca4f0ed |
[scudo][standalone] Skip irrelevant regions during release
With the 'new' way of releasing on 32-bit, we iterate through all the regions in between `First` and `Last`, which covers regions that do not belong to the class size we are working with. This is effectively wasted cycles. With this change, we add a `SkipRegion` lambda to `releaseFreeMemoryToOS` that will allow the release function to know when to skip a region. For the 64-bit primary, since we are only working with 1 region, we never skip. Reviewed By: hctim Differential Revision: https://reviews.llvm.org/D86399 |
|
|
Kostya Kortchinsky
|
998334da2b |
[scudo][standalone] Change the release loop for efficiency purposes
Summary: On 32-b, the release algo loops multiple times over the freelist for a size class, which lead to a decrease in performance when there were a lot of free blocks. This changes the release functions to loop only once over the freelist, at the cost of using a little bit more memory for the release process: instead of working on one region at a time, we pass the whole memory area covered by all the regions for a given size class, and work on sub-areas of `RegionSize` in this large area. For 64-b, we just have 1 sub-area encompassing the whole region. Of course, not all the sub-areas within that large memory area will belong to the class id we are working on, but those will just be left untouched (which will not add to the RSS during the release process). Reviewers: pcc, cferris, hctim, eugenis Subscribers: llvm-commits, #sanitizers Tags: #sanitizers Differential Revision: https://reviews.llvm.org/D83993 |
|
|
Kostya Kortchinsky
|
79de8f8441 |
[scudo][standalone] Release smaller blocks less often
Summary: Releasing smaller blocks is costly and only yields significant results when there is a large percentage of free bytes for a given size class (see numbers below). This CL introduces a couple of additional checks for sizes lower than 256. First we want to make sure that there is enough free bytes, relatively to the amount of allocated bytes. We are looking at 8X% to 9X% (smaller blocks require higher percentage). We also want to make sure there has been enough activity with the freelist to make it worth the time, so we now check that the bytes pushed to the freelist is at least 1/16th of the allocated bytes for those classes. Additionally, we clear batches before destroying them now - this could have prevented some releases to occur (class id 0 rarely releases anyway). Here are the numbers, for about 1M allocations in multiple threads: Size: 16 85% freed -> 0% released 86% freed -> 0% released 87% freed -> 0% released 88% freed -> 0% released 89% freed -> 0% released 90% freed -> 0% released 91% freed -> 0% released 92% freed -> 0% released 93% freed -> 0% released 94% freed -> 0% released 95% freed -> 0% released 96% freed -> 0% released 97% freed -> 2% released 98% freed -> 7% released 99% freed -> 27% released Size: 32 85% freed -> 0% released 86% freed -> 0% released 87% freed -> 0% released 88% freed -> 0% released 89% freed -> 0% released 90% freed -> 0% released 91% freed -> 0% released 92% freed -> 0% released 93% freed -> 0% released 94% freed -> 0% released 95% freed -> 1% released 96% freed -> 3% released 97% freed -> 7% released 98% freed -> 17% released 99% freed -> 41% released Size: 48 85% freed -> 0% released 86% freed -> 0% released 87% freed -> 0% released 88% freed -> 0% released 89% freed -> 0% released 90% freed -> 0% released 91% freed -> 0% released 92% freed -> 0% released 93% freed -> 0% released 94% freed -> 1% released 95% freed -> 3% released 96% freed -> 7% released 97% freed -> 13% released 98% freed -> 27% released 99% freed -> 52% released Size: 64 85% freed -> 0% released 86% freed -> 0% released 87% freed -> 0% released 88% freed -> 0% released 89% freed -> 0% released 90% freed -> 0% released 91% freed -> 0% released 92% freed -> 1% released 93% freed -> 2% released 94% freed -> 3% released 95% freed -> 6% released 96% freed -> 11% released 97% freed -> 20% released 98% freed -> 35% released 99% freed -> 59% released Size: 80 85% freed -> 0% released 86% freed -> 0% released 87% freed -> 0% released 88% freed -> 0% released 89% freed -> 0% released 90% freed -> 1% released 91% freed -> 1% released 92% freed -> 2% released 93% freed -> 4% released 94% freed -> 6% released 95% freed -> 10% released 96% freed -> 17% released 97% freed -> 26% released 98% freed -> 41% released 99% freed -> 64% released Size: 96 85% freed -> 0% released 86% freed -> 0% released 87% freed -> 0% released 88% freed -> 0% released 89% freed -> 1% released 90% freed -> 1% released 91% freed -> 3% released 92% freed -> 4% released 93% freed -> 6% released 94% freed -> 10% released 95% freed -> 14% released 96% freed -> 21% released 97% freed -> 31% released 98% freed -> 47% released 99% freed -> 68% released Size: 112 85% freed -> 0% released 86% freed -> 1% released 87% freed -> 1% released 88% freed -> 2% released 89% freed -> 3% released 90% freed -> 4% released 91% freed -> 6% released 92% freed -> 8% released 93% freed -> 11% released 94% freed -> 16% released 95% freed -> 22% released 96% freed -> 30% released 97% freed -> 40% released 98% freed -> 55% released 99% freed -> 74% released Size: 128 85% freed -> 0% released 86% freed -> 1% released 87% freed -> 1% released 88% freed -> 2% released 89% freed -> 3% released 90% freed -> 4% released 91% freed -> 6% released 92% freed -> 8% released 93% freed -> 11% released 94% freed -> 16% released 95% freed -> 22% released 96% freed -> 30% released 97% freed -> 40% released 98% freed -> 55% released 99% freed -> 74% released Size: 144 85% freed -> 1% released 86% freed -> 2% released 87% freed -> 3% released 88% freed -> 4% released 89% freed -> 6% released 90% freed -> 7% released 91% freed -> 10% released 92% freed -> 13% released 93% freed -> 17% released 94% freed -> 22% released 95% freed -> 28% released 96% freed -> 37% released 97% freed -> 47% released 98% freed -> 61% released 99% freed -> 78% released Size: 160 85% freed -> 1% released 86% freed -> 2% released 87% freed -> 3% released 88% freed -> 4% released 89% freed -> 5% released 90% freed -> 7% released 91% freed -> 10% released 92% freed -> 13% released 93% freed -> 17% released 94% freed -> 22% released 95% freed -> 28% released 96% freed -> 37% released 97% freed -> 47% released 98% freed -> 61% released 99% freed -> 78% released Size: 176 85% freed -> 2% released 86% freed -> 3% released 87% freed -> 4% released 88% freed -> 6% released 89% freed -> 7% released 90% freed -> 9% released 91% freed -> 12% released 92% freed -> 15% released 93% freed -> 20% released 94% freed -> 25% released 95% freed -> 32% released 96% freed -> 40% released 97% freed -> 51% released 98% freed -> 64% released 99% freed -> 80% released Size: 192 85% freed -> 4% released 86% freed -> 5% released 87% freed -> 6% released 88% freed -> 8% released 89% freed -> 10% released 90% freed -> 13% released 91% freed -> 16% released 92% freed -> 20% released 93% freed -> 24% released 94% freed -> 30% released 95% freed -> 37% released 96% freed -> 45% released 97% freed -> 55% released 98% freed -> 68% released 99% freed -> 82% released Size: 224 85% freed -> 8% released 86% freed -> 10% released 87% freed -> 12% released 88% freed -> 14% released 89% freed -> 17% released 90% freed -> 20% released 91% freed -> 23% released 92% freed -> 28% released 93% freed -> 33% released 94% freed -> 39% released 95% freed -> 46% released 96% freed -> 53% released 97% freed -> 63% released 98% freed -> 73% released 99% freed -> 85% released Size: 240 85% freed -> 8% released 86% freed -> 10% released 87% freed -> 12% released 88% freed -> 14% released 89% freed -> 17% released 90% freed -> 20% released 91% freed -> 23% released 92% freed -> 28% released 93% freed -> 33% released 94% freed -> 39% released 95% freed -> 46% released 96% freed -> 54% released 97% freed -> 63% released 98% freed -> 73% released 99% freed -> 85% released Reviewers: cferris, pcc, hctim, eugenis Subscribers: #sanitizers, llvm-commits Tags: #sanitizers Differential Revision: https://reviews.llvm.org/D82031 |
|
|
Kostya Kortchinsky
|
c753a306fd |
[scudo][standalone] Various improvements wrt RSS
Summary: This patch includes several changes to reduce the overall footprint of the allocator: - for realloc'd chunks: only keep the same chunk when lowering the size if the delta is within a page worth of bytes; - when draining a cache: drain the beginning, not the end; we add pointers at the end, so that meant we were draining the most recently added pointers; - change the release code to account for an freed up last page: when scanning the pages, we were looking for pages fully covered by blocks; in the event of the last page, if it's only partially covered, we wouldn't mark it as releasable - even what follows the last chunk is all 0s. So now mark the rest of the page as releasable, and adapt the test; - add a missing `setReleaseToOsIntervalMs` to the cacheless secondary; - adjust the Android classes based on more captures thanks to pcc@'s tool. Reviewers: pcc, cferris, hctim, eugenis Subscribers: #sanitizers, llvm-commits Tags: #sanitizers, #llvm Differential Revision: https://reviews.llvm.org/D75142 |
|
|
Kostya Kortchinsky
|
fc69967a4b |
[scudo][standalone] Shift some data from dynamic to static
Summary: Most of our larger data is dynamically allocated (via `map`) but it became an hindrance with regard to init time, for a cost to benefit ratio that is not great. So change the `TSD`s, `RegionInfo`, `ByteMap` to be static. Additionally, for reclaiming, we used mapped & unmapped a buffer each time, which is costly. It turns out that we can have a static buffer, and that there isn't much contention on it. One of the other things changed here, is that we hard set the number of TSDs on Android to the maximum number, as there could be a situation where cores are put to sleep and we could miss some. Subscribers: mgorny, #sanitizers, llvm-commits Tags: #sanitizers, #llvm Differential Revision: https://reviews.llvm.org/D74696 |
|
|
Kostya Kortchinsky
|
993e3c9269 |
[scudo][standalone] Secondary & general other improvements
Summary: This CL changes multiple things to improve performance (notably on Android).We introduce a cache class for the Secondary that is taking care of this mechanism now. The changes: - change the Secondary "freelist" to an array. By keeping free secondary blocks linked together through their headers, we were keeping a page per block, which isn't great. Also we know touch less pages when walking the new "freelist". - fix an issue with the freelist getting full: if the pattern is an ever increasing size malloc then free, the freelist would fill up and entries would not be used. So now we empty the list if we get to many "full" events; - use the global release to os interval option for the secondary: it was too costly to release all the time, particularly for pattern that are malloc(X)/free(X)/malloc(X). Now the release will only occur after the selected interval, when going through the deallocate path; - allow release of the `BatchClassId` class: it is releasable, we just have to make sure we don't mark the batches containing batches pointers as free. - change the default release interval to 1s for Android to match the current Bionic allocator configuration. A patch is coming up to allow changing it through `mallopt`. - lower the smallest class that can be released to `PageSize/64`. Reviewers: cferris, pcc, eugenis, morehouse, hctim Subscribers: phosek, #sanitizers, llvm-commits Tags: #sanitizers, #llvm Differential Revision: https://reviews.llvm.org/D73507 |
|
|
Kostya Kortchinsky
|
6f2de9cbb3 |
[scudo][standalone] Consolidate lists
Summary: This is a clean patch using the last diff of D69265, but using git instead of svn, since svn went ro and arc was making my life harded than it needed to be. I was going to introduce a couple more lists and realized that our lists are currently a bit all over the place. While we have a singly linked list type relatively well defined, we are using doubly linked lists defined on the fly for the stats and for the secondary blocks. This CL adds a doubly linked list object, reorganizing the singly list one to extract as much of the common code as possible. We use this new type in the stats and the secondary. We also reorganize the list tests to benefit from this consolidation. There are a few side effect changes such as using for iterator loops that are, in my opinion, cleaner in a couple of places. Reviewers: hctim, morehouse, pcc, cferris Reviewed By: hctim Subscribers: jfb, #sanitizers, llvm-commits Tags: #sanitizers, #llvm Differential Revision: https://reviews.llvm.org/D69516 |
|
|
Kostya Kortchinsky
|
21c31f5e7b |
[scudo][standalone] Add the memory reclaiming mechanism
Summary: This CL implements the memory reclaiming function `releaseFreeMemoryToOS` and its associated classes. Most of this code was originally written by Aleksey for the Primary64 in sanitizer_common, and I made some changes to be able to implement 32-bit reclaiming as well. The code has be restructured a bit to accomodate for freelist of batches instead of the freearray used in the current sanitizer_common code. Reviewers: eugenis, vitalybuka, morehouse, hctim Reviewed By: vitalybuka Subscribers: srhines, mgorny, delcypher, #sanitizers, llvm-commits Tags: #llvm, #sanitizers Differential Revision: https://reviews.llvm.org/D61214 llvm-svn: 359567 |