tsan while used by golang's race detector was not working on alpine
linux, since it is using musl-c instead of glibc. Since alpine is very
popular distribution for container deployments, having working race
detector would be nice. This commits adds some ifdefs to get it working.
It fixes https://github.com/golang/go/issues/14481 on golang's issue tracker.
Reviewed-in: https://reviews.llvm.org/D75849
Author: graywolf-at-work (Tomas Volf)
realeaseAcquire() is a new function added to TSan in support of the Go data-race detector.
It's semantics is:
void ThreadClock::releaseAcquire(SyncClock *sc) const {
for (int i = 0; i < kMaxThreads; i++) {
tmp = clock[i];
clock[i] = max(clock[i], sc->clock[i]);
sc->clock[i] = tmp;
}
}
For context see: https://go-review.googlesource.com/c/go/+/220419
Reviewed-in: https://reviews.llvm.org/D76322
Author: dfava (Daniel Fava)
Summary:
Don't use weak exports when building tsan into a shared library for Go. gcc can't handle the pragmas used to make the weak references.
Include files that have been added since the last update to build.bat. (We should really find a better way to list all the files needed.)
Add windows version defines (WINVER and _WIN32_WINNT) to get AcquireSRWLockExclusive and ReleaseSRWLockExclusive defined.
Define GetProcessMemoryInfo to use the kernel32 version. This is kind of a hack, the windows header files should do this translation for us. I think we're not in the right family partition (we're using Desktop, but that translation only happens for App and System partitions???), but hacking the family partition seems equally gross and I have no idea what the consequences of that might be.
Patch by Keith Randall.
Reviewers: dvyukov, vitalybuka
Reviewed By: vitalybuka
Subscribers: jfb, delcypher, #sanitizers, llvm-commits
Tags: #llvm, #sanitizers
Differential Revision: https://reviews.llvm.org/D68599
llvm-svn: 373984
Like r367463, but for tsan/{benchmarks,dd,go}.
The files benchmarks aren't referenced in the build anywhere and where added
in 2012 with the comment "no Makefiles yet".
llvm-svn: 367567
See https://reviews.llvm.org/D58620 for discussion, and for the commands
I ran. In addition I also ran
for f in $(svn diff | diffstat | grep .cc | cut -f 2 -d ' '); do rg $f . ; done
and manually updated (many) references to renamed files found by that.
llvm-svn: 367463
Summary:
Combine few relatively small changes into one:
- implement internal_ptrace() and internal_clone() for NetBSD
- add support for stoptheworld based on the ptrace(2) API
- define COMPILER_RT_HAS_LSAN for NetBSD
- enable tests for NetBSD/amd64
Inspired by the original implementation by Christos Zoulas in netbsd/src for GCC.
The implementation is in theory CPU independent through well defined macros
across all NetBSD ports, however only the x86_64 version was tested.
Reviewers: mgorny, dvyukov, vitalybuka, joerg, jfb
Reviewed By: vitalybuka
Subscribers: dexonsmith, jfb, srhines, kubamracek, llvm-commits, christos
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D64057
llvm-svn: 365735
Replace bool workerthread flag with ThreadType enum.
This change is preparation for fiber support.
[dvyukov: fixed build of sanitizer_thread_registry_test.cc]
Author: yuri (Yuri Per)
Reviewed in: https://reviews.llvm.org/D57839
Context: https://reviews.llvm.org/D54889
llvm-svn: 353390
to reflect the new license.
We understand that people may be surprised that we're moving the header
entirely to discuss the new license. We checked this carefully with the
Foundation's lawyer and we believe this is the correct approach.
Essentially, all code in the project is now made available by the LLVM
project under our new license, so you will see that the license headers
include that license only. Some of our contributors have contributed
code under our old license, and accordingly, we have retained a copy of
our old license notice in the top-level files in each project and
repository.
llvm-svn: 351636
Summary:
When possible, switch syscall(2)/__syscall(2) calls
to direct calls of internal libc symbols.
Add a new function to detect address of a libc
symbol of a function that could be intercepted.
With the address detector in GetRealLibcAddress(),
an optional interceptor of libc call will be bypassed.
The original approach with syscall(2)/__syscall(2)
wasn't portable across supported ABIs and CPU
architectures. The indirect syscall interface is
also a candidate for removal in future revisions
of NetBSD, as the C language ABI is not a good
domain for serialization of arbitrary functions
arguments.
Switch the following functions to libc calls:
- internal_mmap()
- internal_munmap()
- internal_mprotect()
- internal_close()
- internal_open()
- internal_read()
- internal_write()
- internal_ftruncate()
- internal_stat()
- internal_lstat()
- internal_fstat()
- internal_dup2()
- internal_readlink()
- internal_unlink()
- internal_rename()
- internal_sched_yield()
- internal__exit()
- internal_sleep()
- internal_execve()
- NanoTime()
- internal_clock_gettime()
- internal_waitpid()
- internal_getpid()
- internal_getppid()
- internal_getdents()
- internal_lseek()
- internal_sigaltstack()
- internal_fork()
- internal_sigprocmask()
- internal_sysctl()
- internal_sigemptyset()
- internal_sigfillset()
- GetTid()
- TgKill()
This revision leaves room for refactoring in subsequent commits.
Reviewers: vitalybuka, kcc, joerg
Reviewed By: vitalybuka
Subscribers: mgorny, fedor.sergeev, jfb, loverszhaokai, devnexen, kubamracek, llvm-commits, ro, #sanitizers
Tags: #sanitizers
Differential Revision: https://reviews.llvm.org/D51419
llvm-svn: 343523
Summary:
This patch adds TSan runtime support for Go on linux-aarch64
platforms. This enables people working on golang to implement their
platform/language part of the TSan support.
Basic testing is done with lib/tsan/go/buildgo.sh. Additional testing will be
done as part of the work done in the Go project.
It is intended to support other VMA sizes, except 39 which does not
have enough bits to support the Go heap requirements.
Patch by Fangming Fang <Fangming.Fang@arm.com>.
Reviewers: kubamracek, dvyukov, javed.absar
Subscribers: mcrosier, dberris, mgorny, kristof.beyls, delcypher, #sanitizers, llvm-commits
Differential Revision: https://reviews.llvm.org/D52167
llvm-svn: 342541
Summary:
Host symbolizer & stacktraces related code in their own RT:
`RTSanitizerCommonSymbolizer`, which is "libcdep" by nature. Symbolizer &
stacktraces specific code that used to live in common files is moved to a new
file `sanitizer_symbolizer_report.cc` as is.
The purpose of this is the enforce a separation between code that relies on
symbolization and code that doesn't. This saves the inclusion of spurious code
due to the interface functions with default visibility, and the extra data
associated.
The following sanitizers makefiles were modified & tested locally:
- dfsan: doesn't require the new symbolizer RT
- esan: requires it
- hwasan: requires it
- lsan: requires it
- msan: requires it
- safestack: doesn't require it
- xray: doesn't require it
- tsan: requires it
- ubsan: requires it
- ubsan_minimal: doesn't require it
- scudo: requires it (but not for Fuchsia that has a minimal runtime)
This was tested locally on Linux, Android, Fuchsia.
Reviewers: alekseyshl, eugenis, dberris, kubamracek, vitalybuka, dvyukov, mcgrathr
Reviewed By: alekseyshl, vitalybuka
Subscribers: srhines, kubamracek, mgorny, krytarowski, delcypher, llvm-commits, #sanitizers
Differential Revision: https://reviews.llvm.org/D45457
llvm-svn: 330131
The current implementation of the Go sanitizer only works on x86_64.
Added some modifications to the buildgo.sh script and the Tsan code
to make it work on powerpc64/linux.
Author: cseo (Carlos Eduardo Seo)
Reviewed in: https://reviews.llvm.org/D43025
llvm-svn: 330122
Summary:
`Symbolizer::PrepareForSandboxing` is empty for all platforms and apparently
has been for a while (D10213). Remove it, and shuffle things around so that the
platform specific code is now in `PlatformPrepareForSandboxing`.
This allows to have one less symbolizer dependency in a common file, which
helps for the upcoming split.
Also remove `SymbolizerPrepareForSandboxing` in tsan_go which appears to not
be used anywhere.
Reviewers: alekseyshl, eugenis, dvyukov, mcgrathr
Reviewed By: alekseyshl
Subscribers: kubamracek, delcypher, llvm-commits, #sanitizers
Differential Revision: https://reviews.llvm.org/D44953
llvm-svn: 329094
This commit breaks actual Go runtime build on gomote builders (10.12) with:
xcode-select: error: tool 'xcodebuild' requires Xcode,
but active developer directory '/Library/Developer/CommandLineTools'
is a command line tools instance
Without this part build works fine.
The original commit does not include any explanation as to why
it is needed.
llvm-svn: 327700
Summary:
Without this diff, the test segfaults. Examining the generated executable
(which gets auto-deleted likely by cmake/ninja) yields this error message:
ThreadSanitizer failed to allocate 0x4000 (16384) bytes at address 1755558480000 (errno: 12)
Note that the address has more than 47 bits, which on amd64 means special
treatment and therefore points out an overflow. The allocation came from
__tsan_map_shadow on a .data pointer, which (on my work Debian-based box)
means the 0x550000000000 range. This doesn't correspond to the constants
mentioned in tsan_platform.h for Go binaries on Linux/amd64.
The diff therefore allocates memory in the sort of area Go programs would,
and prevents the test from crashing. It would be nice if reviewers kindly
considered other setups and architectures :-)
Reviewers: kcc, dvyukov
Subscribers: kubamracek, delcypher, #sanitizers, llvm-commits
Differential Revision: https://reviews.llvm.org/D44071
llvm-svn: 327621
Summary:
Changes:
* Add initial msan stub support.
* Handle NetBSD specific pthread_setname_np(3).
* NetBSD supports __attribute__((tls_model("initial-exec"))),
define it in SANITIZER_TLS_INITIAL_EXEC_ATTRIBUTE.
* Add ReExec() specific bits for NetBSD.
* Simplify code and add syscall64 and syscall_ptr for !NetBSD.
* Correct bunch of syscall wrappers for NetBSD.
* Disable test/tsan/map32bit on NetBSD as not applicable.
* Port test/tsan/strerror_r to a POSIX-compliant OSes.
* Disable __libc_stack_end on NetBSD.
* Disable ReadNullSepFileToArray() on NetBSD.
* Define struct_ElfW_Phdr_sz, detected missing symbol by msan.
* Change type of __sanitizer_FILE from void to char. This helps
to reuse this type as an array. Long term it will be properly
implemented along with SANITIZER_HAS_STRUCT_FILE setting to 1.
* Add initial NetBSD support in lib/tsan/go/buildgo.sh.
* Correct referencing stdout and stderr in tsan_interceptors.cc
on NetBSD.
* Document NetBSD x86_64 specific virtual memory layout in
tsan_platform.h.
* Port tests/rtl/tsan_test_util_posix.cc to NetBSD.
* Enable NetBSD tests in test/msan/lit.cfg.
* Enable NetBSD tests in test/tsan/lit.cfg.
Sponsored by <The NetBSD Foundation>
Reviewers: joerg, vitalybuka, eugenis, kcc, dvyukov
Reviewed By: dvyukov
Subscribers: #sanitizers, llvm-commits, kubamracek
Tags: #sanitizers
Differential Revision: https://reviews.llvm.org/D39124
llvm-svn: 316591
Summary:
This is a pure refactoring change. It just moves code that is
related to filesystem operations from sanitizer_common.{cc,h} to
sanitizer_file.{cc,h}. This makes it cleaner to disable the
filesystem-related code for a new port that doesn't want it.
Submitted on behalf of Roland McGrath.
Reviewers: kcc, eugenis, alekseyshl
Reviewed By: alekseyshl
Subscribers: vitalybuka, llvm-commits, kubamracek, mgorny, phosek
Tags: #sanitizers
Differential Revision: https://reviews.llvm.org/D35591
llvm-svn: 308819
This is a pure refactoring change. It just moves code that is
related to filesystem operations from sanitizer_common.{cc,h} to
sanitizer_file.{cc,h}. This makes it cleaner to disable the
filesystem-related code for a new port that doesn't want it.
Commiting for mcgrathr.
Reviewers: alekseyshl
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D35591
llvm-svn: 308640
To make the TSan external API work with Swift and other use cases, we need to track "tags" for individual memory accesses. Since there is no space to store this information in shadow cells, let's use the thread traces for that. This patch stores the tag as an extra frame in the stack traces (by calling FuncEntry and FuncExit with the address of a registered tag), this extra frame is then stripped before printing the backtrace to stderr.
Differential Revision: https://reviews.llvm.org/D32382
llvm-svn: 301777
There are several problems with the current annotations (AnnotateRWLockCreate and friends):
- they don't fully support deadlock detection (we need a hook _before_ mutex lock)
- they don't support insertion of random artificial delays to perturb execution (again we need a hook _before_ mutex lock)
- they don't support setting extended mutex attributes like read/write reentrancy (only "linker init" was bolted on)
- they don't support setting mutex attributes if a mutex don't have a "constructor" (e.g. static, Java, Go mutexes)
- they don't ignore synchronization inside of lock/unlock operations which leads to slowdown and false negatives
The new annotations solve of the above problems. See tsan_interface.h for the interface specification and comments.
Reviewed in https://reviews.llvm.org/D31093
llvm-svn: 298809
When dealing with GCD worker threads, TSan currently prints weird things like "created by thread T-1" and "[failed to restore the stack]" in reports. This patch avoids that and instead prints "Thread T3 (...) is a GCD worker thread".
Differential Revision: https://reviews.llvm.org/D29103
llvm-svn: 293882
Currently we either define SANITIZER_GO for Go or don't define it at all for C++.
This works fine with preprocessor (ifdef/ifndef/defined), but does not work
for C++ if statements (e.g. if (SANITIZER_GO) {...}). Also this is different
from majority of SANITIZER_FOO macros which are always defined to either 0 or 1.
Always define SANITIZER_GO to either 0 or 1.
This allows to use SANITIZER_GO in expressions and in flag default values.
Also remove kGoMode and kCppMode, which were meant to be used in expressions,
but they are not defined in sanitizer_common code, so SANITIZER_GO become prevalent.
Also convert some preprocessor checks to C++ if's or ternary expressions.
Majority of this change is done mechanically with:
sed "s#ifdef SANITIZER_GO#if SANITIZER_GO#g"
sed "s#ifndef SANITIZER_GO#if \!SANITIZER_GO#g"
sed "s#defined(SANITIZER_GO)#SANITIZER_GO#g"
llvm-svn: 285443
We're not building the Go runtime with -mmacosx-version-min, which means it'll have a minimum deployment target set to the system you're building on. Let's make the code compile (and link) with -mmacosx-version-min=10.7.
Differential Revision: http://reviews.llvm.org/D20670
llvm-svn: 271833
Summary:
As suggested by kcc@ in http://reviews.llvm.org/D20084#441418, move the CheckFailed and Die functions, and their associated callback functionalities in their own separate file.
I expended the build rules to include a new rule that would not include those termination functions, so that another project can define their own.
The tests check-{a,t,m,ub,l,e,df}san are all passing.
Reviewers: llvm-commits, kcc
Subscribers: kubabrecka
Differential Revision: http://reviews.llvm.org/D20742
llvm-svn: 271055
Current interface assumes that Go calls ProcWire/ProcUnwire
to establish the association between thread and proc.
With the wisdom of hindsight, this interface does not work
very well. I had to sprinkle Go scheduler with wire/unwire
calls, and any mistake leads to hard to debug crashes.
This is not something one wants to maintian.
Fortunately, there is a simpler solution. We can ask Go
runtime as to what is the current Processor, and that
question is very easy to answer on Go side.
Switch to such interface.
llvm-svn: 267703
This is reincarnation of http://reviews.llvm.org/D17648 with the bug fix pointed out by Adhemerval (zatrazz).
Currently ThreadState holds both logical state (required for race-detection algorithm, user-visible)
and physical state (various caches, most notably malloc cache). Move physical state in a new
Process entity. Besides just being the right thing from abstraction point of view, this solves several
problems:
Cache everything on P level in Go. Currently we cache on a mix of goroutine and OS thread levels.
This unnecessary increases memory consumption.
Properly handle free operations in Go. Frees are issue by GC which don't have goroutine context.
As the result we could not do anything more than just clearing shadow. For example, we leaked
sync objects and heap block descriptors.
This will allow to get rid of libc malloc in Go (now we have Processor context for internal allocator cache).
This in turn will allow to get rid of dependency on libc entirely.
Potentially we can make Processor per-CPU in C++ mode instead of per-thread, which will
reduce resource consumption.
The distinction between Thread and Processor is currently used only by Go, C++ creates Processor per OS thread,
which is equivalent to the current scheme.
llvm-svn: 267678
Currently ThreadState holds both logical state (required for race-detection algorithm, user-visible)
and physical state (various caches, most notably malloc cache). Move physical state in a new
Process entity. Besides just being the right thing from abstraction point of view, this solves several
problems:
1. Cache everything on P level in Go. Currently we cache on a mix of goroutine and OS thread levels.
This unnecessary increases memory consumption.
2. Properly handle free operations in Go. Frees are issue by GC which don't have goroutine context.
As the result we could not do anything more than just clearing shadow. For example, we leaked
sync objects and heap block descriptors.
3. This will allow to get rid of libc malloc in Go (now we have Processor context for internal allocator cache).
This in turn will allow to get rid of dependency on libc entirely.
4. Potentially we can make Processor per-CPU in C++ mode instead of per-thread, which will
reduce resource consumption.
The distinction between Thread and Processor is currently used only by Go, C++ creates Processor per OS thread,
which is equivalent to the current scheme.
llvm-svn: 262037
FreeBSD also needs to have sanitizer_linux_libcdep.cc included,
otherwise linking will fail with "undefined reference to
`__sanitizer::GetRSS()'".
While here, tabify the FreeBSD part, similar to the other parts.
llvm-svn: 260839
When ASan currently detects a bug, by default it will only print out the text
of the report to stderr. This patch changes this behavior and writes the full
text of the report to syslog before we terminate the process. It also calls
os_trace (Activity Tracing available on OS X and iOS) with a message saying
that the report is available in syslog. This is useful, because this message
will be shown in the crash log.
For this to work, the patch makes sure we store the full report into
error_message_buffer unconditionally, and it also strips out ANSI escape
sequences from the report (they are used when producing colored reports).
I've initially tried to log to syslog during printing, which is done on Android
right now. The advantage is that if we crash during error reporting or the
produced error does not go through ScopedInErrorReport, we would still get a
(partial) message in the syslog. However, that solution is very problematic on
OS X. One issue is that the logging routine uses GCD, which may spawn a new
thread on its behalf. In many cases, the reporting logic locks threadRegistry,
which leads to deadlocks.
Reviewed at http://reviews.llvm.org/D13452
(In addition, add sanitizer_common_libcdep.cc to buildgo.sh to avoid
build failures on Linux.)
llvm-svn: 253688
When ASan currently detects a bug, by default it will only print out the text
of the report to stderr. This patch changes this behavior and writes the full
text of the report to syslog before we terminate the process. It also calls
os_trace (Activity Tracing available on OS X and iOS) with a message saying
that the report is available in syslog. This is useful, because this message
will be shown in the crash log.
For this to work, the patch makes sure we store the full report into
error_message_buffer unconditionally, and it also strips out ANSI escape
sequences from the report (they are used when producing colored reports).
I've initially tried to log to syslog during printing, which is done on Android
right now. The advantage is that if we crash during error reporting or the
produced error does not go through ScopedInErrorReport, we would still get a
(partial) message in the syslog. However, that solution is very problematic on
OS X. One issue is that the logging routine uses GCD, which may spawn a new
thread on its behalf. In many cases, the reporting logic locks threadRegistry,
which leads to deadlocks.
Reviewed at http://reviews.llvm.org/D13452
(In addition, add sanitizer_common_libcdep.cc to buildgo.sh to avoid
build failures on Linux.)
llvm-svn: 251577
Previously, Android target had a logic of duplicating all sanitizer
output to logcat. This change extends it to all posix platforms via
the use of syslog, controlled by log_to_syslog flag. Enabled by
default on Android, off everywhere else.
A bit of cmake magic is required to allow Printf() to call a libc
function. I'm adding a stub implementation to support no-libc builds
like dfsan and safestack.
This is a second attempt. I believe I've fixed all the issues that
prompted the revert: Mac build, and all kinds of non-CMake builds
(there are 3 of those).
llvm-svn: 243051
This is done by creating a named shared memory region, unlinking it
and setting up a private (i.e. copy-on-write) mapping of that instead
of a regular anonymous mapping. I've experimented with regular
(sparse) files, but they can not be scaled to the size of MSan shadow
mapping, at least on Linux/X86_64 and ext3 fs.
Controlled by a common flag, decorate_proc_maps, disabled by default.
This patch has a few shortcomings:
* not all mappings are annotated, especially in TSan.
* our handling of memset() of shadow via mmap() puts small anonymous
mappings inside larger named mappings, which looks ugly and can, in
theory, hit the mapping number limit.
llvm-svn: 238621
In Go mode the background thread is not started (internal_thread_start is empty).
There is no sense in having this code compiled in.
Also removes dependency on sanitizer_linux_libcdep.cc which is good,
ideally Go runtime does not depend on libc at all.
llvm-svn: 229396
TSAN_SHADOW_COUNT is defined to 4 in all environments.
Other values of TSAN_SHADOW_COUNT were never tested and
were broken by recent changes to shadow mapping.
Remove it as there is no reason to fix nor maintain it.
llvm-svn: 226466
Return a linked list of AddressInfo objects, instead of using an array of
these objects as an output parameter. This simplifies the code in callers
of this function (especially TSan).
Fix a few memory leaks from internal allocator, when the returned
AddressInfo objects were not properly cleared.
llvm-svn: 223145
AddressInfo contains the results of symbolization. Store this object
directly in the symbolized stack, instead of copying data around and
making unnecessary memory allocations.
No functionality change.
llvm-svn: 221294
The new storage (MetaMap) is based on direct shadow (instead of a hashmap + per-block lists).
This solves a number of problems:
- eliminates quadratic behaviour in SyncTab::GetAndLock (https://code.google.com/p/thread-sanitizer/issues/detail?id=26)
- eliminates contention in SyncTab
- eliminates contention in internal allocator during allocation of sync objects
- removes a bunch of ad-hoc code in java interface
- reduces java shadow from 2x to 1/2x
- allows to memorize heap block meta info for Java and Go
- allows to cleanup sync object meta info for Go
- which in turn enabled deadlock detector for Go
llvm-svn: 209810
Introduce DDetector interface between the tool and the DD itself.
It will help to experiment with other DD implementation,
as well as reuse DD in other tools.
llvm-svn: 202485
This is intended to address the following problem.
Episodically we see CHECK-failures when recursive interceptors call back into user code. Effectively we are not "in_rtl" at this point, but it's very complicated and fragile to properly maintain in_rtl property. Instead get rid of it. It was used mostly for sanity CHECKs, which basically never uncover real problems.
Instead introduce ignore_interceptors flag, which is used in very few narrow places to disable recursive interceptors (e.g. during runtime initialization).
llvm-svn: 197979
The flag allows to bound maximum process memory consumption (best effort).
If RSS reaches memory_limit_mb, tsan flushes all shadow memory.
llvm-svn: 191913
Dmitry Vyukov