Summary:
1. Do not create DFSan labels for the bytes which we do not trace. This is where we run out of labels at the first place.
2. When dumping the traces on the disk, make sure to offset the label identifiers by the number of the first byte in the trace range.
3. For the last label, make sure to write it at the last position of the trace bit string, as that label represents the input size, not any particular byte.
Also fixed the bug with division in python which I've introduced when migrated the scripts to Python3 (`//` is required for integral division).
Otherwise, the scripts are wasting too much time unsuccessfully trying to
collect and process traces from the long inputs. For more context, see
https://github.com/google/oss-fuzz/issues/1632#issuecomment-481761789
Reviewers: kcc
Reviewed By: kcc
Subscribers: delcypher, #sanitizers, llvm-commits
Tags: #llvm, #sanitizers
Differential Revision: https://reviews.llvm.org/D60538
llvm-svn: 358311
Summary:
Before this change, MutateWithMask used to assert that Mask should be
of sufficient length (>= Size of the input). However, in real cases we may have
inputs that are longer than the Mask they have inherited from the based inputs.
Reviewers: kcc, morehouse
Reviewed By: kcc
Subscribers: delcypher, #sanitizers, llvm-commits
Tags: #llvm, #sanitizers
Differential Revision: https://reviews.llvm.org/D60571
llvm-svn: 358207
Summary:
In case the current corpus input doesn't have bytes going into the
focus function, MutateWithMask is useless and may fail gently, allowing the
default mutation routine happen, rather than crashing on an assertion.
For more context and the initial fix suggestion, see:
https://github.com/google/oss-fuzz/issues/1632#issuecomment-481862879
Reviewers: kcc, morehouse
Reviewed By: kcc
Subscribers: delcypher, #sanitizers, llvm-commits
Tags: #llvm, #sanitizers
Differential Revision: https://reviews.llvm.org/D60567
llvm-svn: 358190
Summary:
When using libfuzzer on Windows, in the contents of a crash sample, bytes that can be mistaken for a \n are replaced by a \r\n sequence. As a consequence, crashes are not reproducible. This patch will open files in binary mode to fix this issue. The patch does not affect POSIX systems.
Patch by tuktuk
Reviewers: kcc, vitalybuka
Reviewed By: vitalybuka
Subscribers: dexonsmith, jdoerfert, llvm-commits, #sanitizers
Tags: #llvm, #sanitizers
Differential Revision: https://reviews.llvm.org/D60008
llvm-svn: 357807
Summary:
This fixes inconsistent symbol visibility. This shows up as a linker
warning if r336238 (43f633564e338a6dde83d49a48e5bfcbfdce292c) is
reverted.
```
ld: warning: direct access in function 'fuzzer::CleanseCrashInput(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, fuzzer::fuzzer_allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, fuzzer::FuzzingOptions const&)' from file '/Volumes/data/dev/llvm/upstream/master/builds/projects/compiler-rt/lib/fuzzer/tests/libRTFuzzerTest.x86_64.a(FuzzerDriver.cpp.o)' to global weak symbol 'fuzzer::Command::ignoreRemainingArgs()::kIgnoreRemaining' from file 'FuzzerTestObjects.FuzzerUnittest.cpp.x86_64.o' means the weak symbol cannot be overridden
at runtime. This was likely caused by different translation units being compiled with different visibility settings.
```
r336238 just hid the issue rather than fixing the real issue. On macOS
and other platforms we usually compile with `-fvisibility=hidden` but
the unit tests were compiled without this flag.
Reviewers: george.karpenkov, kubamracek, kcc, yln
Subscribers: mgorny, #sanitizers, llvm-commits
Tags: #llvm, #sanitizers
Differential Revision: https://reviews.llvm.org/D58055
llvm-svn: 355143
Summary:
Port libFuzzer's fork mode to Windows.
Implement Windows versions of MkDir, RmDir, and IterateDirRecursive to do this.
Don't print error messages under new normal uses of FileSize (on a non-existent file).
Implement portable way of piping output to /dev/null.
Fix test for Windows and comment fork-sigusr.test on why it won't be ported to Win.
Reviewers: zturner
Reviewed By: zturner
Subscribers: kcc, zturner, jdoerfert, #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D58513
llvm-svn: 355019
Summary:
When running the standalone main on a large corpus, I eventually get a
EMFILE error ("Too many open files").
Patch by Paul Chaignon
Reviewers: kcc, vitalybuka
Reviewed By: vitalybuka
Subscribers: lebedev.ri, #sanitizers, llvm-commits
Tags: #llvm, #sanitizers
Differential Revision: https://reviews.llvm.org/D58622
llvm-svn: 354918
According to the logs and local debugging there were two issues:
1) tsan tests listed libc++.a before the source file. That's usually
ok for shared libraries, but the linker will not add symbols from
a static library unless needed at that time. As a result the tests
that rely upon symbols from the library (and not only include the
headers) had undefined references.
To solve this I'm adding a new substitution %link_libcxx_tsan which
expands to libc++.a if available.
2) The target Fuzzer-x86_64-Test linked in SANITIZER_TEST_CXX_LIBRARIES
which defaults to -lstdc++. This resulted in error messages like
hidden symbol '_ZdlPv' is not defined locally
hidden symbol '_Znwm' is not defined locally
when using GNU gold (ld.bfd and lld are fine). Removing the linkage
is fine because we build a custom libc++ for that purpose.
llvm-svn: 354231
This changes add_custom_libcxx to also build libcxxabi and merges
the two into a static and hermetic library.
There are multiple advantages:
1) The resulting libFuzzer doesn't expose C++ internals and looks
like a plain C library.
2) We don't have to manually link in libstdc++ to provide cxxabi.
3) The sanitizer tests cannot interfere with an installed version
of libc++.so in LD_LIBRARY_PATH.
Differential Revision: https://reviews.llvm.org/D58013
llvm-svn: 354212
Add missed value "libcxxabi" and introduce SANITIZER_TEST_CXX for linking
unit tests. This needs to be a full C++ library and cannot be libcxxabi.
Recommit r354132 which I reverted in r354153 because it broke a sanitizer
bot. This was because of the "fixes" for pthread linking, so I've removed
these changes.
Differential Revision: https://reviews.llvm.org/D58012
llvm-svn: 354198
Add missed value "libcxxabi" and introduce SANITIZER_TEST_CXX for linking
unit tests. This needs to be a full C++ library and cannot be libcxxabi.
Differential Revision: https://reviews.llvm.org/D58012
llvm-svn: 354132
The __sanitizer_acquire_crash_state function has int return type, but the
fuzzer's external function definitions give it bool.
Places where __sanitizer_acquire_crash_state is declared:
include/sanitizer_common/sanitizer_interface_defs.h
lib/sanitizer_common/sanitizer_interface_internal.h
lib/sanitizer_common/sanitizer_common.cc
lib/fuzzer/FuzzerExtFunctions.def (this is the only bool)
llvm-svn: 353596
Summary:
Use `_write` instead of the deprecated alias `write` on Windows.
Also, remove comment saying RawPrint is untested on Windows.
Reviewers: vitalybuka
Reviewed By: vitalybuka
Subscribers: vitalybuka
Differential Revision: https://reviews.llvm.org/D57589
llvm-svn: 353108
Summary:
Use RawPrint instead of Printf for instrumentation warning because
Printf doesn't work on Win when instrumentation is being
initialized (since OutputFile is not yet initialized).
Reviewers: kcc
Reviewed By: kcc
Differential Revision: https://reviews.llvm.org/D57531
llvm-svn: 352789
Summary:
Set default `ASAN_OPTIONS` when running libFuzzer tests. This allows us
to remove special casing in code for Darwin where we usually pass
`abort_on_error=0` to override platform defaults for tests.
A previous commit changed the code to make the tests pass:
7764a04af0
Adapted a few tests to use `%env_asan_opts=` instead of directly setting
the environment variable.
rdar://problem/47515276
Reviewers: kcc, george.karpenkov
Differential Revision: https://reviews.llvm.org/D57465
llvm-svn: 352711
Summary:
Use dllexport for all declarations in FuzzerInterface.h Use it for clang
even though clang supports default visibility attribute to prevent a
warning from being thrown when LLVMFuzzerMutate is defined with dllexport.
This makes `FUZZER_INTERFACE_VISIBILITY` (FuzzerInterface.h) consistent with
`ATTRIBUTE_INTERFACE` (FuzzerDefs.h) when using clang on Windows.
Reviewers: vitalybuka, morehouse
Reviewed By: morehouse
Differential Revision: https://reviews.llvm.org/D57305
llvm-svn: 352395
Summary:
Remove this feature as it is unused, buggy, and not worth correcting
since the forkserver makes it difficult.
Reviewers: morehouse, jfb
Reviewed By: morehouse
Differential Revision: https://reviews.llvm.org/D57308
llvm-svn: 352392
Summary:
Enable building libFuzzer with MSVC.
* Don't try to include <endian.h> in FuzzerSHA1.cpp. MSVC
doesn't have this header, and WINDOWS is always little
endian (even on ARM)
Subscribers: srhines, mgorny, javed.absar, kristof.beyls
Differential Revision: https://reviews.llvm.org/D56510
llvm-svn: 351855
to reflect the new license.
We understand that people may be surprised that we're moving the header
entirely to discuss the new license. We checked this carefully with the
Foundation's lawyer and we believe this is the correct approach.
Essentially, all code in the project is now made available by the LLVM
project under our new license, so you will see that the license headers
include that license only. Some of our contributors have contributed
code under our old license, and accordingly, we have retained a copy of
our old license notice in the top-level files in each project and
repository.
llvm-svn: 351636
Summary:
Remove code for handling unstable edges from libFuzzer since
it has not been found useful.
Differential Revision: https://reviews.llvm.org/D56730
llvm-svn: 351262
Summary:
Use alternatename for external functions only when using
MSVC since Clang doesn't support it and MSVC doesn't support
Clang's method (weak aliases).
Reviewers: morehouse
Reviewed By: morehouse
Subscribers: rnk, thakis, mgorny
Differential Revision: https://reviews.llvm.org/D56514
llvm-svn: 351152
Summary:
Replace calls to builtin functions with macros or functions that call the
Windows-equivalents when targeting windows and call the original
builtin functions everywhere else.
This change makes more parts of libFuzzer buildable with MSVC.
Reviewers: vitalybuka
Reviewed By: vitalybuka
Subscribers: mgorny, rnk, thakis
Differential Revision: https://reviews.llvm.org/D56439
llvm-svn: 350766
Include CompilerRTCompile in fuzzer tests explicitly. Otherwise, when
building only libFuzzer, CMake fails due to:
CMake Error at cmake/Modules/AddCompilerRT.cmake:395 (sanitizer_test_compile):
Unknown CMake command "sanitizer_test_compile".
Call Stack (most recent call first):
lib/fuzzer/tests/CMakeLists.txt:53 (generate_compiler_rt_tests)
Differential Revision: https://reviews.llvm.org/D55378
llvm-svn: 348524
This fixes the issue introduced in r345765 which changed the way in
which the embedded libc++ is being built but omitted tests.
Differential Revision: https://reviews.llvm.org/D54058
llvm-svn: 346052
libc++ now supports customizing the ABI namespace directly from the
CMake build so we no longer need to rely on custom CFLAGS.
Differential Revision: https://reviews.llvm.org/D53939
llvm-svn: 345765
Summary:
Silence warning when linking unittest binary by not passing
-lstdc++ to the linker since it is ignored.
Reviewers: morehouse
Reviewed By: morehouse
Subscribers: mgorny
Differential Revision: https://reviews.llvm.org/D53225
llvm-svn: 344480
Without this change, tests in coverage.test and dump_coverage.test are
failing on non-x86_64 platforms.
The diff is copied from sanitizer_common library, an alternative would
be to link it together with libFuzzer.
Differential Revision: https://reviews.llvm.org/D53040
llvm-svn: 344104
This change fixes two aspects of logging on Fuchsia:
1. When logging to stdout, the output file descriptor should not be
closed in ExecuteCommand, as it prevent fdio_spawn_etc from succeeding
in subsequent calls.
2. When logging to a file in anything other than standalone Zircon, the
log file needs to be created in mutable storage. The best indicator
where this is relative to the Fuchsia component will be given by
'-artifact_prefix=...', so save the log file relative to that path.
Patch by: aarongreen
Differential Revision: https://reviews.llvm.org/D52562
llvm-svn: 343607
Summary:
Replace FuzzerExtFunctionsDlsymWin.cpp with FuzzerExtFunctionsWeakAlias.cpp
to get externally defined functions (eg: LLVMFuzzerInitialize,
LLVMFuzzerCustomMutator, etc) working again.
Also enable tests that depended on these functions (on windows)
Reviewers: rnk, morehouse
Reviewed By: rnk, morehouse
Subscribers: rnk, morehouse, mgorny
Differential Revision: https://reviews.llvm.org/D51700
llvm-svn: 342698
This fixes building on a case sensitive filesystem with mingw-w64
headers, where all headers are lowercase, and matches how these
headers are included elsewhere in compiler-rt.
Also include these headers with angle brackets, as they are system
headers.
Differential Revision: https://reviews.llvm.org/D51913
llvm-svn: 341983
Summary:
Enable tests on Windows and make check-fuzzer pass on it. Make
check-fuzzer pass on Windows by fixing libFuzzer, fixing tests, and by
disabling tests on Windows. Most of these are disabled temporarily as
support for the tests and the features they test will be added
incrementally. Other tests will not be enabled since they require
things that are not on Windows (eg: afl_driver tests). Every test
that was explicitly disabled on Windows has a comment explaining why
(unless obvious like merge-posix.test).
The lit.cfg file was modified to support running tests on windows.
fuzzer-dirs.test was fixed by making the Windows implementation print
the same error message as the posix version.
merge-control-file.test was fixed by making the test binary end with
the ".exe" extension (on all platforms).
Patch By: metzman
Reviewers: morehouse
Reviewed By: morehouse
Subscribers: srhines, mgorny
Differential Revision: https://reviews.llvm.org/D51549
llvm-svn: 341385
Summary:
Port libFuzzer to windows-msvc.
This patch allows libFuzzer targets to be built and run on Windows, using -fsanitize=fuzzer and/or fsanitize=fuzzer-no-link. It allows these forms of coverage instrumentation to work on Windows as well.
It does not fix all issues, such as those with -fsanitize-coverage=stack-depth, which is not usable on Windows as of this patch.
It also does not fix any libFuzzer integration tests. Nearly all of them fail to compile, fixing them will come in a later patch, so libFuzzer tests are disabled on Windows until them.
Patch By: metzman
Reviewers: morehouse, rnk
Reviewed By: morehouse, rnk
Subscribers: #sanitizers, delcypher, morehouse, kcc, eraman
Differential Revision: https://reviews.llvm.org/D51022
llvm-svn: 341082
Summary:
This was an experimental feature. After evaluating it with:
1) https://github.com/google/fuzzer-test-suite/tree/master/engine-comparison
2) enabling on real world fuzz targets running at ClusterFuzz and OSS-Fuzz
The following conclusions were made:
1) With fuzz targets that have reached a code coverage plateau, the feature does
not improve libFuzzer's ability to discover new coverage and may actually
negatively impact it.
2) With fuzz targets that have not yet reached a code coverage plateau, the
feature might speed up new units discovery in some cases, but it is quite
rare and hard to confirm with a high level on confidence.
Revert of https://reviews.llvm.org/D48054 and https://reviews.llvm.org/D49621.
Reviewers: metzman, morehouse
Reviewed By: metzman, morehouse
Subscribers: delcypher, #sanitizers, llvm-commits, kcc
Differential Revision: https://reviews.llvm.org/D51455
llvm-svn: 340976
Summary:
Port libFuzzer to windows-msvc.
This patch allows libFuzzer targets to be built and run on Windows, using -fsanitize=fuzzer and/or fsanitize=fuzzer-no-link. It allows these forms of coverage instrumentation to work on Windows as well.
It does not fix all issues, such as those with -fsanitize-coverage=stack-depth, which is not usable on Windows as of this patch.
It also does not fix any libFuzzer integration tests. Nearly all of them fail to compile, fixing them will come in a later patch, so libFuzzer tests are disabled on Windows until them.
Reviewers: morehouse, rnk
Reviewed By: morehouse, rnk
Subscribers: #sanitizers, delcypher, morehouse, kcc, eraman
Differential Revision: https://reviews.llvm.org/D51022
llvm-svn: 340949
Summary:
Port libFuzzer to windows-msvc.
This patch allows libFuzzer targets to be built and run on Windows, using -fsanitize=fuzzer and/or fsanitize=fuzzer-no-link. It allows these forms of coverage instrumentation to work on Windows as well.
It does not fix all issues, such as those with -fsanitize-coverage=stack-depth, which is not usable on Windows as of this patch.
It also does not fix any libFuzzer integration tests. Nearly all of them fail to compile, fixing them will come in a later patch, so libFuzzer tests are disabled on Windows until them.
Patch By: metzman
Reviewers: morehouse, rnk
Reviewed By: morehouse, rnk
Subscribers: morehouse, kcc, eraman
Differential Revision: https://reviews.llvm.org/D51022
llvm-svn: 340860
Now that all Zircon calls have been transitioned to take time as signed
value, we can transition back to ZX_TIME_INFINITE, undoing the change
made in r337802.
Differential Revision: https://reviews.llvm.org/D51266
llvm-svn: 340764
Summary:
Since we're casting from double to size_t during input selection, we
really want a discrete distribution over size_t rather than a piecewise
distribution over doubles.
Reviewers: kcc
Reviewed By: kcc
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D50356
llvm-svn: 339973
Summary:
We only run the 3rd check if 2nd check finds unstable edges.
3rd UpdateUnstableCounters is now merged with ApplyUnstableCounters to only run 1 iteration.
Patch by Kyungtak Woo (@kevinwkt).
Reviewers: Dor1s, metzman, morehouse
Reviewed By: Dor1s, morehouse
Subscribers: delcypher, #sanitizers, llvm-commits, kcc
Differential Revision: https://reviews.llvm.org/D50411
llvm-svn: 339249
Summary:
There may be cases in which a user wants to know which part of their code is unstable.
We use ObservedFuncs and UnstableCounters to print at exit which of the ObservedFunctions
are unstable under the -print_unstable_stats flag.
Patch by Kyungtak Woo (@kevinwkt).
Reviewers: Dor1s, metzman, morehouse
Reviewed By: Dor1s, metzman, morehouse
Subscribers: delcypher, #sanitizers, llvm-commits, kcc
Differential Revision: https://reviews.llvm.org/D50264
llvm-svn: 339081
Summary:
Added functions that calculate stats while fuzz targets are running and give
mutations weight based on how much new coverage they provide, and choose better
performing mutations more often.
Patch by Kodé Williams (@kodewilliams).
Reviewers: Dor1s, metzman, morehouse
Reviewed By: Dor1s, morehouse
Subscribers: delcypher, kcc, llvm-commits, #sanitizers
Differential Revision: https://reviews.llvm.org/D49621
llvm-svn: 338776
Summary:
Added a new mode within flag -handle_unstable for new unstable handling algorithm that does the following:
When an edge is shown as unstable, copy to UnstableCounters the value 0.
During ApplyUnstableCounters we copy back the value 0 to ModuleInline8bitCounters if the edge was unstable.
This way we would be ignoring completely features that were collected through non-determinism.
Unstable hits would be counted as if it never hit.
Reviewers: metzman, Dor1s, kcc, morehouse
Reviewed By: metzman, morehouse
Subscribers: delcypher, llvm-commits, #sanitizers
Differential Revision: https://reviews.llvm.org/D49684
llvm-svn: 337853
This is a preparation for breaking change when all Zircon calls that
take time as an argument will start using signed valued. We will
transition back to ZX_TIME_INFITINE after all the changes to these
symbols are done and become part of the Fuchsia SDK.
Differential Revision: https://reviews.llvm.org/D49694
llvm-svn: 337802
The path to headers which are installed into libc++ build directory
has changed in r337727 which broke the libFuzzer build.
Differential Revision: https://reviews.llvm.org/D49705
llvm-svn: 337775
Summary:
Created unstable_handle flag that takes 1 or 2, depending on the handling type.
Modified RunOne to accommodate the following heuristic:
Use the first CollectFeatures to count how many features there are.
If no new features, CollectFeatures like before.
If there is new feature, we run CB 2 more times,
Check which edges are unstable per input and we store the least amount of hit counts for each edge.
Apply these hit counts back to inline8bitcounters so that CollectFeatures can work as intended.
Modified UnstableCounters to 8int_t and created a bitset UnstableSet to tell which edges are unstable.
Patch by Kyungtak Woo (@kevinwkt).
Reviewers: Dor1s, metzman, morehouse
Reviewed By: Dor1s, morehouse
Subscribers: delcypher, #sanitizers, llvm-commits, kcc
Differential Revision: https://reviews.llvm.org/D49525
llvm-svn: 337696
Matt Morehouse