Fuzz Testing#

Fuzz testing automatically generates randomized inputs to find crashes, assertion violations, and other bugs that hand-written test cases miss. Xmera uses Google FuzzTest to write fuzz tests alongside regular GTest unit tests.

Goals#

Fuzz testing in Xmera serves three purposes:

  1. Find crashes and undefined behavior – the fuzzer explores input combinations that humans rarely think to try (overflows, empty arrays, extreme values).

  2. Verify algorithmic invariants – rather than checking a single expected output, fuzz tests assert properties that must hold for all valid inputs (e.g., “pixel count is non-negative”, “merged region does not exceed sum of inputs”).

  3. Continuous regression coverage – a nightly CI job runs each fuzz target for an extended duration and persists a corpus of interesting inputs. Over time the corpus grows, giving each subsequent run a head start.

Property-Based Tests (Fuzz Tests)#

Traditional unit tests verify specific input/output pairs:

// Unit test: one specific scenario
EXPECT_EQ(algorithm.update(knownInput), expectedOutput);

Property-based fuzz tests instead assert invariants that hold across the entire input domain:

// Fuzz test: any valid input must satisfy these properties
void fuzzAlgorithm(int32_t x, int32_t y) {
    auto result = algorithm.update(x, y);
    EXPECT_GE(result.count, 0);           // never negative
    EXPECT_LE(result.count, x + y);       // bounded by inputs
}
FUZZ_TEST(MySuite, fuzzAlgorithm)
    .WithDomains(fuzztest::InRange(0, 1000),
                 fuzztest::InRange(0, 1000));

When built without fuzzing instrumentation (the default), each FUZZ_TEST runs as a normal parameterized GTest with a fixed set of seed inputs – so it participates in regular CI without any special build flags. When built with FUZZTEST_FUZZING_MODE=ON, the same test becomes a coverage-guided fuzzer that mutates inputs to maximize code coverage.

Creating a Fuzz Test#

File layout#

Place fuzz tests next to the existing unit tests for a module. The naming convention is test_<moduleName>_fuzz.cpp:

src/fswAlgorithms/<category>/<moduleName>/
    tests/
        CMakeLists.txt
        test_<moduleName>.cpp          # regular GTest
        test_<moduleName>_fuzz.cpp     # fuzz tests

Writing the test file#

A fuzz test file has three parts:

  1. Includes – the algorithm header, GTest, and FuzzTest:

    #include "../myAlgorithm.h"
#include "gtest/gtest.h"
#include <fuzztest/fuzztest.h>

  2. Test functions – each function takes fuzzed parameters and asserts invariants. The function must not return a value; use EXPECT_* / ASSERT_* macros to check properties:

    void fuzzMyAlgorithm(int32_t inputA, double inputB) {
    MyAlgorithm algo;
    auto result = algo.compute(inputA, inputB);

    // Assert invariants -- properties that must hold for ALL inputs
    EXPECT_GE(result.value, 0);
    EXPECT_NO_THROW(algo.reset());
}

  3. Registration – the FUZZ_TEST macro registers the function and specifies input domains:

    FUZZ_TEST(MyAlgorithmFuzz, fuzzMyAlgorithm)
    .WithDomains(fuzztest::InRange(0, 1000),      // inputA
                 fuzztest::InRange(-1.0, 1.0));    // inputB

    Common domain combinators:

    • fuzztest::InRange(lo, hi) – uniformly sample an integer or float range

    • fuzztest::OneOf(domain1, domain2, ...) – pick from several sub-domains

    • fuzztest::Arbitrary<T>() – any value of type T

    • fuzztest::VectorOf(domain).WithMaxSize(n) – variable-length containers

    See the FuzzTest domain reference for the full list.

CMake integration#

In the module’s tests/CMakeLists.txt, add a conditional fuzz target:

if(XMERA_ENABLE_FUZZTESTS)
  fuzztest_setup_fuzzing_flags()

  add_executable(test_myModule_fuzz
    ../myAlgorithm.cpp
    test_myModule_fuzz.cpp
  )

  target_link_libraries(test_myModule_fuzz PRIVATE
    Eigen3::Eigen
    fuzztest::fuzztest
    fuzztest::fuzztest_gtest_main
  )

  gtest_discover_tests(test_myModule_fuzz
    PROPERTIES
      LABELS "fuzz\;fuzz-smoke"
  )
endif()

Key points:

  • Guard everything with if(XMERA_ENABLE_FUZZTESTS) so the default build is unaffected.

  • Call fuzztest_setup_fuzzing_flags() before add_executable to configure the necessary compiler instrumentation.

  • Link against fuzztest::fuzztest and fuzztest::fuzztest_gtest_main (not GTest::gtest_main).

  • Apply both the fuzz and fuzz-smoke CTest labels so the target is picked up by both smoke and long-running fuzz workflows.

Writing good invariants#

The hardest part of fuzz testing is choosing what to assert. Some strategies:

  • No crash – the simplest invariant. If the function segfaults or throws an unhandled exception on any input, that is a bug.

  • Output bounds – result values stay within a known range (EXPECT_GE(result, 0)).

  • Conservation – output does not exceed a known function of the inputs (e.g., merged pixel count <= sum of input pixel counts).

  • Idempotence – calling the function twice with the same input gives the same result.

  • Round-trip – encode then decode returns the original value.

  • Reference oracle – compare a fast implementation against a known-correct slow implementation.

Running Fuzz Tests#

There are three ways to run fuzz tests, depending on what you need.

As regular unit tests (smoke test)#

Configure with the fuzz-smoke-test preset and run via CTest. Each FUZZ_TEST executes a small fixed set of seed inputs as a normal GTest – no coverage-guided fuzzing, just a quick sanity check:

cd src
cmake --preset fuzz-smoke-test
cmake --build ../build --parallel
cd ../build && ctest -C Release -L fuzz-smoke

This is what PR CI runs. It finishes in seconds.

As a coverage-guided fuzzer (local)#

Configure with the fuzz-test preset (enables FUZZTEST_FUZZING_MODE), then run a specific binary with --fuzz_for:

cd src
cmake --preset fuzz-test
cmake --build ../build --parallel
./build/path/to/test_myModule_fuzz --fuzz_for=120s

The fuzzer will mutate inputs for the specified duration, printing new coverage discoveries to the terminal. If it finds a crash, it prints the reproducer input and exits with a non-zero status.

To run all fuzz targets for a fixed duration, use the helper script:

.github/scripts/run_long_fuzzers.sh ./build ./fuzz-logs --fuzz-for 120s

With a persistent corpus#

A corpus is a set of inputs that the fuzzer has previously found interesting (i.e., they exercise distinct code paths). Persisting the corpus between runs lets the fuzzer pick up where it left off instead of starting from scratch.

To save and reuse a corpus, set the FUZZTEST_TESTSUITE_OUT_DIR and FUZZTEST_TESTSUITE_IN_DIR environment variables:

# First run -- seed the corpus
mkdir -p /tmp/fuzz-corpus
FUZZTEST_TESTSUITE_OUT_DIR=/tmp/fuzz-corpus \
FUZZTEST_TESTSUITE_IN_DIR=/tmp/fuzz-corpus \
./build/path/to/test_myModule_fuzz --fuzz_for=60s

# Subsequent runs -- the fuzzer loads previous findings and continues
FUZZTEST_TESTSUITE_OUT_DIR=/tmp/fuzz-corpus \
FUZZTEST_TESTSUITE_IN_DIR=/tmp/fuzz-corpus \
./build/path/to/test_myModule_fuzz --fuzz_for=60s

Corpus files are written incrementally during fuzzing – they survive crashes and early termination. This is in contrast to --corpus_database, which only writes the corpus after a clean exit.

The run_long_fuzzers.sh script handles corpus management automatically when given the --corpus flag:

.github/scripts/run_long_fuzzers.sh ./build ./fuzz-logs \
    --fuzz-for 120s --corpus .fuzztest_corpus

This creates per-binary subdirectories under .fuzztest_corpus/ (e.g., .fuzztest_corpus/test_regionsOfInterest_fuzz/) and sets the environment variables for each binary.

Nightly CI Fuzzing#

The nightly-long-tests.yml workflow runs every night and:

  1. Restores the corpus cache from prior runs.

  2. Configures and builds with the fuzz-test preset.

  3. Runs run_long_fuzzers.sh with a 120-second (or configurable) duration per target.

  4. Saves the updated corpus back to the GitHub Actions cache.

  5. Uploads logs and the corpus as workflow artifacts.

The corpus accumulates over time. Each nightly run starts from the previous night’s corpus, so coverage improves monotonically.

To trigger a manual nightly run with a custom duration, use the workflow dispatch from the Actions tab and set the fuzz_duration input (e.g., 10m).

Reproducing a Failure#

When the fuzzer finds a crash, it prints the failing input to the console. To reproduce it:

  1. Copy the FUZZ_TEST_* reproducer line from the log.

  2. Set the FUZZ_TEST_MINIMIZER_INPUT environment variable as shown in the log and re-run the binary. The fuzzer will minimize the input to the smallest case that still triggers the failure.

  3. Add the minimal reproducing input as a regular GTest case so the fix is permanently covered by CI.