Our test methodology is all about stable, verifiable, and reproducible results that can be easily communicated.

Our core test executor is an awesome piece of open source technology called Phoronix Test Suite. Phoronix ships with a number of "pre-made" tests that can be found on the Open Benchmarking website. Given the needs of our CIDR infrastructure, we often build our own Phoronix compatible tests. While the Open Benchmarking test suite is comprehensive, we tune our test suite to measure or stress very specific elements of a workload - this is not often possible with the "off-the-shelf" versions of a benchmark.

By using an open source and standardized test executor we're enabled to share our workload library much more efficiently. Phoronix runs basically everywhere and is packed full of features, so our path to reproducing a test run is as simple as installing our Phoronix compatible test suite (known as Phoronix Test Extensions) locally on a system which already has Phoronix installed.

In addition to a flexible test library, we focus on system configuration details. For each and every test run we capture a snapshot of more than 15,000 data points about the system under test. This data set includes:

1. Installed Software Libraries (including versions or git hashes) 2. Hardware Specifics (such as memory configuration, drive models, etc.) 3. Kernel Configuration Details (page size, scaling governors, etc.) 4. Board Settings and UEFI Configuration Values (such as turbo mode, nodes per socket, etc.) 5. And many more...

By capturing a complete snapshot of the state of the system plus an easily identifiable and uniquely versioned test, we ensure future reproducibility and comparability.