Faster Rust Testing at Scale: cargo-nextest in Practice
In a Nutshell
Rain created cargo-nextest to solve the speed and observability problems that cargo test faces with large Rust test suites, delivering up to 3x faster execution through process-per-test isolation, built-in retries, and rich configuration. The tool's most powerful recent features are full-fidelity test recording for CI replay and first-class debugger integration that preserves environment setup. The underlying philosophy emphasizes handling every failure mode—crashes, signals, resource exhaustion—rather than just the happy path, which is why roughly 80% of the codebase addresses error cases.
These notes were generated by AI and may contain inaccuracies.
Vitaly introduces the RustRover livestream series and announces RustRover's implementation of support for cargo nextest. Rain, the author of cargo nextest, is the guest speaker. A link to the RustRover blog post about the implementation is available in the video description.
Rain is a software engineer with 14-15 years of industry experience. Originally from India, Rain has been based in the San Francisco Bay Area since 2012. Career history includes work as a contractor at Mozilla, 10 years at Facebook/Meta, and currently at Oxide Computer Company. Rain has been writing Rust professionally since 2017, representing 9.5 years of professional Rust experience.
Cargo nextest is described as a next-generation test runner for Rust. Benchmarks show it can be up to three times faster than cargo test. Features include flaky test detection, JUnit output, handling of tests that terminate abnormally including segfaults, a rich configuration language allowing per-test configuration, retries, and recording runs for playback in CI. The tool has strong CI focus while also addressing local interactive developer experience.
Rain's first computer was an old 386. At around age 6, Rain began exploring the computer and discovered a book on GW Basic in their father's bookshelf. This led to writing simple programs and falling in love with programming at a young age. Rain later pursued a CS degree and has continued programming professionally.
Rain's introduction to Rust occurred during work on Meta's source control team from 2012 to 2018. Meta uses a fork of Mercurial called Sapling rather than Git for source control. Mercurial was written in Python and C, creating performance bottlenecks due to Python's speed limitations despite algorithmic optimizations. An opportunity arose to write a Mercurial server called Mononoke in Rust. Rain joined as the Mercurial expert alongside one Rust expert on the team. Rain learned Rust to work on this project and fell in love with the language.
Rain gives a shoutout to Jujutsu, a version control system that combines the best ideas from Mercurial and Git, written in Rust. Rain recommends checking out Jujutsu for those interested in where version control should go.
Rain's first Rust project involved writing an incremental parser for the Mercurial wire protocol, handling data from the wire without collecting it in memory. This was async code before async/await, requiring handwritten state machines in 2017. The experience was initially rough, but Rain realized the problem could be approached in a testable way, making it easier through extensive testing.
Cargo nextest originated in 2021, not during the initial Rust work. After working on Mononoke for a couple of years (now in production since 2019), Rain moved to Facebook's Libra blockchain project. Around 2020-2021, the team faced numerous flaky tests and needed better detection and handling methods. Rain developed nextest as part of addressing this problem, though it grew beyond flaky test detection.
Oxide sells what they call a "cloud computer" - a cloud-like experience on purchased hardware rather than rented from providers like AWS, Azure, or GCP. The product is a data center rack-sized computer with up to 32 compute sleds, fully vertically integrated from hardware through mechanical engineering to software, UI, and APIs. Customers bring the rack to their data center or office, provide power and networking, and run workloads including VMs, Terraform, and containers. Oxide uses Rust extensively throughout the stack from embedded firmware to control plane layers. Rain works on the higher control plane layer and devtools. The target market is larger enterprises needing significant compute capacity.
If going back to 2017, Rain would rethink async semantics more deeply. While async Rust is powerful, it has foot guns not present in synchronous Rust. Rain gave a talk at RustConf last year called "Canceling Async Rust" exploring these issues. The suggestion is that async Rust's design made sense given the constraints, but those constraints should have been examined more carefully, such as whether it must work with embedded systems.
Rain identifies async drop as a feature that would fix some current issues. Async cancellation is described as a large general problem that might be addressed through marker traits or type system logic. Rust's approach to backward compatibility is praised, and Rain suggests improvements might be possible through a combination of the addition system and new features, though changes are harder now than they would have been in 2017.
The primary problem cargo nextest addresses is speed and observability of large test suites for large network services. While cargo test works well for core algorithms and data structures, it doesn't scale to large services, applications, or high-complexity projects with integration and end-to-end tests. Nextest is designed for large network services, large CLI tools, or projects with high complexity and scope.
Users should consider switching if unhappy with cargo test in specific ways. Flaky tests are better handled by nextest's retry capability, which cargo test lacks. Nextest's process-per-test model addresses issues where tests can corrupt state for other tests, which is problematic for GPU testing against DirectX, OpenGL, or Vulkan since these APIs can only initialize one instance per process. Cargo test's shared process model forces reuse of potentially corrupted graphical contexts. Speed improvements are significant for large services, with nextest often several times faster than cargo test, saving both CI compute costs and local developer time.
The coolest recent feature allows recording every test run with full fidelity. Users can fetch CI runs locally, examine what each test did, and replay CI execution locally. This has been used to find bugs. Additional features include rerunning only failed tests from previous runs, and generating perf traces in the Chrome trace event format. Users can upload traces to analyze test scheduling, identify blocking tests, find slowest tests, and get graphical views of execution.
Nextest has first-class debugger support that addresses cargo test's limitation where running test binaries directly loses the environment setup including environment variables. Using --debugger with a specified debugger name provides full hooks for IDE integration including rust-analyzer. This configures the complete environment matching normal test execution, enabling accurate debugging of tests that depend on specific environment configurations.
Many recent features have been added based on requests from coworkers at Oxide and the open-source community. Oxide uses nextest extensively, though it was not originally developed there.
One requested feature is the ability to dump standard output and standard error for tests that are currently running. On any OS, pressing the T key (T for tango) produces this output. On macOS, control T can also be used, which corresponds to SIGINFO. This provides a live view showing how long each test has been running along with its standard output and standard error. This capability was added after a coworker needed it to debug a complex test that could not be reproduced when run individually but would appear when executed as part of a larger test suite.
Feature development balances coworker requests with open source community input. Many contributors have added features to cargo-nextest, and the project maintains a healthy open source ecosystem.
Cargo-nextest ranks among the most popular cargo tools on crates.io. It is used throughout the industry, including at large technology companies, though specific organizations cannot always be named publicly. Searching GitHub repositories of major tech companies for "nextest" reveals widespread adoption. AWS is a confirmed heavy user, with multiple feature requests originating from their developers.
The hardest part of building cargo-nextest was implementing the core test state machine logic. While the project contains parsers, reporting, and other components, the most complex element is the fundamental logic for running a test and observing every possible outcome—what can go right or wrong. This state machine was expressed using async Rust and contains dozens of states.
Although async Rust is commonly associated with high-concurrency scenarios like serving millions of requests per second, cargo-nextest typically runs only as many tests as available CPU cores or threads—usually at most 32 concurrent tests on consumer hardware. The state machine's complexity has grown as new features were added. Async Rust proved powerful for expressing this complexity in a manageable way, though it introduced its own challenges.
Rust allows developers to aim higher than garbage-collected languages or even C and C++. The creator would have difficulty building cargo-nextest in any other language. Rust enables solving problems that are significantly harder or impossible elsewhere.
Cargo-nextest provides multiple hooks for extensibility. It offers machine-readable formats used by tools like Rust's test infrastructure. Debugger support is available. Setup scripts allow pre-test actions such as initializing databases or seeding data before test execution. Code coverage is supported, and Miri integration enables testing unsafe Rust code for correctness.
Third-party tools have built integrations with cargo-nextest. Extension points are added as needed to facilitate further integration.
Cargo-nextest does not currently support remote test execution, where coordination occurs on one machine and actual execution happens across a fleet of optimized servers. The creator hopes others will explore remote execution, containerized execution, and similar ideas—either as alternatives or as extensions. Extension points can be added if contributors wish to work on these features.
Developers have approached the maintainer with specific feature requests that were implemented rapidly. One such request from a JetBrains developer (Vlad) was implemented almost immediately after being requested.
For embedded Rust testing, wrapper scripts allow wrapping test execution in shell scripts that can send commands to embedded hardware and set up the required environment. When only one or a few pieces of embedded hardware are available, tests must run serially. Cargo-nextest includes built-in rate limiting that enforces running specific test subsets (such as embedded tests) one at a time while allowing other tests (such as in-memory unit tests) to run in parallel.
The smart scheduler handles mixed test suites automatically. Cargo-nextest itself is too complex to run on embedded hardware, but it supports dispatching tests to embedded targets via wrapper scripts and target runners.
Developer tools, regardless of scale, must prioritize correctness above other concerns. While large-scale services focus primarily on handling enormous user volumes, developer tools face different complexity: they must handle every case where something goes wrong. A good test runner is defined as a tool that handles all failure modes, not just the happy path. Approximately 80% of cargo-nextest's code addresses the various ways tests can misbehave—running out of memory, receiving signals, crashing, segfaulting, and other failure modes.
Production compilers follow a similar pattern, where most code handles error cases. This focus on unhappy paths distinguishes good developer tools from great ones, even though it multiplies engineering effort.
Developer tools should pay particular attention to error messages. When things go wrong, developers are having a bad day, and tools should help rather than obstruct. Error messages should be helpful rather than simply dumping large stack traces. Tools should handle signals like control-C gracefully and produce clean shutdowns.
Many Rust developer tools are well-written, but more focus is needed on testing failure scenarios, using fuzzers and property-based tests, and ensuring the tool remains robust when users are under pressure—such as during incidents when systems are down.
AI tools can be extremely powerful for experienced software engineers when used correctly. They excel at addressing technical debt and performing large refactors. However, concern exists about common usage patterns where developers generate 5,000-line pull requests without reviewing them, creating review burdens for coworkers.
The philosophy for using AI tools prioritizes robustness and rigor over maximum speed. The goal is creating code that coworkers can understand and that is fully comprehensible top to bottom, rather than maximizing feature delivery velocity.
One effective use is adversarial usage: writing unsafe Rust code by hand, then asking the AI to attempt breaking it by finding holes, bugs, or writing tests that demonstrate failures. This treats the AI as a point of tension rather than merely an assistant.
The recommendation is to use AI tools thoughtfully and mindfully, serving coworkers and users, rather than focusing solely on delivery speed or "vibe coding."
The speaker's philosophy when building developer tools is that if the human uses the tool wrong, it is the tool's fault. This approach has been applied to Next.js with a focus on good error messages to teach users how to use tools correctly. The principle is to not blame the human when they make mistakes, but rather improve the tool so that users get it less wrong.
This principle becomes more challenging with AI tools because they are open-ended chat boxes where users can request any action. The speaker used Cloud Code as an example, specifically the slash insights feature that scans usage history and suggests ways to automate tasks. However, the insights simply state "you should automate" without considering whether automation is actually beneficial or the downsides of shipping code without understanding what it does.
The speaker deliberately chooses not to automate certain things because they want to understand what they're doing thoroughly. The core principle is that tool makers should serve customers first rather than focusing on automating as much as possible.
cargo-hack is managed by Taiki Endo, a software engineer from Japan. This tool handles crates with many features by running tests against every possible combination of features. It intelligently minimizes combinations based on feature dependencies—for example, if feature A depends on feature B, it won't test unnecessary combinations.
cargo-expand is used when developing proc macros. It produces the output of a proc macro on screen, allowing developers to see what output their macros generate during development. This functionality may also be available in rust-analyzer.
cargo-semver-checks addresses the challenge of maintaining semantic versioning compatibility when releasing new versions of Rust crates. The tool detects both simple cases like adding parameters to functions and complex cases such as accidentally removing thread safety from types that previously had Send and Sync bounds. The speaker uses this tool whenever releasing a new library version to determine whether to bump the major version.
The speaker concluded by thanking the developers of these tools and the host for the interview about cargo-nextest.
Keep JetBrains in your library
Save the videos and channels worth coming back to, and find them again in one place.
