The Evolution of Async Rust: From Tokio to High-Level Applications
In a Nutshell
Carl Lodge explains that Tokio became Rust's default async runtime due to being first and building the ecosystem around it, while most applications should use its work-stealing scheduler rather than thread-per-core approaches for practical performance. The conversation highlights Rust's expansion beyond systems programming into higher-level web applications, with ongoing work on Toasty as an ergonomic ORM/query engine and future frameworks to address the current gap in industrial-grade application frameworks. Key technical insights include io_uring's limited real-world benefits for sockets, the cooperative scheduling nature of async Rust requiring careful yielding, and the importance of defensive cancellation handling.
These notes were generated by AI and may contain inaccuracies.
Vitaly Bragilevsky from JetBrains Rust team hosts a livestream joined by Carl Lodge from Tokyo. Tokyo is described as the prominent project in the Rust ecosystem. Tokyo is hosting its first conference in Portland at the end of April, with the website tokio.com. The conference marks 10 years since Tokyo was announced in July. Tokyo and Rust have become one of the default ways companies and people build infrastructure-level software for networking services. If building a database or proxy from scratch, Rust is the likely choice due to its maturity and correctness.
Portland was chosen because Carl Lodge lives there, making organization easier with a local partner. The conference is called Tokyo Conf but is really async network programming with Rust. The program committee helped select talks from the community building amazing things.
Async is a general programming construct pattern that has been around for a long time. While many associate async with performance and handling large numbers of concurrent connections, async also provides added flexibility in code structure. Writing timeouts with threads is surprisingly hard, requiring setting timeouts on reads and handling interrupts, which is quite painful. With async code, timeouts can be thrown everywhere easily. Cancellation and managing multiple in-process tasks efficiently is both performance-wise and architecturally a big reason to use async programming.
The early days of async await and the future trait were driven by cancellation and the ability to do timeouts easily, leveraging Rust's properties. With async cancellation in Rust, a future or async block can be dropped, leveraging Rust's drop handler for cleanup, making it nice for managing and canceling work in progress.
Async is both about performance and a way of managing lots of in-flight threads of logic well. It is also about code quality. For managing code with lots of inbound events that can happen anytime, async is a good approach. UI-driven development is all async and event-driven.
Carl started using Rust 12 or 13 years ago, before the 1.0 days. He was working at a company building a performance monitoring tool for Ruby. The Ruby gem agent was taking too much CPU cycle, requiring a switch to native code. While C was the obvious choice, Carl had limited C and C++ experience. Yehuda Katz suggested trying Rust, which Mozilla was building as an experimental safer language for browsers. Yehuda argued it was better to use Rust even though immature than trust C code in production.
After getting into Rust, Carl wanted to build a side project experimenting with a database using non-blocking sockets instead of blocking ones. He started by binding epoll, creating the mio library for zero-cost binding to epoll. This became the foundation for writing non-blocking IO code with Rust. However, it was very verbose and boilerplate heavy, requiring manual state machine writing, which was error-prone.
The future trait was the first abstraction, driven by Rust team members Aaron Turon and Alex Crichton. It encapsulated individual component state machines but still required manual state machine writing. Async await came next, designed by the language team. Rust's async await achieves safe async programming with zero-cost abstractions, which was not obvious to achieve.
Java's virtual threads are similar to the general concept of coroutines and lightweight threads like Go has. Rust actually started with lightweight virtual threads and coroutines well before 1.0. Rust was announced around the same time as Go, with Graydon building an OCaml-flavored version. There were green threads, everything Go had, with an OCaml-like syntax and garbage collector.
Go took off while Rust didn't initially, partly because Google backed Go. When Carl approached Rust, they had added the borrow checker, were removing the garbage collector, and still had lightweight green threads. The consensus was to remove them because Rust was heading toward zero-cost performance equivalent to C or C++. Managing stacks for green threads added overhead even to non-green thread usage. The decision was to remove them completely. There was no plan for async await at that point.
When canceling a future with an HTTP request in flight, the behavior depends on implementation. If the connection is held in the task and the future owns the connection, dropping it immediately terminates the connection. The future is a struct holding the Tokyo socket handle, and dropping recursively calls drop handlers, closing the socket.
With Hyper, requests spawn a separate Tokyo task owning the connection, with message passing between the future and socket. Dropping the future drops only that future, not the other task. The channel sends a signal for cleanup. Using different tasks means no immediate drop, but mechanisms based on Rust's drop handler cascade cleanup.
The pattern seems clean and Rustish - dropping values automatically cleans up. However, async await syntax does not expose that code can be dropped at any time. When a caller wraps async code in a timeout, the future drops without warning. Developers need to be defensive since code can stop running anytime, but this causes bugs and handling is not obvious.
Tokyo became the de facto default async runtime because it was first. Hyper and the ecosystem built around Tokyo, and there wasn't a good reason not to use it. Other async crates exist but are more specialized - Embassy for embedded contexts without requiring an allocator, and io_uring-first runtimes. For most use cases, Tokyo provides the ecosystem needed.
io_uring is interesting but presents different trade-offs depending on the use case. For file system APIs, it is a really good library to use. However, it is a lot more complicated to use than epoll, which is very simple. While io_uring does support networking like sockets, there is less of a reason to use it for that purpose.
Benchmarks exist that show performance wins with io_uring for sockets, but in the real world there have been very limited real world performance benefits. The main performance gains come when there is a use case where batching up syscalls can provide a performance win, and there aren't a ton of use cases for that. Those are really going to be pretty specialized low-level cases.
To get the benefit from io_uring, the IOU ring specific API must be used, which means giving up on the async await Tokio socket patterns. There haven't been real performance benefits seen with swapping out IOU ring for sockets under the hood in Tokio.
What is interesting about Tokio and the async await pattern is the ability to mix and match IOU ring specific APIs with Tokio, which is demonstrated with the Tokio Uring crate. A third-party IOU ring library can be built that takes full advantage of the design with socket APIs designed for using with IOU ring, and these can be mixed with Tokio on the same thread.
For file system APIs, it would make sense to use io_uring, but historically io_uring has had way more CVEs and security issues than the main mature Linux APIs, which puts a dampener on wanting to switch to them as the main API. For the file system APIs that Tokio provides out of the box that look like the standard library file system APIs, there is probably less of a reason to use io_uring because file system access is not the main use case.
If building a database with a write-ahead log or pulling from a B-tree, APIs that touch the syscalls directly will likely be used, and in that case a third-party crate that mixes with the Tokio runtime can be used.
There is an ongoing effort by a contributor to add io_uring for the file system APIs, which will most likely happen this year. The short version is that if io_uring is needed and there is a clear use case where benefits can be gained from using it, it can be mixed with Tokio. But it is less clear for everyone that it is the right choice as compared to epoll, because epoll is really good, solid, mature, and quite optimized.
Epoll has been bulletproof for 30 years. The io_uring API is really complicated compared to epoll, though there are use cases for using it and it can be done with Tokio by looking at the Tokio-uring crate.
For the Tokio runtime itself, as little inventing as possible was done. The approach was to do what is known to work already, drawing from a wealth of prior art in asynchronous runtime design patterns. Go was studied heavily as the runtime is quite good, and Erlang's green threaded runtime with its years of maturity and investment was also examined.
The pattern of a work stealing scheduler where tasks are scheduled on the current thread and other threads manage self-balancing is a well-established pattern that works for most people. One design principle for Tokio is to have something that is easy to use for most people and good enough for most people, but have escape hatches or ways to do more specific things.
For managing networking systems and big modern servers with lots of cores, concurrency and parallelization must be managed while keeping as much locality as possible by keeping memory on the thread and doing as little synchronization as possible. One way to do that is to have one thread per core with absolutely no sharing, no work stealing, nothing, where every single thread is almost like an entire instant server. This assumes talking between threads is really expensive and everything is isolated.
This works and is the design path for building infrastructure level software where the last juice of performance needs to be extracted and there is willingness to put in the engineering work. ScyllaDB is an example of this pattern where existing big infrastructure is rewritten to be fast.
The downside is being on the hook to make sure every single thread has balanced work, and if it gets out of balance there will be threads that are underutilized and some that are overutilized, which turns to latency.
As a default for most people where they don't need to over-prioritize performance optimization, and realistically the vast majority of apps even though using Rust and caring about performance do not need that level of fine-tuning, just writing code and letting the Tokio multi-threaded work stealing runtime manage balancing work between threads is good enough. There is a bit of a synchronization cost associated with it but in most cases it doesn't matter.
As Tokio and Rust get more mainstream adoption, the runtime has even tried to be too smart, and there is consideration for making the runtime more defensive against engineering mistakes because most of the time when people build apps with Tokio they're not actually worrying about the fine performance details of their app.
Async Rust involves cooperative scheduling where the thread runs the task until it yields back to the scheduler. With async await in Rust, the only time yielding happens is when calling dot await, which is the only point at which the Tokio runtime can repurpose that thread to run a different task.
This is usually good enough, but if a CPU heavy routine is accidentally called, there is no yielding and that thread stalls, which can result in added latency. The Tokio runtime tries to mitigate this by moving tasks from that thread to other threads, but there is only so much that can be done at the user level in a runtime.
When using any async runtime, there is responsibility for writing well-behaved apps that yield at regular intervals.
The hope is to reach a point where no one needs to touch poll and pin, and they are really only used when writing primitives that interact with Tokio. These concepts are very complicated, but the design of the language makes them elegant given the constraints.
Most developers using Tokio nowadays hopefully don't have to touch poll and pin. They are really only needed when implementing something at the edge of the async Rust ecosystem, like when integrating other things or writing primitives.
The best way to debug async Rust really depends on the use case and preferences. Print line is used a lot, as are debuggers. The biggest challenge comes when there is a stuck task.
Tokio console was an attempt to address this, along with async backtrace crates which try to get access to show where tasks are stuck. These are still trying to get stabilized with a few challenges. The go-to tools are async backtrace debuggers and tracing logs or print line, and LLMs like Claude are actually pretty good at tracing why a test is failing and explaining why.
There is tooling that would simplify debugging async Rust applications. The biggest pitfall stems from developers accidentally canceling something and not handling the cancellation appropriately. The best way would be lints and warnings that can catch those cancellations and bubble them up.
Better clippy lints that can do full stack analysis would be really good. If there were better static analysis Rust libraries that could give the full Rust type system static analysis like trait resolution as a queryable interface, it would be much easier to build tools that catch patterns or highlight danger zones.
The hypothesis is that combining these static analysis tools with an LLM, along with a failing test, could solve 90% of issues with debugging common Rust issues.
High tail latencies in production applications where P99 is going up is a harder problem because there might be one main obvious thing or it might be a combination of paper cuts or some secondary thing causing it. This is a hard problem in any language.
The approach is to understand how the runtime works including the multi scheduler, yielding, and cooperative scheduling, then pair that with Tokio runtime metrics and good instrumentation in the application.
One common reason for high tail latencies is not being a good cooperative scheduling citizen, where chunks of logic between dot await calls are too big. To debug this, Tokio runtime has a poll time histogram with instrumentation for poll times, which is the time spent between dot await calls.
If poll times start seeing more than the one millisecond range, that is suspicious and should be instrumented. Once that is identified as the cause, other tools can be used to bisect and figure out what it is.
Tokio console was an attempt to have a tool that lets you see all the tasks in the system where you can say this task owns this socket and show what it is waiting on before running. The first implementation built on top of tracing had some issues, and it would be nice to revisit with the newer async backtrace infrastructure. The role of Tokio console was an interface to get access to async backtraces and debug stuck tasks.
Historically, Rust came to be as a systems programming language as an alternative for C or C++ to build browsers, and more generally to build anything historically used C for like games or infrastructure level software. Rust has fulfilled its promise for infrastructure level software, where for any greenfield development it is by far the best choice because it is much easier to build software with Rust and Tokio that can meet stringent performance requirements.
For higher level applications like a back end for a mobile or web app where performance is not the main concern, Rust may not be considered because there are already well-established ecosystems to build those kinds of apps. Throughout Rust development experience, the belief was held that Rust is an infrastructure language that is really good for that, but the cost makes it not the best fit for higher level languages where productivity is more important.
The view is evolving that everything is not black and white. Productivity is complicated, and Rust has gotten a foothold in organizations because of its clear use case. When Rust is injected into a business for a clear use case, developers get onboarded, learn Rust, build internal tooling and libraries, and then want to build web apps for infrastructure level tools with Rust because all those libraries exist.
In theory, building the same kind of thing as a Node app or TypeScript would be more productive, but the existing libraries make building with Rust desirable. There is going to be an expansion of Rust in higher level use cases, and this is being seen today.
Users are trying to do this but hit the ceiling where libraries at this higher level just don't exist. Rust has a really established and mature ecosystem at lower level, with libraries like Tokio, font rendering libraries, and GPU libraries. The breadth of use cases that can be done with Rust well is staggering, but once reaching higher level there is less because no one has tried to build it.
Rust could be a really nice language to build higher level web apps even for greenfield because while initial prototyping might be
Rust offers strong properties for long-lived applications despite not being optimal for initial prototyping. The type system provides excellent safety guarantees, and performance benefits are appealing even when raw speed isn't critical. However, productivity at higher abstraction levels remains a challenge.
A significant portion of Rust's productivity limitations stems from library design rather than inherent language constraints. Many libraries prioritize zero-cost abstractions at the expense of ease of use, with overuse of traits and complex lifetime management creating barriers. The speaker reflects on their own past work, noting they would not design Tower the same way today due to excessive trait complexity.
"I've written libraries that I would say like going back and looking at them cough cough tower. Anyway, sorry. I would not do it again that way. Like the way it's designed like the trait insanity is high."
Rust's trait system enables preventing runtime errors through compile-time state modeling, which appeals to developers coming from dynamic languages. This approach ensures errors never occur in production because the compiler prevents invalid states. However, finding the right balance between safety guarantees and developer productivity remains essential.
Toasty represents an effort to build productive, ergonomic Rust libraries focused on ease of use rather than maximum performance. The project lives on the Tokio GitHub repository to facilitate contributor access and team management. Currently marked as incubating, a beta announcement is planned for TokioConf.
The speaker welcomes feedback from early users attempting real applications with Toasty. After completing Toasty, the next focus will be exploring how to simplify building traditional web applications in Rust.
The goal is not to replace TypeScript for most web applications, but to serve organizations already invested in Rust infrastructure. High-level web applications built in Rust become more practical when teams can leverage existing language expertise across the stack.
JetBrains observes a significant gap in Rust's ecosystem compared to Java/Kotlin's Spring Boot ecosystem. While Axum receives attention within the Tokio ecosystem, there's a clear need for high-quality, industrial-level application frameworks. The speaker confirms this is their goal and they're actively working toward this outcome.
The speaker's background includes significant experience with Ruby on Rails, including time on the core team. A pivotal career moment came from witnessing the original Rails screencast demonstrating building a blog in 15 minutes. This demonstrated how Rails transformed web development from verbose, boilerplate-heavy approaches (PHP, ASP.NET, early Java frameworks) to ergonomic, streamlined development.
The aim is to recapture the initial magic of Rails-era web development—opinionated, streamlined building that follows best practices for rapid development. Modern frameworks have grown more complex, but there's potential to bring back simplicity for the 90% use case while providing escape hatches for custom requirements.
A common ORM complaint is that users must abandon the ORM entirely when moving beyond the happy path. Toasty is designed as an application-level query engine rather than a traditional ORM, enabling mixing SQL queries with Toasty-level application queries. This approach provides the needed flexibility without requiring complete abandonment of the higher-level abstraction.
Toasty's internal structure functions as a query engine: users define their application data model, build application-level queries, and the engine transforms these to database-level operations. This architecture enables the flexible escape hatch system while maintaining the productivity benefits of higher-level abstractions.
High-level web frameworks fundamentally depend on data modeling capabilities. Starting with Toasty addresses the core requirement—how little code is needed to move from data model to UI. Community involvement through trying Toasty, providing feedback, and attending TokioConf discussions is encouraged.
A recent example shared involves a service rewritten from Spring Boot to Rust, achieving 40 seconds of startup time savings. This illustrates the performance advantages that make Rust attractive for production systems beyond just web frameworks.
Keep JetBrains in your library
Save the videos and channels worth coming back to, and find them again in one place.
