Building AI applications in Rust
In a Nutshell
Rust is the best language for AI development because its compiler, borrow checker, and static analysis provide clear guidance that benefits both humans and AI-generated code. Rig is a Rust library that abstracts LLM provider differences, enabling unified AI application development with support for agents, tools, RAG, and local models via Candle integration. AI coding tools have transformed development workflows, allowing developers to manage multiple projects simultaneously while maintaining code quality through rigorous review processes, though the Rust community faces ongoing challenges with AI-generated contributions and open source maintenance.
These notes were generated by AI and may contain inaccuracies.
Orhun, Rust developer advocate at JetBrains and lead maintainer of Ratatui, introduces a four-part series on Rust and AI collaboration with the Rust Foundation. The series explores multiple angles including using Rust to build AI-powered applications and using AI to build software in Rust. Multiple voices from the Rust community will examine both opportunities and challenges.
Stefan Cornfki, known online as Gold, Silver, Copper, joins as the lead maintainer of Rig, a Rust library for interacting with LLMs that supports major providers, AI agents, and various database integrations. He has been an open source developer for approximately 10 years.
Stefan initially heard about Rust at version 1.0 in 2015 but found it too new to adopt. He began programming in Rust full-time around 2023, motivated by his desire to consolidate C, Python, and JavaScript into a single language. He notes that Rust has evolved significantly since its early days, with the borrow checker becoming much more powerful and the compiler handling lifetime annotations automatically.
Stefan's background is in computational linguistics and natural language processing. He has been familiar with text generation techniques for over a decade, with early interests in text games like Zork from the 1970s-1980s written in Lisp. He began using machine learning around 2021 with the Spacy library for syntax tree generation, which impressed him with its accuracy compared to handrolled parsers. He was familiar with LLMs as early as 2019, including GPT-2 models and AI Dungeon, an early LLM-powered text game.
In the early days with GPT-2, AI was considered more of a toy than a serious tool. The ecosystem relied on a few libraries like Kold AI for inference, and developers often used Kaggle and Google Colab for better hardware access. Coding models existed but were largely unusable. AI-assisted programming has improved significantly in the last two years.
Stefan's video setup uses Bevy with Ratatui rendering to a texture on a TV model, with a machine learning model converting webcam input into 3D bones for tracking. His background with Ratatui stems from wanting to export TUI applications to the web and use them in GUI contexts, which was difficult in Python but possible in Rust due to Ratatui's abstractions allowing custom rendering backends.
Stefan argues Rust is the best language for AI to write because its helpful compiler features benefit both humans and AI. The Rust compiler provides detailed error messages and automatic guidance on fixes, unlike C or C++ compilers. Static analysis tools are built into the compiler rather than existing as external tools, and features like the borrow checker, borrow semantics, mutation in the type system, and lifetime annotations feed directly into these checks.
The borrow checker constraints make the ecosystem more cohesive with a single build system and consistent guarantees across all libraries. While some argue the borrow checker prevents otherwise legal code, these situations are rare, and the upcoming Polonius borrow checker will address remaining edge cases. The unified ecosystem eliminates the need for multiple build systems across projects.
Stefan's favorite aspects of Rust include its performance and cross-platform compatibility, particularly the ability to compile to WebAssembly. He appreciates having a powerful, stable language with good type system features without needing the complexity of languages like Haskell.
Rig provides a unified interface across LLM providers including Gemini, OpenAI, and Anthropic. Each provider has its own API with varying degrees of OpenAI compatibility, and Rig abstracts these differences so developers can write the same code and swap providers seamlessly. Without Rig, developers would need to implement their own abstraction layer for each provider.
Stefan knew about Rig for a couple of years while researching the Rust AI ecosystem. Josh, the previous lead maintainer, was transitioning to another project and looking for someone to continue the work. Stefan reached out, interviewed with Josh and the ZeroX Playgrounds team, and became the maintainer after using Rig for several projects.
A demo application called Rat Code was built as a Codex-style terminal coding agent using Rig and Ratatui. The application features a prompt input and displays responses from OpenAI's LLM service. It was inspired by an article from Josh about rewriting cloud code.
Async is generally needed when using Rig due to network communications, though Rig attempts to hide as much async complexity as possible from users.
Completions is one of the APIs most LLM providers offer for text completion. An agent represents a level of abstraction above the raw LLM, which is simply text-in-text-out. Agents can be given tools that allow them to create files, read files, use the shell, or perform other actions. The preamble serves as a system prompt that guides the AI's behavior and personality at the beginning of each request.
Tools are Rust functions that AI agents can execute, implemented via a trait with name, arguments, input/output schema, and error type. Tools can represent simple calculators, RAG (retrieval augmented generation) database queries that return semantically appropriate documents, or even other agents that can be called as tools.
An agent itself implements the tool trait, allowing agents to function as tools for other agents. This creates recursive capabilities where one agent can delegate work to another agent through the same tool interface.
Tool parameters represent inputs similar to function parameters. Rig uses JSON schema to define these inputs because AI systems communicate through text, and JSON provides a structured text format. Different applications require different parameter types:
- Calculator applications use numeric inputs
- RAG applications may use vector inputs for semantic search
- Agent-to-agent communication uses string prompts as inputs
The call function accepts a context parameter implemented as a type map (hashmap keyed by type). This context system enables tools to maintain state across calls, supporting use cases like:
- Counting tool invocations
- Caching previous answers
- Storing state-dependent information
A tool implementation requires three core elements: parameters, call function, and name. The example demonstrates three fundamental tools for a coding agent:
- Read file tool that performs OS-level file operations
- Write file tool for file modifications
- Run shell tool using Tokio for bash execution
The example demonstrates that a functional coding agent requires only three tools: read file, write file, and bash execution. This approach mirrors minimalistic implementations like Pi, which uses the same three-tool architecture for capable coding agents.
Rig provides a hook system that functions as middleware, allowing observation and modification of inputs and outputs at every stage of model execution. This system supports advanced features including:
- Context injection at various execution stages
- Tool call recovery mechanisms to handle cases where models don't execute correctly on first attempt
The stream prompt functionality uses Rig's streaming API, which sends responses in chunks rather than complete blocks. This approach enables responsive AI agents by displaying output progressively as it's generated, rather than waiting for complete transmission.
Max turns limits the number of iterations an AI can perform on a task, preventing infinite loops that older LLMs were prone to experience. The system enforces a maximum of 20 turns by default.
Rig extensively uses Rust's advanced language features including compile-time macros, traits, and type system capabilities. The tool trait provides the foundation for extensibility, allowing any existing function to become a tool through proper trait implementation. Type erasure techniques help abstract complexity while maintaining good developer experience.
Rig maintains a low-level library design that provides maximum user control while avoiding unnecessary async code complexity and type leakage. Compile-time macros create a DSL for provider registration, enabling new provider additions in approximately 10 lines of code.
Each agent operates asynchronously and connects to a provider during execution. Multiple agents can run concurrently, communicating through Tokio channels or by using agents as tools for other agents. This architecture supports complex multi-agent workflows.
Rig supports multiple vector databases including SQLite, PostgreSQL, SurrealDB, and Neo4j. RAG (Retrieval Augmented Generation) works by converting text into vector embeddings, enabling semantic similarity searches. When querying, both questions and documents become vectors, allowing the system to find semantically relevant content to enhance LLM context.
Vector embeddings originated from search engine technology, where websites are indexed as vectors and search queries are converted to vectors for matching. Google effectively uses RAG principles. LLMs benefit from RAG when functioning as search engines, retrieving top results to provide more accurate responses.
Rig uses separate traits for different provider endpoints including messages and completions endpoints. The naming convention "completions" reflects legacy terminology from early provider integrations and remains unchanged due to lack of compelling reasons to modify established patterns.
Rig employs a cassette testing system with 1,700 HTTP traffic recordings stored as YAML files. The system records live provider interactions, then uses a mock server to replay communications during CI testing. Every pull request undergoes testing against all 1,700 recordings to ensure compatibility. The Rust implementation enables rapid test execution, completing thousands of requests in seconds.
Rig Candle integrates with Hugging Face's Candle library for local model inference, allowing direct embedding of model weights into Rust binaries. This approach eliminates server requirements and supports WebAssembly deployment, enabling local AI applications without external dependencies.
Rig is used in production environments including St. Jude's Children's Hospital medical chatbot. Multiple companies use Rig internally, with active community contributions from commercial applications.
Rig enables AI application development in Rust, distinct from standard Rust programming. Key differentiators include direct model weight embedding without server requirements, WebAssembly compatibility, and seamless integration with the Rust ecosystem including libraries like Bevy, Ratatui, and Tokio.
Rig supports local models through multiple pathways: Ollama, Llama.cpp, and Candle for weight embedding, providing flexible deployment options.
The primary Rust limitation is compilation time for larger projects, though this is offset by the compiler's static analysis benefits. Ecosystem gaps occasionally require C++ or Python libraries for niche functionality. The type system, while powerful, cannot match Haskell's capabilities for advanced type-level programming due to borrow checker constraints.
The speaker has not written code by hand since the previous Christmas, instead using AI coding tools exclusively. This approach has increased productivity significantly, enabling simultaneous work on multiple projects through parallel AI agent usage. The transition began with small, reviewable changes and evolved to comprehensive AI-driven development.
While AI-generated code can create quality concerns, it has raised standards in open source communities by making basic implementations commonplace. Recognition now requires unique and creative contributions rather than functional but unremarkable applications. AI can produce advanced code and fill skill gaps, such as Rust macro development.
AI can turn code into Rust macros, allowing developers to read generated code and accelerate learning. When used carefully, AI accelerates development and increases code quality. The quality of the final product depends on the person using the tool rather than the tool itself.
When working with multiple agents and "vibe coding," context switching becomes a challenge. The recommended approach is to use a terminal multiplexer like Zelage with multiple tabs open. Multiple Claude instances and coding agents run simultaneously across different tabs.
A typical workflow involves switching between tabs to run programs and test changes. Notes are maintained in Helix text editor to help remember context across projects. The setup remains simple with just a terminal multiplexer, multiple tabs, coding agents, and testing workflows.
All generated code is reviewed before merging, especially for Rig as a library that multiple people depend on. The approach differs based on project seriousness: for serious projects like Rig, all AI-generated code is read thoroughly; for less serious projects like games, code review is skipped in favor of playtesting and providing feedback to the AI.
Programming enjoyment has increased rather than decreased. The satisfaction has shifted from "writing code" to "creating cool projects." This change allows working on more projects simultaneously, including projects that were previously avoided due to required knowledge or time constraints. Friends who don't program much are impressed by what they can accomplish with terminal coding agents.
Good engineering doesn't require fully understanding all code, though understanding code makes someone a better engineer. When vibe coding in Rust successfully, familiarity with the Rust ecosystem remains essential - knowing which libraries to use and what constitutes idiomatic Rust code. The role shifts to that of a senior-level manager directing AI as a junior developer.
Deep technical knowledge becomes more important, not less. AI raises the skill ceiling of programming by enabling work on larger-scale projects and learning advanced techniques. The programmer must still prompt the AI and provide direction, making domain familiarity crucial for better results.
Several factors contribute to opposition against AI in the Rust community. Early AI hype led to inexperienced developers creating vulnerable applications. Security-critical industries like automotive and aerospace require engineers to fully understand their work. Some developers have poor experiences with limited tools like GitHub Copilot on GPT-4. Enterprise restrictions limit AI model choices. Societal concerns include job displacement fears and environmental impact. Ideological reasons include copyright concerns about training data.
The recommendation is to try Claude Code with tools like Aider or Claude Code with Cursor in the terminal. Developers should experiment with creative applications like game development or academic-level research using terminal coding agents.
Token limits represent a current bottleneck, with multiple maximum subscriptions being fully utilized. Local models are considered important for reducing dependency on paid services. A $200 subscription provides significant value for developers compared to multiple entertainment subscriptions.
Output quality matters more than token efficiency. Progress is expected in multi-agent systems, advanced retrieval mechanisms, memory systems, and context management. Coding harnesses represent the area where significant evolution will occur, as the underlying models are already highly capable.
An experimental runtime using Bevy is being developed internally at Rig. The goal is to use Bevy's entity component system to create coding agents as Bevy entities, leveraging the Bevy ecosystem for visualization. This approach would create a coding agent with an actual game engine, providing an ideal simulation sandbox for AI.
Major AI companies including Anthropic, OpenAI, and xAI are moving internally to Rust. The previous conception that Python or JavaScript were optimal for AI due to training data is changing. AI now writes Rust better than Python or JavaScript, and companies are shifting to Rust for its superior static analysis tools as more codebases become AI-generated.
Bevy changed from a strict no-AI contribution policy to allowing AI contributions. The previous policy created paranoia among maintainers who couldn't distinguish between human and AI-written code, leading to accusations. The new policy requires contributors to fully understand their code while discouraging pure AI submissions for this security-critical library.
Open source faces new challenges with AI-generated contributions. Previously, coding effort naturally limited contribution volume, but now bots scrape GitHub to create meaningless PRs. Finding the middle ground between pure no-AI policies (causing paranoia) and AI free-for-all policies remains an open community challenge. Legal questions about copyrighting AI-produced work under MIT/Apache licenses also need resolution.
Asking AI to review pull requests creates loops where each review finds new issues, resulting in overcomplicated code fixing non-existent bugs. The ideal approach is to avoid introducing bugs initially rather than relying on AI to identify and fix them through iterative review cycles.
When working with AI-generated code, it's important to distinguish between which bugs or AI findings are acceptable and which are not. For focused PRs, concentrate on ensuring the specific feature or change works correctly. If you want to address bugs, create a separate PR to avoid cross-contamination between different types of work.
Open source maintenance with AI requires skill to prevent PRs from ballooning into giant, unfocused changes. When working on genuinely large features in large projects, large PRs are inevitable, but they should be kept as focused as possible. Address AI review findings in a separate PR after merging your feature, doing a dedicated pass for bug fixes and focusing reviews on those fixes rather than attempting everything simultaneously.
The discussion covered numerous topics including rig, code review, Rust features, and controversial topics in Rust development. This was a joint live stream with JetBrains and the Rust Foundation.
Gold, Silver, Copper can be found on Discord and GitHub under the handle "Gold, Silver, Copper." Additional resources include the Playgrounds Discord for ZeroX Playgrounds, the company sponsoring rig development, and the grindhouse Discord server where Gold, Silver, Copper and Orhoon participate.
Thanks were extended to JetBrains for their support, with recognition of their PyCharm usage for Python development and the popularity of JetBrains IDEs across various programming languages, including AI agent features and tab completion. The Rust Foundation was thanked for their sponsorship and contribution to the Rust ecosystem.
RustRover was recommended as an IDE option for those writing Rust who prefer JetBrains products. The session concluded with appreciation for all participants and a farewell to viewers.
Keep JetBrains in your library
Save the videos and channels worth coming back to, and find them again in one place.
