Spec-Driven AI Development – Episode 2
In a Nutshell
Spec-driven development keeps humans in the loop by having agents generate code only after humans review and approve specifications, rather than allowing direct commits. The main challenge shifts from generating code to managing hundreds of thousands of specification documents and preventing agents from bloating code with irrelevant edge cases. Working in small, reviewable units on branches with frequent commits, using plan mode for upfront decisions, and treating specifications as the source of truth enables effective human oversight while leveraging agent capabilities.
These notes were generated by AI and may contain inaccuracies.
Paul Everett, developer advocate at JetBrains, hosts this live stream as a follow-up to a previous session on spec-driven development. He is joined by colleague Dimitri Jemerov, known as the creator of PyCharm and a key architect of the IntelliJ platform.
Dimitri spent significant time working on the IntelliJ platform, transforming a single Java IDE into the product line including PyCharm, RubyMine, and other IDEs built on top. He was also involved in the early days of Kotlin and co-authored a book on Kotlin with Lana.
The session aims to demonstrate spec-driven development in action on a real project rather than a dashboard demo. The goal is to show the best of both worlds: agents can generate code, but humans remain in the loop to review and approve before committing to main. The human in the loop needs effective tools for working with large software development projects.
Dimitri raises the issue of AI fatigue as a major challenge. Agents can generate hundreds of thousands of lines of code at the cost of tokens without significant effort. The challenge shifts from generating code to determining whether the generated code meets actual needs and is appropriate for the application.
AI systems frequently generate code to handle edge cases that may not exist in the actual application. Examples include handling zero-width non-breaking joiner spaces in Unicode strings. This results in bloated code that could be solved with one-tenth the amount of code if written by a human. The volume of code makes it exhausting for developers to review and make decisions about each edge case.
Dimitri highlights a critical challenge: managing the volume of specifications that accumulates over years of project development. IntelliJ IDEA has resolved a couple hundred thousand issues in its tracker. Converting all decisions made (not always covered by issues) would result in approximately half a million individual specifications. The questions remain: how to manage half a million markdown documents, how to fit them in an agent's context window, and how to ensure the agent accesses relevant specification documents.
Paul conducts an informal poll asking whether viewers currently allow agent-generated code to be committed directly without human review. Multiple responses indicate developers prefer remaining in the loop, with some noting this might change in the future.
Several frameworks implement spec-driven development:
- GitHub's Spec Kit was one of the first to popularize the approach
- Kira from Amazon is an IDE built around spec-driven development principles
- Anti-gravity from Google emphasizes a plan mode as a first-class UI citizen
- Agent OS by Brian Castle provides commands, skills, and automations for Claude Code implementing his philosophy of spec-driven development
Brian Castle describes a three-layer context approach: company-wide decisions, product direction, and feature-specific specifications.
The largest projects successfully generated by agents are those with well-known specifications. Anthropic's agents built a C compiler capable of compiling the Linux kernel. OpenAI built a web browser and a Java language server (with an order of magnitude more lines of code than expected for a Java language server). No agents have yet built projects where requirements are not well-known.
When Dimitri served as head of product for the IntelliJ department, his primary concern was ensuring the right features were built within a single conceptual framework that remains discoverable by users without confusing them through different paradigms for different features.
A quote from Corey Doctorow's PyCon keynote is referenced. For a project like IntelliJ, specifications would amount to megabytes or potentially gigabytes of markdown files—too large for context windows and creating maintenance burdens.
Agent OS provides commands that implement a workflow connecting agent machinery to produce results according to specific requirements. The workflow includes:
- Product conversation: documenting what the product is, who it's for
- Tech stack decisions recorded in markdown
- Provisional roadmap of features and directions
- Reading markdown files, asking questions, reaching agreement before proceeding on features
- Each feature includes planning, implementation, and artifact documentation
Dimitri notes that some specifications contain more text than would be required to write the code directly. For example, adding two optional keyword-only parameters requires simply adding the parameters rather than explaining the AI to do so. Over-specification makes review harder because agents can generate 100-item checklists where 98 items are obvious and 2 are critically wrong, making it challenging to identify the problematic items.
The conversation references CASE tools from the 1990s using Rational Unified Process to generate code from specifications. Dimitri notes that generating code from UML was a hot topic in the early 2000s but didn't gain widespread adoption. The interesting connection is non-text representations of code. For a long time, working with code as text was necessary because it's easiest to store in git and all tools work with it. However, when not typing code directly, non-text representations may make reading easier through visual means.
Sean Grove from OpenAI gave a talk titled "Spec is the new code." The discussion explores whether Python code could become an artifact of a higher-order language, similar to how Python itself abstracts memory management.
The demonstration uses an existing published Python package for testing HTML accessibility that was not originally written with spec-driven development. The project will introduce spec-driven development into an existing codebase. The package targets testing library functionality from the JavaScript and React world, ported to Python. A key challenge is that it uses a Python 3.14 feature after the training cutoff date, requiring extensive work to teach the model about this new feature.
Before starting work, the process includes checking for uncommitted work, ensuring a clean working state, and reviewing git history. The first real step in the Agent OS workflow is having a conversation about what the product actually is.
The /plan-product command in Agent OS initiates the planning process. It contains principles reflecting Brian Castle's approach to spec-driven development and instructs the agent to ask follow-up questions. The command generates three markdown files: mission, roadmap, and tech stack. Since this step generates no code, branch creation is typically not needed for this initial planning step.
The product is identified as a library for testing accessibility based on HTML, targeting developers and testers with no dependencies. The project is a port of a testing library from JavaScript/React. The first feature to add is a new kind of query. The existing todo.md file contains organized work items from before adopting spec-driven development.
The agent correctly identifies that the first roadmap item is adding a "by commit" query. The roadmap is structured with phases, where phase one and phase two relate to this query implementation.
The specification documents must be kept up to date because they steer all future work. The goal is to avoid retyping context in every prompt. Agent OS and other agents have tools to read artifacts and rewrite them to be context-friendly.
From a software development lifecycle perspective, these specifications serve as a contract with the agent. The question is raised whether this functions as a contract with business stakeholders. Dimitri notes that specifications don't necessarily need to be developer-oriented, observing that the current library's target audience is developers.
Specifications for developer-oriented tools use terms like views and controllers. Applications targeted at end users should have specifications phrased in terms of end user features. This level of detail is probably too detailed for executives. There may need to be a hierarchy with a very high level specification at the top.
The product mission is targeted at management. UX engineers can look at specifications that describe how individual features work. Operations requires a separate set of specifications describing how the application is deployed and monitored.
Research territory exists around sub-agents who receive a subset of configuration context and instructions. These specialized agents do their job well without bloating the main context. They may run in parallel with proper orchestration.
Instead of going into chat, write a markdown document first containing all thinking. Point the spec-driven development process at that document and get it to interview you. Businesses probably have these documents lying around. They go out of date very quickly. Microsoft Word is a write-once format.
The project has the start of spec-driven development. The tech stack has been specified so the agent makes all decisions based on that. The agent asks great questions about decisions related to other projects, initiatives, or thinking.
Work in really small units on branches with frequent commits. The surface area of review becomes smaller. Context management improves. If units are too small, development slows significantly. Asking the agent to build every individual function and check correctness, then break development into multiple steps like writing a spec, task list, then the function, spends significant time and tokens without much value.
The right size of chunk depends on the project. The intuition points to the order of magnitude of a few hundred lines of code, the size of PRs that were comfortable reviewing before AI tools existed.
The platform remains open for plugging into other agents. Models leapfrog each other. Being tied to just one agent is not advisable. Choices include writing custom agents using ACP and experimenting with Python agents.
The agent can create specifications and steering files without human input. This fills in details and limits what the engineer must load into their brain. The context window applies to humans as well.
For new projects, if accepting that the project might suddenly be in Rust rather than Go, the agent can make all decisions. Agents do not care about the language they write in. The runtime environment matters more. Switching between JVM, .NET, and Go still matters.
The roadmap contains features. Work proceeds one feature at a time in small units. The prompt tells Claude Code to find the roadmap, find the next item, make a branch, go into plan mode for conversation, then run the agent OS command to work on the specification.
The feature adds a new kind of query to query HTML not just by element but also by comment. The agent asks follow-up questions about decisions like whether to return text or the element.
Plan mode uses Opus which does more thinking and keeps track of the context window. The art involves managing context rot where context in the middle loses importance relative to context on the ends.
Software becoming much cheaper means more custom software will be developed. Companies that would not deal with SAP can now afford custom systems tailored for specific requirements and processes. People who can maintain software over the long term will remain in demand.
The tools are not currently a threat of replacing serious software engineers. Skills must be learned to bring engineering to agents to get the best of both worlds. More programming can be done with acceleration, but results should match what would have been written personally.
If doing a GUI or website, a hand sketch on paper or Figma document can be fed into the specification. This is retained over time so when the next person wants to see the why behind a commit, they can find the inputs the human gave.
The agent asks good questions during feature conversations. The agent knows about other similar implementations and asks whether the same behavior is desired. An HTML comment query might need clarification about whether the parent element or the actual text is wanted. The decision was made to return the text of the comment, not the parent element.
Five features from now, decisions might change. The question arises whether to go back and change all specifications or treat code as the source of truth. The direction is that specifications become the source of truth. Andre Breslav, original designer of Kotlin, is working on a startup designing a programming language consisting of specifications. The entire software development workflow consists of writing and changing specifications. When understanding of the task changes, existing specifications are changed rather than writing new ones. The hard problem is determining the right amount of code that needs to change based on specification changes.
If a model could be trained to understand specifications as the thing for discovering context rather than going through code, radical improvements could result. This may be achievable with existing models. Agent OS represents a big step in that direction.
Overspecifying and constraining the agent fills up context. Typing less and letting the agent explore produces better results.
No formal process demands AI use. Adoption is based on individual developer initiative. People see results colleagues are getting. A lively Slack channel discusses experiences and specific approaches. Company products are not mandated. Many people use Claude Code on the command line.
During planning, the agent asks about skills to inject. The agent asks follow-up questions about whether to match anywhere in the comment or require an exact match. The instruction given was to use the same policy as other queries like Titan. The agent looks at existing code to see how similar decisions were made previously.
Plan mode saves time and money by writing down a better starting point. Fewer hops are needed to reach the right destination.
To become an effective developer, all steps of figuring out how coding works and how software works must still be gone through. Senior developers cannot be created by just sitting around prompting an agent.
At the end of planning, the agent states what it thinks is the truth for the feature. This includes what the feature needs to do, decisions made, and subtasks for completing the work. Implementation perspective corrections can be made before proceeding.
Agent OS wires in a verification step before finishing a feature. For Python projects without a compiler, the steps include running the Python formatter, Python linter, Python type checker, and all tests. If any fail, the agent returns to work until all errors are resolved.
No strong opinion exists on whether to run the type checker on each subtask or only at the end for languages like Python.
Skills and plugins exist for code simplification. These find mistakes beyond what a linter could do. An ecosystem exists where people share skills and get better together.
Questions exist about whether this level of tooling can perform new kinds of code analysis. Linters can report 100,000 warnings for a codebase. The IntelliJ codebase produces tens of thousands of warnings. In 90% of cases, these warnings are not cared about. AI tools carry the danger of finding things considered issues, adding code to fix them, when they were not actually issues. Security issues often involve weird contexts that would never be encountered in actual production use.
Agents will fix things that are not really issues and bloat code. Fixes may not understand the other side of the codebase where another fix exists.
Context is very important. A person has project context in their head, even if fuzzy. An agent never has complete context. An agent looks at specific pieces of code, runs grep commands, reads markdown files, and that is all it sees. Agents have never seen the rest of the codebase.
Putting all knowledge in CLAUDE.md or specification files may not scale.
The mode of sitting in the commit tool window watches changes as they are made. Early review starts using the diff viewer. Scrolling through changes allows reviewing as work proceeds. Docstrings are written, types are returned. Helper functions are created. Structural pattern matching is used when appropriate per the given rules.
Implementation complete reports state what was done. Files written include the implementation plan, specification, and all specification-related materials. Modified files include functions added for the new query type. The split between helper and user-facing code enhances testability. Unit tests can be written on the smaller part. The smaller part can be reused in other query helpers.
Documentation was written including examples explaining usage to users. Quality tests ran successfully.
After implementation, the human reviews code. Test runners can be executed. Naming decisions that do not match preferences can be changed using refactoring tools. Refactor rename trusts the IDE to find all places needing renaming. Work moves from the non-deterministic side to the deterministic side when the human takes control.
Documentation targets other people using the feature, not other agents. Time is spent ensuring explanations are correct for users. The IDE is used heavily with its markdown editor, jump to symbol capability, and ability to run Sphinx for documentation preview.
Human in the loop time is essential for review work. The best possible tool requires sitting in the IDE with no big context switch, allowing the two systems to work together effectively.
Dmitri addressed whether to let the agent write tests for code the agent wrote. While Trisha suggested it might not be the best idea, Dmitri's view is that it is the right thing to do. Writing tests requires making sure they are done in the right way, but writing tests can be fairly cumbersome with a lot of setup code needed.
Dmitri often finds that when working on personal projects, he wants to move quickly without the level of standards required for team work. He often skips writing tests because he knows he can fix issues if they break. With the agent, this step can be handled by having the agent figure out the cases that need to be tested and write the tests. If the agent has good examples and patterns to follow, it can be very valuable.
A small but significant practice is telling the agent to generate commit messages, perform merges, and delete branches to save the work of typing and performing these actions manually. While it's cheaper and faster to do these tasks directly, the convenience of delegating them to the agent represents a broader question about where work should be done.
The choice between working in the fast, cheap, deterministic environment versus the clever, unpredictable, and expensive side represents the tip of the iceberg for where work will be conducted in 2026.
Dmitri addressed questions about languages and agent effectiveness across different programming languages. The size of the language specification is not what matters. There is substantial open source C++ code, giving agents the opportunity to learn C++ coding patterns. There are also many coding standards that explain the right way to use C++ in different project contexts.
Dmitri hasn't personally used agents for C++ development but doesn't see why this would be an issue. C++ being a statically typed language means the compiler can verify many things, and while not as strict as Rust, many language-specific problems will be caught by the compiler.
An interesting point emerges at the intersection between the non-deterministic side and the deterministic side. The question becomes how agents and models can have a conversation via the compiler to try and get to the right result the first time instead of the cycle of write code, fail, write code, fail.
Developers are switching away from building to revealing, which represents a different skill where velocity is exhausting. This goes beyond review fatigue to a fundamental issue where developers end up with a system they don't understand. While you can ask the agent how the system works and receive an answer, knowing something and being able to ask something are very much not the same thing.
Dmitri expressed excitement about the unexplored space in infusing IDE features with AI so that it's not just two separate things - a separate IDE and separate terminal - but rather features in the terminal and IDE that are powered by AI and do things in a smarter way because the AI is available to them.
Simple examples include generating commit messages with AI, which was one of the first features added to the AI system and continues to be popular, and when renaming a method, the AI can suggest a name much more meaningfully than the heuristics previously used.
There are many dozens of small points where IDE features can be made smarter by asking AI what the right thing to do is.
The biggest risk is loss of understanding, where developers end up with a large system they don't actually know what's inside. They don't know how it can break or the assumptions the agent made about how it's supposed to work.
As the system evolves and the agent continues developing it, unexpected changes can affect users because the agent can change things without comprehensive specifications. Users become confused when the system works differently today than it did yesterday because the agent made changes that weren't expected.
An offer was posted in the chat for IntelliJ JetBrains AI Ultimate free for one month with a commercial license for up to 10 seats. The tool provides access to machinery related to keeping the human in the loop.
Everyone is learning this together with no right answers but many wrong answers to explore throughout the year. The focus should remain on users and the product being built for them - these are tools, but the priority is giving users what they want to give them.
Keep JetBrains in your library
Save the videos and channels worth coming back to, and find them again in one place.
