Refinement-First Processes for AI Development | Stacy Cashmore | .NET Day 2026
In a Nutshell
Stacy Cashmore shifted from "optimization-first" AI development—where agents generated massive, broken codebases she couldn't control—to a "refinement-first" approach using specialized agents, OpenSpec specifications, test-first development with mutation testing, and vertical slice architecture. The key change was restructuring her role from passive reviewer of agent output to active problem-solver who starts with hypothesis generation and keeps humans in control of architecture decisions and quality gates. This eliminated the vicious cycle of AI writing bad code, then worse fixes, and restored developer engagement by focusing on solving user problems rather than managing code volume.
These notes were generated by AI and may contain inaccuracies.
Stacy Cashmore introduces the topic of "optimization-first" processes for developing artificial intelligence. The speaker explains that for the past two years, many people online have claimed that artificial intelligence is the right path and that its development will solve everything needed. However, the promised results were never fulfilled, particularly at the beginning.
The speaker describes going in circles with AI usage. People online were promoting AI solutions for blog posts, YouTube videos, and LinkedIn posts, claiming these tools would solve all problems. When tested on simple tasks, results were excellent, but when applied to difficult situations, results were terrible, producing poor code quality, bad user interfaces, and jobs that couldn't perform as intended.
Several benefits emerged from AI implementation. There was no more annoying routine code, and significant time was saved by avoiding writing repetitive code. AI proved useful for converting between languages, with a specific example being the conversion from TypeScript Playwright to C# Playwright, which was accomplished over several weeks or months intermittently while working in the background. Unfamiliar APIs became much easier to work with, demonstrated by writing Whisper AI programming in Python despite never having written in it before, enabling a working prototype within a day.
The negative aspects included consuming huge numbers of tokens for tasks that ultimately failed on the most difficult challenges. AI created massive amounts of code requiring extensive review before proceeding, with pull requests containing 500 files being particularly problematic. The speaker felt disconnected from the problem and solution, acting more like a mediator adding virtually nothing. AI would go completely off course, provide far more code than needed, and functions that didn't work. The system never checked whether outputs worked, frequently providing "Hi, here it is, I'm done" messages that resulted in code failing to assemble, malfunctioning on startup, or being functionally defective.
When code didn't work, the AI would respond with "You're absolutely right," "I'm sorry about that," and provide another change. When that failed, it would say "Good point," "I'm glad you noticed that," then restore the matter to its original state, wavering between these two states without solving the problem.
Stacy Cashmore introduces herself as a technology expert at DevOps for Omniplan, a financial consulting software company based in Amsterdam, Netherlands. She leads the AI product, figuring out how to use artificial intelligence in processes and applications in ways that are financially viable and resource-efficient. She has been a Microsoft MVP in development technologies since 2020.
After observing others successfully using AI, the speaker realized their approach was wrong. Through reading and watching videos, they identified tools and processes needed to overcome problems. Solutions were introduced step by step to understand the effect of each change and identify new pain points.
The first solution was using proxies, initially implementing "Squad" by developer Brady Gaster. This open-source program provides specialized agents for codebases, including architects, developers, testers, and stakeholders. The key insight was that agents worked well not because of multitasking but because of their specialization, with each agent having a very specific set of responsibilities.
Squad was eventually discontinued not due to issues with the tool, but because it became too expensive for their use case with too many agents. They began creating their own agents to find the perfect equilibrium point, though this still left them struggling to maintain the entire course of action, with too much or too little code being created and broken codes still running.
The next tool implemented was OpenSpec, another open-source tool installed via npm that provides specific commands, skills, and controls for software agents. The path allows exploring the codebase and problem, then suggesting changes. OpenSpec creates multiple documents including a proposal for changes, a design document, Gherkin-type specifications for verification criteria, and a task list with steps like 1, 2, 3, 2.1, 2.2, 2.3.
At this stage, manual changes could be made if something appeared wrong, or prompts could be used to have OpenSpec make changes. Both methods were kept available depending on the approach type and estimated time required.
Having specifications didn't solve the problem of agents thinking something was fixed when it wasn't. The solution was starting with test-first development, ensuring successful green tests after reading. At any point where tests remained red, agents knew more work was needed. This prevented broken software builds and functions that didn't work as expected.
Six months prior, the speaker would have said humans should write tests, but agents are now the best option since humans don't like writing tests. It's acceptable to have agents write tests provided humans review them and approve before implementation. Developer agents should not write the tests they will solve, requiring different agents for this separation.
Green tests don't guarantee good tests. Mutation testing using Stryker.NET ensures tests actually do what they're supposed to do. A review agent runs Stryker on the code to verify tests provide expected coverage. The goal is 70-80% coverage, as higher percentages result in flimsy tests with diminishing returns on investment.
AI prefers Python, but Stryker.NET produces JSON output. Cloud was consuming 23% of available code parsing JSON files with Python scripts. The solution was forcing use of JQ for direct queries on JSON output, saving significant code and time.
Interruptions from permission requests were causing mental burden and preventing optimal tool usage. The solution was running all agents in Docker containers as isolated environments with access only to the code folder. This allowed dangerous permission overrides since code could be reverted or containers deleted and recreated if problems occurred. Agents lack push authorization, providing significant productivity boosts with two-hour background runs without intervention.
The original Onion Architecture with API, application, kernel, continuity, and infrastructure layers didn't work well with agents, who had to examine entire codebases and sometimes placed files in incorrect layers. The solution was shifting to vertical slices, using a project for each vertical slice within the software solution. This allowed agents to look only at relevant projects, resulting in fewer codes, greater speed, and fewer inaccurate results.
Despite technical improvements, the speaker still felt bored, as guiding agents wasn't why they entered the field or why they programmed after work. Programming is enjoyable due to complex problems requiring real fights to solve, the satisfaction of making the world a better place, and helping users have happier lives while enabling communities to accomplish more with fewer resources.
The approach needed to change from the boring world of receiving changes, reviewing changes, and moving to the next change. Instead, involvement should begin from the start of problem-solving. Claude was brought in as a facilitator, never leading but challenging thinking without taking control.
Work elements in Azure DevOps debugging were changed from Epics being umbrellas for features, features for stories, and stories for tasks. Now Epics represent problems being solved, making them shorter-lived but disappearing faster to show clear progress. Multiple hypotheses were selected using Claude to generate different solution ideas, preventing settling for the first solution. Alternative solutions were documented for potential fallback use.
Features and stories went through similar processes, ensuring understanding of why and how each part fit the plan. A set of questions was used with Claude as a reference point, ensuring different system parts were considered. Claude identified which questions could be answered from conversations and only prompted addressing remaining questions before proceeding.
Manual copying and pasting of eight questions, specifications, and OpenSpec guides created errors and was disliked. An MCP server was added so that when conversations indicated work items were ready, Claude would determine what to include and create it automatically, maintaining human control and review while eliminating manual work.
The debugging process was moved from the Cloud desktop application to Cloud Code via command-line interface, providing direct access to all code and repositories. This enabled larger changes by allowing system analysis across various repositories to identify needed modifications from the start.
The revision process was moved to the job task creation stage. Previously, revisions occurred long before programming, forcing context changes and knowledge loss. Now, starting with a large Epic, once the first feature is completely refined into code stories, a new agent executes those stories while work continues on stories for other features.
The traditional image of a developer sitting in the basement drinking cola and eating pizza, as described in the nineties, will no longer exist. Developers have always needed to be involved in the revision process, but with artificial intelligence, this has become mandatory. The rapid pace at which artificial intelligence will change and the rapid pace at which its cost will change means this will not be a short journey before things settle down.
Research has been conducted on whether chatbots are harmful to our minds and affect our critical thinking. People have been observed saying, "Well, that must be true because the artificial intelligence said so," which is completely wrong. Developers need to remain in control when it comes to editing and programming, using AI agents as a tool rather than an entity to which tasks are handed over without thinking.
Work should be done in small parts because the more you give artificial intelligence a part, the harder it is to refine it. Reviewing a pull request consisting of a thousand files should be avoided. The chances of work going astray are reduced when parts are small, and if a deviation does occur, its impact will be less because we are dealing with that small part.
Stories that take no more than a few hours to complete with agents should be targeted to ensure that what is produced is small enough to be safely reviewed.
Boredom has never been eliminated from development entirely, as there are always times when satisfaction is not felt. However, the difference is that boredom is no longer the default state when using AI agents. The mind is always used to try new things, experiment, and put the user first, which brings energy and enthusiasm back into the work.
These processes do not remain constant. In the Agile world, if you're working in an Agile way and doing the same things you were doing 12 months ago, you're not agile. AI agents are very new, and this field is constantly changing. Processes are probably changed at least every two weeks, and sometimes daily whenever improvements can be tried. This must continue.
Lines of code were a poor measure during manual development, and when artificial intelligence develops, it becomes a worse measure. In March of the current year, people were boasting that they write 37 lines of code per day, but they didn't boast about whether they were producing actual value. Some repositories have been examined and the code was not impressive.
The notion that the use of AI agents should be about producing tens of thousands of lines of code must be abandoned. Time should be used to solve user problems in a better way and produce more valuable software.
The first lecture attended in 1994 when starting university was in software engineering, where the lecturer tried to instill the idea that software developers and software engineers shouldn't just be typists of data. Developers should be the ones who solve the problems, with code being just a tool used to achieve that. The solution created for the user is what should be focused on.
This principle remains as true today as it was in 1994. Let artificial intelligence do the writing. It is up to developers to think, solve problems, and come up with great things that serve the community.
The importance of specifications, keeping the agent's context on track, and specialized agents were identified as major pain points in AI development. Keeping the fun in work so that developers don't simply turn into machines was also emphasized, along with automating things that can be easily automated.
The number of lines of code does not equal the quality of the code or problem-solving abilities. This is why developers get paid at the end of the day, and no one gets paid per line of code.
When trying an agent last year, developers probably got caught in a vicious cycle where an application is written, agents generate code, the code is bad, another order will fix it, it produces worse code, and then something else breaks. This situation snowballs quickly.
Developers can mess up the codebase themselves quite well without needing it to escalate that quickly, but agents do the same thing.
The journey has been long with many mistakes made and many oaths sworn. The hope is that people can learn from this journey and these mistakes. The problems encountered are not new problems—it's a new way of uncovering problems, but they are the same problems faced in software development since it began in the 1990s.
These problems can be seen much faster when handed over to an agent who can produce 37,000 lines of code in a single day, and all problems are suddenly revealed. The issues are amplified in many ways.
Imposter syndrome is sometimes felt regarding what is happening with proxy programming and processes. Hearing that other people have encountered these problems but are constantly improving and changing their processes is reassuring. Continuously improving and changing processes is really essential, and forward momentum should be maintained.
The session concluded with appreciation for the practical comments shared. A blog post covering the discussed content was mentioned as forthcoming. Questions were invited from the audience following the presentation.
Keep JetBrains in your library
Save the videos and channels worth coming back to, and find them again in one place.
