The limiting factor—how to design an AI software factory for speed | Geoff Charles (Ramp CPO)
In a Nutshell
Ramp's AI software factory removes sequential bottlenecks in product development by deploying specialized agents: Glass for problem definition from customer data, Inspect for code generation (75% of PRs), Review Buddy for automated code review (93% of PRs), and Testo for QA testing (425 errors caught in 30 days). The system automates small loops and coordination through Gadget, enabling 60% of customer UX issues to be fixed within 24 hours while shifting product managers toward technical architecture, taste-making, or general management roles. Speed comes from continuously identifying and eliminating the next constraint rather than optimizing individual tasks.
These notes were generated by AI and may contain inaccuracies.
The speaker opened by acknowledging the previous speakers and their preparation. They rewrote their speech three times after learning storytelling techniques. The core theme centers on speed—specifically, the speaker's personal passion for speed demonstrated through participation in the 24 Hours of Lemons race, where participants buy a $500 car and race for 24 hours. Their car did not finish the race due to damage, which was entirely the driver's fault.
In professional racing, the driver accounts for only 15% of race impact. The real determining factor is the interaction between driver and car, and between car and team. Victory depends on removing obstacles that hinder the leadership process rather than simply driving faster. The best drivers can only perform to the extent that the system allows.
In the 1950s, tire changes took 67 seconds. Today, they take only 1.8 seconds. This improvement did not come from asking mechanics to work 37 times harder, but from identifying weaknesses and eliminating them through specialized jobs, better technology, and more training. Formula 1 teams change 90% of car parts every year, with only 10% of the 16,000 parts remaining the same year-over-year.
The speaker emphasizes that organizations are in a race to build using artificial intelligence, where speed and execution are everything. For AI development, the race starts when a problem arises and ends when the customer receives a product that solves it. AI removes obstacles but then shifts them. The winning team is the one that can quickly find obstacles, remove them, and move to the next obstacle.
The traditional product development lifecycle involves defining, specifying, building, and improving. Engineers have automated much of their work, making programming easier. The bottleneck has now shifted to product managers, requiring more work in definition, collaboration, coordination, shipping, testing, and releasing. The speed of identifying bottlenecks and redesigning systems around them determines race outcomes.
The first step is identifying problems through various data sources including Gong, Zendesk, Log Rocket, surveys, and executive emails. Context exists everywhere with many barriers and opinions. The initial bottleneck is filtering noise from signal. Learning management programs proved inadequate, with one million unique symbols representing less than 0.5% of Gong scripts at Ramp.
The team started small by creating a dedicated channel for constructive criticism, posting daily positive customer quotes. This quickly became unmanageable, requiring a complete redesign. They created a customer data analysis agent that retrieves data from all company sources using traditional ETL processes and vector search, grouping around context while understanding products, teams, and features.
After implementation, the focus shifted to maximizing user access through experiments including a Slack proxy for questions and an HTML dashboard. Podcasts were particularly effective, with the speaker noting it was "a great way to start your day" to listen to a hundred customers pointing out product flaws. The goal is providing access to data as quickly as possible while using data to identify precisely which customers to engage with directly.
Once focus areas are determined, the next step is moving from idea to complete project definition. Rather than asking AI "What do you want to build?", which may not be optimal, the approach integrates AI into existing systems. Ramp developed "Glass," an AI agent connected to all systems including Snowflake data, user research, and customer tasks. Glass understands product strategy, specification definition, and the software database.
This allows AI to serve as a technical leader and thinking partner, eliminating the need for product managers to ask engineers feasibility questions. Glass can identify and build working prototypes using understanding of product principles, design systems, and software databases.
Engineering teams do not want individual details, prototypes, or lengthy specifications. They need a combination of qualitative and quantitative data demonstrating the problem exists, actual requirements usable by programming agents, and a prototype for inspiration. This forms the foundation contract for building.
While building is no longer the primary obstacle, when one obstacle is removed, another appears. The first construction obstacle was programming. Ramp developed "Inspect," their own programming agent, chosen for robust architecture and codebase understanding. Inspect operates within Slack, can be activated in under 5 seconds, and displays actual products in deployment previews for product manager interaction.
Inspect has recorded one million sessions, with 75% of pull requests built by the agent. In the last month, 1,000 requests were submitted by non-engineering professionals. Automated programming is most effective with strong infrastructure and robust software databases.
Increased code generation created code review bottlenecks. Engineers faced reviewing enormous code volumes. Ramp created "Review Buddy," which understands the software database, quality checks, and security concerns. It identifies appropriate human reviewers and understands context and motivations behind code changes, enabling it to scrutinize AI usage.
Review and collaboration now operate with clear context. 93% of pull requests are processed automatically by Review Buddy, allowing top engineers to focus on the 7% of requests involving critical questions about correctness and risk.
Testing was the next bottleneck. The traditional QA process involved creating environments, testing modifications, and verifying feature completeness. Ramp developed "Testo," a browser-based QA agent that runs the product in 100 different formulations based on production data, interacting with the product like actual users.
Testo receives instructions such as "Pay the bill, but divide it into installments" and provides feedback including errors, design issues, and system language problems. These observations feed back into the product development lifecycle. Testo detected 425 errors in the last 30 days that should have been caught internally rather than by customers.
With fast-paced work and high product output, human attention became the next bottleneck. Product managers were overwhelmed with notifications, creating coordination challenges. Too many procedures slow developers, while too few create confusion.
The solution treats each question as an API within a designed system. Tools were built to understand question purposes and link them to official records across Notion roadmaps, specifications, customer calls, Slack, Linear, and tickets. The principle requires organizations to be clear enough for agents to read and understand.
The "Gadget" system handles questions like project status, launch readiness, and next steps with evidence. It updates roadmaps, provides status updates, and alerts people about late deliverables. For sales team questions about features, availability in Brazil, use cases, and pricing, Gadget provides comprehensive answers and refers unclear questions to appropriate people.
Gadget can write help center articles, blog posts, and customer emails for product launches. 85% of questions directed to product managers are now answered entirely by AI, with unanswered questions returned to the system for resolution.
Product managers naturally focus on small, interactive, trusted, and easy items that provide dopamine hits from solving visible problems. This represents a mistake. The approach should automate away from these small loops. For most small issues, AI takes over the entire loop by directing to appropriate teams, matching with accumulated to-do lists, removing duplicates, tallying, sorting, and executing plans.
Human oversight occurs via Slack with questions like "Is this ready? I'm ready to program." The system writes code, performs testing and CI/CD, and reviews the knowledge base upon release. Humans follow these superficially. This automation enables 60% of user experience issues identified by customers, salespeople, customer experience managers, or internal teams to be fixed within 24 hours.
Three questions arise about this approach. First, measuring progress while maintaining quality is more difficult than measuring lap times. In 1974, Niki Lauda told Enzo Ferrari that his car was bad in driving, handling, and braking, then worked with engineers to fix it—similar to Ramp's culture.
Speed assessment comes from hiring drivers who understand real speed, placing them in responsible positions, and allowing them to challenge leadership. This may be difficult for some leaders but necessary for facing challenges from those who know true speed.
The second question concerns budget and resource limitations when competing against unrestricted companies. In 2006, Audi had a slow car and asked how to win without being faster than everyone. Their answer was fuel efficiency—fewer stops meant more track time. They won Le Mans three consecutive years. Constraints force selection of a field for global excellence. Accept limitations but not weaknesses. Find areas with tenfold multiplication potential.
The third question addresses whether product managers' jobs are being automated. The speaker believes it's positive that product managers operate outside areas engineers control, enabling greater impact. Three evolutionary paths exist:
- Technical path: Product managers who build the physical factory, identifying bottlenecks and removing organizational obstacles. They ship products that help build customer products or enable AI to do so.
- Taste maker: The engine who sets standards for great products and good taste—things AI cannot do.
- General manager: Product managers expand beyond product organizations to include marketing, sales, growth, and operations, owning business results and leading entire functions.
Speed is not about writing code faster but identifying and removing bottlenecks in build processes. Everyone has a starting point with always-present obstacles. Once the first obstacle is solved, another appears, requiring rapid overcoming. Product leaders should reduce focus on delivered products and increase focus on work environments that enable faster product building.
Everything demonstrated is already outdated, like Formula 1 teams copying each other. The focus should be on moving beyond current achievements. The best Ferrari is always the next one, and the best product is always the next release. This begins at the software factory created today. The goal is community sharing about building products that offer more to customers.
Keep Lenny's Podcast in your library
Save the videos and channels worth coming back to, and find them again in one place.
