SAP: Bringing the ‘Operating System’ of a Company into the AI Era with CTO Philipp Herzig
In a Nutshell
SAP CTO Philipp Herzig outlines SAP's transformation of its enterprise "operating system"—spanning finance, HR, supply chain, and more—into an AI-powered platform, embedding generative UIs, proactive agents, and a knowledge graph to blend structured/unstructured data for 30% efficiency gains and faster outcomes. Key innovations include Joule AI copilot for consulting/travel, Relational Pre-Trained Transformers (RPT) for scalable predictions outperforming LLMs on tabular data like demand forecasting, and evals-driven agent development to ensure verifiability at massive scale across 20,000 APIs. SAP prioritizes customer ROI through hybrid pricing, data harmonization, and upleveling roles by automating mundane tasks, positioning it to win the enterprise AI outcome race.
These notes were generated by AI and may contain inaccuracies.
There's a lot of exciting opportunities and new things you can build that we only dreamed of in the last 20 years at least since I'm a developer. SAP serves some of the largest customers with a lot of heritage and complex landscapes. They can reduce 30% of their efforts to get to the outcome faster, which directly reduces costs. The time is clearly over where you design software that requires the intelligence to sit in front of the computer. In classical software, you designed a user interface to teach a human how they get their tasks done by clicking through the UI. Essentially, this is over.
Hey listeners, welcome back to No Priors. Today with Philipp Herzig, CTO of SAP, the enterprise juggernaut. Topics: SAP's AI strategy, why SAP has endured and thrived through several technology transitions, why entrepreneurs underestimate challenges of scale, why AI is a business model transition not just a technology transition, why LLMs are not enough for predictive analytics, traveling salesman problem in the real world and Strait of Hormuz.
Welcome, Philipp. >> Yeah, it's a pleasure to be here. Thank you.
Everybody knows SAP but for lots of engineers or people not close to larger enterprise systems, they don't know the breadth and function of the platform. Describe what SAP does for customers.
SAP is market leader in enterprise software applications and platforms. 400,000 enterprise customers running finance, HR, supply chain, manufacturing execution, logistics, warehouse management, sales, services, commerce, procurement end to end. SAP has the broadest portfolio for running the business end to end. SAP started with giving real-time insight. SAP is the operating system of a company to manage everything from order to cash or source to pay end to end for companies around the globe.
SAP has stood the test of time through multiple technology and market cycles. SAP is the largest market cap enterprise software vendor versus the last generation like Salesforces.
How has SAP endured? What makes it durable?
What makes it durable: even with low costs of building using agentic coding and powerful models, something has always prevailed. SAP founded in 1972. Founders at IBM went to each customer implementing finance systems repeatedly. They realized it doesn't scale economically. They created standard software. That stood the test of time because companies need end-to-end management.
Transformed from mainframe to client-server, internet, mobile, now AI. Software evolved with technologies. What hasn't changed: customers seek outcomes and ROI to get things done.
AI helps get more done in the enterprise, what SAP stands for. With breadth of portfolio and customers, SAP helps achieve more by deeply embedding AI, AI agents, transforming user interface to get more done in any industry. SAP can continue because AI adoption in enterprise is not where it should be. There's an AI innovation race and AI outcome race with a growing gap. SAP focuses on customers getting outcomes from AI given the foundation, while re-engineering the entire system with AI in a new way.
As CTO, broad purview including AI strategy internal and for customers. Top priorities for organization and where SAP is on reimagination journey.
SAP is all in on AI. Everybody uses agentic coding for productivity boost across programming languages.
Focus on customer outcomes. Example: Joule for consulting, one of fastest growing AI products. Helps consultants in SAP projects or complex landscapes move to cloud, adopt AI capabilities. Reduces 30% of efforts to get outcome faster, reduces costs. Example: Joule for Concur with travel booking agents, expense agents.
From CTO perspective, three things massively changing like move from on-prem to cloud. Initially thought lift on-prem to internet and call it cloud, but realized CI/CD means daily deploys, multi-tenancy scaling. Software re-engineered for real SaaS.
With AI, same on three levels:
Generative UI and Proactive Multimodal Systems
The time is over designing software requiring intelligence in front of computer. Classical software: design UI based on user research to teach humans tasks via clicks. Now generative UI, dynamically generated. Example: analytical questions, deep research on tariffs, new taxes, Strait of Hormuz impact on supply chain analyzed with SAP data. Exciting opportunities dreamed of for 20 years. System more multimodal, proactive: runs overnight, alerts "Hey Sarah, have you looked at this? Problem on sales side, order entry down, recommendations." Supply chain problems for oil and gas customers, replan options.
Business Processes with Agents
Order to cash was rigid. Now agents blend structured and unstructured seamlessly for more work. Move from software as a service to service as software or outcome as a service.
Data Layer
SAP has valuable data: general ledger, invoices, warehouse, inventory. Combine with other data for harmonized semantic view. AI only as powerful as the data.
Transforming data side for globally harmonized data model to fuel AI.
Biggest challenge: teaching AI to do the right thing at scale. Not AI itself. Two years ago, easy to build RAG chatbot on 10 documents. At scale with SAP/large customers: 100 documents harder, thousands engineering challenge.
Example: Joule, SAP US employee asks travel policy vs German employee: connect to master data (location, country, payroll, taxes).
RAG simple for 10 APIs, context bloat at 100, SAP has 20,000 APIs. Problem of scale end to end for integrated experience across finance, HR, supply chain.
Most important from development perspective: people start writing evals.
Agentic coding works because verifiable by outcome: compiles, unit tests. Combined with taste and engineering, Anthropic/OpenAI built phenomenal code models.
For reliable outcomes in finance: need data with input/output for validation. Mindset shift: describe boundary conditions to coding agent (security, data privacy, code quality for day 2+ maintenance), evals to verify agent does what supposed to.
Test-driven development coming back. In CS student days, Google guys said go home at 5pm because wrote tests. Nobody did it: more fun to code first, product managers gave messy requirements, iteratively discovered behavior.
Now behavior and code automated, so describe right outcome. Changes developer work. Models improved since December last year.
Can agents/models in enterprise like SAP improve compounding like code generation in verifiability/evaluation against intent?
Starting condition great in two lanes. System of record: know exact outcomes (database, order to cash process, accounts receivable, taxes).
Challenge: system data insufficient for autonomous enterprise, increasing agent agency. Agents start with human in loop, ask clarifying questions.
Capture tribal knowledge (in heads, Slack, Teams, phone). Past: process mining, now agent mining: record decision traces, user context.
Detect anomalies (UK/Australia vs standard operating procedure) or improvements to elevate to new global standards for efficiency.
Leads to data flywheel: every trace/input/observability generates new data for evals, verified outputs, optimize toward outcomes.
Strong view on agents in business processes: computer use or code/tool use on APIs?
No finite answer. UI automation clunky/slow but phenomenal. Majority will be tool calling, agents in background. Structured integration preferred over always-open browser/headless. Computer use for missing APIs/legacy systems.
Most productive first in unstructured world: LLMs good at text/images. Services, support, sales, knowledge work with documents. Joule for consulting: unstructured info, quick ROI.
Harder combining tool use: models learn tool use, build orchestrators, disambiguate (order: maintenance/sales/purchase).
Now chat with data: natural language to SQL, converse to viewpoint, pin as tile to manage problem for weeks.
You move on, delete that tile, and so on. This combination of the structured unstructured world is required if you want to go into the tabular world. Lots of data in finance is stored in tables, sales, supply chain, and so on. Unlocking that took time, but now through the SAP knowledge graph, the glue between natural language and structured data in the system, it brings this together.
LLMs aren't the be-all end-all for predictive and tabular models. Why can't we just use tools and calculations external to the model in combination with LLMs?
From a business motivation, LLMs handle the unstructured world well, but to plan forward and make good decisions in a company, you need predictions: demand forecast depending on seasonality for retail products, manufacturing planning, cash flow prediction.
Input variables include days sales outstanding, determined by whether customers are paying (classification) or payment delay (regression).
Large language models are not made for predictions; they generate one token after another in sequence-to-sequence modeling. For predictions and what-if analysis, go back to classical machine learning like XGBoost or AutoGluon AutoML approaches. The problem is it doesn't scale; no democratization in the predictive space. You need talented data scientists.
Example: Pharmaceutical company operating in 90 countries needs payment delay prediction models for each, resulting in 180 models to train, create data for, feature engineer.
SAP has hundreds of thousands of tables with this information. Apply the same idea as LLMs/multimodal models did for unstructured data to structured data: provide a little context, small amount of data, and make high-accuracy predictions.
This was two years of research, published at Europe and other conferences. Called RPT (Relational Pre-Trained Transformers), based on transformer architecture but very different. Released it, seeing very good results in classification, regression, time series. This will be huge, allowing more people to make predictions that LLMs struggle with.
Hype vs. adoption in enterprises: innovation race vs. outcome race.
Primary problem is data, often disaggregated due to past purchasing decisions, M&A bringing different IT landscapes. This limits AI potential.
Next: integrate safely. Customers who did their homework on data integration (discussed for 10-15 years) have an easier time reaping AI benefits.
Scale is a problem: bigger, more complex landscapes need unified experience.
Security: gap between innovation like open models and deployment. Nobody deploys GitHub models directly. Example: LightLLM vulnerability two weeks ago steals keys and credentials. Critical for chief information security officers.
Teams operating out of SAP a year ago will get rid of mundane work like collecting information, preparing PowerPoints.
Faster, better decisions; automate mundane work. Run more scenarios, get deeper insights faster for strategic thinking.
Analogy: Finance shared service center worker equivalent to junior developer with Cloud Code or Copilot today. They uplevel: supervise code, give feedback, capture essence, review, think about next requirements and differentiation.
Every role uplevels; mundane work pushed to agents. People achieve much more with intelligence baked into the system, eliminating unfun tasks.
SAP mostly seat-based licensed today, with exceptions like Concur, Fieldglass, business network.
With AI, moving to consumptive world, then outcome-based like Sierra. Currently hybrid: consumptive with seat elements.
Joint journey with customers not ready for pure consumptive (need predictability, trust outcomes, fear cost explosion). Hybrid meets customers where they are, provides cost control.
Transition pace unclear; wide range of customer opinions, some forward-leaning, others demanding classical models. It's a journey.
Broad market pivot from SaaS/software valuation. Challenge real, affects incumbents differently: winners and losers.
Winners focus on adoption and outcomes for customers. Technology doesn't matter; make it disappear, get outcomes in front of customers. Produce and price for win-win.
SAP architecture flexible: partnerships with all LLMs, invest in differentiating features vs. commoditized tech stack. Bake in enterprise qualities, integration; turn on capabilities instantaneously for quick ROI.
Focus on business outcomes differentiates winners from losers.
Spend most time reviewing progress with teams across layers: database to models to UI. Give guidance, feedback, learn new things, study outside developments.
Do prototypes; while speaking, running command line interface instances, prototyping, trying things out to see what works, then inspire the team.
Keep No Priors: AI, Machine Learning, Tech, & Startups in your library
Save the videos and channels worth coming back to, and find them again in one place.
