Build, Buy, or Orchestrate?
The executive guide to AI sourcing in the age of Generative AI – and why 95% of enterprise initiatives fail to deliver measurable ROI
At Korra, we deploy on-premises AI knowledge systems into environments where data cannot leave the building. Semiconductor manufacturing facilities, field service organizations with strict IP constraints, operational environments where a cloud dependency is a disqualifier, not a tradeoff.
From that position, we watch the same failure repeat across industries: a leadership team sees an AI demo, approves a build, and eighteen months later the system is not in production. This report draws on MIT Project NANDA (July 2025), RAND Corporation, and Gartner to explain why – and to offer a practical architectural framework for making better sourcing decisions. It is written for technology executives, not data scientists.
The Prototyping Illusion
Why AI accelerated the build decision – and why that acceleration is dangerous
A maintenance engineer at a tier-1 semiconductor equipment company spent three weeks building an internal RAG chatbot. It answered questions from the service manual beautifully. His manager saw the demo. The project got a budget and a team of four. Eighteen months later, the tool was still not in production, and two of those engineers had moved on to other work.
That story is not unusual. It is, right now, the modal outcome for enterprise AI projects.
AI coding tools have made it genuinely fast to build a working prototype. A skilled engineer can have a retrieval system pulling answers from a document corpus in a long weekend. That speed is real. The trap is what the speed hides: the gap between a demo and a production system is not a short walk. It is a different kind of work entirely.
The first 80% of a build – the part that works in the demo – accounts for roughly 20% of total lifetime engineering cost. The remaining 20% of work consumes the other 80%.
This is not a motivational problem. It is a structural one. The demo works because you controlled the inputs. Production does not let you control the inputs.
The RAG Iceberg Effect
Retrieval-Augmented Generation (RAG) is the foundation of any AI system that answers questions from internal documents. A RAG prototype is satisfying to build. You chunk some PDFs, embed them, wire up a vector store, call an LLM, get answers. Two or three days. It feels like most of the problem is solved. It is not.
The demo worked because your documents were clean. In production, documents arrive scanned at odd angles, contain nested tables that standard chunking destroys, are versioned in ways your retrieval index does not track, and change faster than a weekly re-index cycle can handle. The system that manages all of that reliably – with role-based access controls, continuous evaluation, cost optimization, and auditability – is the iceberg under the waterline. The demo is just the tip.
The industry data points the same way: Gartner predicted that 30% of GenAI projects would be abandoned after proof of concept by the end of 2025, citing poor data quality, inadequate risk controls and escalating costs.
Surface – The 80% Prototype (Days to Build)
- Prompt crafting and basic document chunking
- Connecting document files to an LLM API
- Basic semantic search and answer retrieval
- A working demo on curated, clean documents
Below the Surface – The Final 20% (~80% of Lifetime Cost)
- Scanned PDFs, nested tables, embedded charts, non-standard formats
- Document freshness pipelines – keeping vector indices updated as sources change
- Tenant isolation and role-based access control (RBAC) across user segments
- Continuous evaluation harnesses for retrieval precision, semantic drift, factual correctness
- Token and vector cost optimization – unoptimized RAG can scale to six-figure monthly API fees
- Non-deterministic failure handling and fallback logic for LLM errors
- Security, compliance, and data residency controls
Build to Learn vs. Build to Run
There is a useful distinction that most organizations miss: “Build to Learn” versus “Build to Run.” A prototype is a disposable requirements document. It tells you what the real system needs to do – who the users are, what they actually ask, where the edge cases live. That is genuinely valuable.
The mistake is treating the prototype as the foundation of the production system. The code that made the prototype satisfying to build is usually the wrong structure for a system that has to run reliably for three years. Moving from prototype to production requires a formal gate review – validating security architecture, observability, evaluation coverage, and long-term maintenance funding – not just continued iteration on the demo.
Key Takeaways – Part 1
- AI tools create fast demos; they do not reduce production complexity
- The final 20% of production-readiness accounts for roughly 80% of lifetime engineering cost
- Many GenAI projects never make it past the proof of concept – the demo is the easy part
- Treat prototypes as disposable – never use them as the foundation for a production build
The Failure Data
What the research actually shows – and where the failures really come from
The numbers are not ambiguous. Ninety-five percent of enterprise GenAI initiatives have yielded zero measurable return on investment, according to MIT Project NANDA’s July 2025 study, based on a review of more than 300 publicly disclosed AI initiatives, 52 structured interviews and 153 senior-leader surveys. The global spend being evaluated: $30 to $40 billion.
RAND Corporation puts the overall AI initiative failure rate above 80% – roughly double the historical failure rate for non-AI IT projects. Gartner lands in similar territory: it predicted that 30% of GenAI projects would be abandoned after proof of concept by the end of 2025, and its 2026 Hype Cycle for Generative AI warns that at least half of GenAI projects will overrun their budgets due to poor architectural choices and lack of operational know-how.
| Study / Institution | Key Finding | Primary Root Cause |
|---|---|---|
| MIT Project NANDA (July 2025) | 95% of GenAI deployments show zero measurable P&L return | Static systems failing the “Learning Gap”; weak integration with daily workflows |
| RAND Corporation (2024) | 80%+ failure rate – 2× higher than traditional IT projects | Strategic misalignment between leadership expectations and operational reality |
| Gartner (2024) / Gartner Hype Cycle (2026) | 30% of GenAI projects abandoned after PoC by end of 2025; 50%+ expected to overrun budgets | Poor data readiness, unmanaged operational costs, absence of risk controls |
| MIT Project NANDA – sourcing | Vendor partnerships succeed ~67% of the time; internal builds about one-third as often | Underestimation of ongoing maintenance, model drift, and integration TCO |
The Real Root Cause: Not the Models
The natural assumption is that the AI models are not good enough. That assumption is wrong. In most failed deployments, the models are more than capable of performing the task. The failure happens before the model ever runs a query on real user data.
MIT Project NANDA names the structural culprit the “Learning Gap.” Most internal enterprise AI builds rely on rigid prompt pipelines. They do not retain feedback from users. They do not adapt as workflows evolve. They were configured against the data that existed at deployment and cannot update dynamically as that data changes. When these static systems encounter real enterprise data – siloed, inconsistently formatted, updated constantly – output quality degrades, users stop trusting the system, and adoption collapses.
Purchased AI tools and vendor partnerships succeeded about 67% of the time. Internal builds succeeded only one-third as often.
The Cost Trajectory of Internal Builds
Internal builds look deceptively affordable at the start. A working demo emerges in days. Leadership sees that and concludes the hard work is done. It is not. The demo represents roughly 80% of the surface problem. The remaining 20% – compliance adjustments, model monitoring, pipeline refactoring, data quality remediation, integration maintenance – consumes 80% of the lifetime budget.
Measured over an 18-month window, the total cost of ownership of a custom internal build routinely runs several times that of a commercial solution. The drivers are predictable: model drift requires continuous monitoring and retraining. Enterprise data environments change. Regulatory requirements evolve. Each of these generates compounding maintenance obligations that were not in the original budget.
Vendor-partnered implementations succeed at three times the rate not because the vendor’s model is fundamentally better, but because the vendor has absorbed those compounding costs across many deployments. The failure modes are known. The architecture has been stress-tested. The 67% success rate reflects amortized expertise, not superior technology.
Key Takeaways – Part 2
- 95% of enterprise GenAI investments yield zero measurable ROI – across $30-40B in global spend
- The failure driver is not model capability – it is data unreadiness and static architectures
- Internal builds succeed only one-third as often as vendor-partnered implementations
- Once maintenance is included, internal builds typically cost several times more than commercial solutions over 18 months
Own or Orchestrate?
The architectural framework that replaces the build vs. buy binary
The binary choice between building and buying is no longer the right question. It was designed for a world where software was either proprietary or packaged. The modern AI stack has a third option: rent infrastructure, buy operational tooling, and focus internal engineering on the layer where your competitive advantage actually lives.
The right question is not “build or buy?” It is: what do we need to own, and what can we orchestrate?
The Three-Layer Framework
Utility commodity. Building proprietary base models or purchasing on-premises GPU clusters is the AI equivalent of building your own power plant. Rent compute on demand via managed cloud APIs. Base foundation models – whether commercial frontier models or open-weight architectures – should be rented, not owned. In air-gapped environments, cloud rental is off the table, but the logic holds: buy standard hardware and use open-weight models. You still buy – you just don’t rent.
Model routing gateways, evaluation pipelines, prompt versioning, vector indexing, semantic caching, security firewalls. Buy an established platform here unless you are operating five or more complex production AI models simultaneously. Below that threshold, building custom LLMOps generates negative ROI in virtually every case.
Proprietary business logic, domain-specific workflows, specialized agentic task sequences, and the interfaces your users actually touch. This is where competitive differentiation lives. Internal engineering capacity should concentrate here, building on top of purchased infrastructure and platform tooling – not reinventing the layers below.
The Sourcing Decision Matrix
| Approach | Typical Use Cases | Rationale | Recommendation |
|---|---|---|---|
| BUY Off-the-shelf SaaS | IT helpdesks, HR portals, standard customer support, generic document search | Vendors amortize document parsing, compliance, and evaluation costs across many clients | Buy for immediate time-to-value and fixed subscription costs |
| WRAP / HYBRID Managed infra + custom app | Unique workflows needing custom logic built on standard retrieval infrastructure | Procure managed vector databases and platform APIs; focus internal effort on custom workflow integration only | Wrap managed platform APIs with custom business logic |
| BUILD Custom ground-up architecture | Retrieval IS the core product; structure-heavy formats no vendor handles | In-house builds justified when standard retrieval fails or proprietary pipelines drive primary market defensibility | Build from scratch only when retrieval capability is your primary competitive differentiator |
The Three-Question Executive Test
Before approving any custom AI development, apply these three questions:
The Legitimate Cases for a Custom Build
One condition genuinely justifies building from scratch at the infrastructure or platform layer: your retrieval capability is literally your product. If the quality of how you index and surface information is what you sell, outsourcing it hands your core IP to a vendor.
Specialized data formats are often cited as a second reason, but usually they are not. Build the connectors and the translation layer to standard formats in-house, and buy the rest – that is orchestration, not a ground-up build. The exception is when the meaning lives in the structure itself – BOM hierarchies, CAD geometry, revision relationships, time-ordered equipment logs – and flattening it destroys what users are searching for. Even then, look first for a vendor whose retrieval understands that structure. Only when none exists is a deeper build justified.
Data residency is not on the list either. An air-gap requirement rules out cloud APIs, not vendors. It makes the iceberg bigger – you now own GPU sizing, offline model updates, patching and evaluation without external tooling – which makes an internal build harder, not more justified. The right move is to buy from the small set of vendors built to run inside your walls.
Outside that one condition, the instinct to build layers one or two is almost always an engineering preference dressed as a strategic argument.
What is worth building is the logic on top. Your LLM, vector database and search engine are the same as everyone else’s. Knowing that alarm E-4021 on a Gen-3 chamber at a specific fab means “check the RF match first” – and that only engineers under certain contracts may see that fix – is logic only your company has. That is what to build.
Key Takeaways – Part 3
- The build vs. buy binary is broken – the right question is: what do we need to own vs. orchestrate?
- Infrastructure and base models: always rent. Platform/LLMOps: buy unless you run 5+ production models. Application logic: build or orchestrate tightly
- The one real reason to build from scratch: retrieval IS your product. Specialized formats need custom connectors, not a custom platform. Air-gap narrows the vendor list – it is not a reason to build
- Apply the three-question test before approving any custom AI development
Conclusion
Generative AI has disrupted classical software procurement logic – not by simplifying the decision, but by making the wrong decision easier to make. AI coding tools create impressive demos in hours. Those demos create a false signal that production is a short step away. It is not.
The path forward is architectural rather than binary. Rent infrastructure. Buy or hybrid the operational layer. Build or orchestrate tightly at the application layer where differentiation actually lives. Fix data readiness before touching model selection. Prototype aggressively but treat every prototype as disposable – it is a requirements document, not a foundation.
The organizations that capture sustained value from AI are not the ones that build the most. They are the ones that own the right things and orchestrate everything else. The 67% success rate for vendor-partnered implementations versus roughly 22% for internal builds reflects that discipline, not the quality of the underlying technology.
For environments where data genuinely cannot leave the building – where air-gap is a requirement, not a preference, and cloud dependencies are disqualifiers rather than tradeoffs – the framework still holds, but the vendor list gets much shorter. That is the specific problem Korra was built to solve: AI knowledge systems that operate fully on-premises, so air-gapped teams can buy instead of build.
