The demo took a week. The production version has been three months and counting, and not one hour of that was spent on the model.
This is the most predictable pattern in enterprise AI. The intelligence is bought. The data plumbing is built, and nobody budgeted for it.
Where the Time Actually Goes
| Task | Share of effort | Why it surprises people |
|---|---|---|
| Access and permissions | Large | Every system has a different owner and a different answer |
| Extraction and normalisation | Large | Documents are far messier than sample sets |
| Chunking and metadata | Underestimated | This decides retrieval quality more than the model does |
| Freshness and sync | Ongoing | Not a project, a permanent commitment |
| Permission propagation | Hard | Users must only see what they are allowed to see |
| Model integration | Small | The part everyone plans for |
The Permission Problem
This is the one that kills projects late, which is the worst time.
Your source systems have granular permissions. Your index probably does not. Build a retrieval system without propagating those permissions and you have created an elegant way for anyone to read anything, discoverable by the first curious employee who asks about salaries.
Design permission filtering into retrieval from the first sprint. Retrofitting it means rebuilding the index, and usually the schema with it.
Chunking Deserves More Respect
Teams spend weeks evaluating models and an afternoon on chunking, then wonder why answers are poor.
How you split documents determines what can be retrieved at all. Split mid-table and your numbers become meaningless. Split too small and context is lost. Split too large and precision collapses.
The practical advice is unromantic: chunk along the document natural structure rather than by character count, keep tables intact, carry section headings into each chunk as metadata, and test with real questions before you scale the pipeline.
Freshness Is a Product Decision
Someone will ask how current the data is, and the honest answer is often worse than anyone assumed.
Decide the acceptable staleness per source and build to it. Pricing might need to be minutes fresh while policy documents can be a week behind. Then show the freshness in the interface, because an answer that quietly relies on last quarter data is worse than no answer, and considerably harder to detect.
How to Scope It Honestly
- Count your sources and multiply. Each new source is not incremental. It brings its own auth, format and owner.
- Do the permission design in week one. Not week ten.
- Build the evaluation set before the pipeline. Fifty real questions with known good answers, so changes can be judged rather than debated.
- Start with one source, end to end. A complete narrow pipeline teaches you more than three half-finished broad ones.
- Budget the maintenance. Sources change shape. This is a running cost, not a build cost.
Conclusion
Assume the data work will take several times longer than the model work, because it reliably does. Design permission propagation in the first week, treat chunking as a first-class engineering decision rather than a default setting, and set explicit freshness targets per source. Build one source end to end before adding a second. The teams who ship are not the ones with better models. They are the ones who took the plumbing seriously in month one.
Frequently Asked Questions
Can we skip the data work with a bigger context window?
For small corpora, sometimes. It fails on cost, latency and quality as soon as your content grows, and it does nothing about permissions.
How do we handle documents that change constantly?
Incremental sync with change detection, and a freshness indicator in the interface. Full reindexing on a schedule is simpler and gets expensive quickly.
Who should own this work?
Data engineering, with security involved from the start. Handing it to application developers is how permission propagation gets discovered in a penetration test.