The RAG Triangle

Master the trade-offs between accuracy, latency, and cost in RAG.

Why Your System Is Always Locked Between Accuracy, Latency, and Cost

If you’ve ever “improved” a RAG pipeline and somehow made it worse, welcome to the club. Some teams approach Retrieval-Augmented Generation like there’s a hidden perfect configuration: the right chunk size, the right vector DB, the right reranker, the right prompt… and then boom — production-grade truth.

Reality is less magical and more like physics: RAG is always a balance between three constraints: accuracy, latency, and cost. Whatever you build is effectively locked inside that triangle. You can move inside it. You can’t escape it.

Why this matters

Because RAG isn’t a demo. It’s infrastructure. The moment your RAG system becomes part of how people work — support, legal, security, IT, sales — your users will judge it on three things:

Is it correct? (Accuracy)
Is it fast enough to use? (Latency)
Is it cheap enough to scale? (Cost) Your roadmap becomes a constant negotiation between these three. And the negotiation is brutal because improvements rarely come for free.

The triangle (and why “best RAG” doesn’t exist)

Think of RAG as a triangle:

Accuracy: grounded, relevant retrieval + correct synthesis
Latency: time-to-answer (especially P95/P99)
Cost: dollars per successful resolution (not per request) You can optimize any one corner aggressively. But the other corners will push back.

Common trade-offs you’ve already seen (even if you didn’t call them that)

Want higher accuracy? You add reranking, query rewriting, multi-step retrieval, more context, more evals → latency and/or cost increase.
Want lower latency? You reduce steps, reduce context, skip reranking, use smaller models → accuracy decreases (or you push cost elsewhere, like caching and preprocessing).
Want lower cost? You cut model calls, tokens, experiments, observability → accuracy degrades silently or latency rises because you need more fallbacks and human review. And if you’re building an agentic RAG flow (multiple steps), the triangle bites harder.

If each step is 80% reliable, a 3-step chain is:

**0.8 × 0.8 × 0.8 = 51%**

That’s not “pretty good.” That’s coin-flip automation with better formatting.

How to choose a point in the triangle (a practical framework)

Stop asking: “What’s the best RAG architecture?” Start asking: “What is the minimum acceptable floor for each constraint in this use case?”

Here’s a simple decision framework you can use in one meeting:

Step 1: Define the floor (your non-negotiables)

Pick minimum acceptable thresholds for each constraint. If you can’t state these floors clearly, you don’t have a RAG problem — you have a product definition problem.

Accuracy floor (pick 1–2 that match your risk):

Unsupported-claim rate: e.g., “≤ 1% answers contain claims not backed by sources”
Grounded answer rate: e.g., “≥ 85% answers include correct supporting evidence”
Abstention behavior: e.g., “abstain/escalate when evidence coverage < X”

Latency floor:

Define P95 (and sometimes P99): e.g., “P95 ≤ 3.5s for support, ≤ 10s for legal”
Also define where latency matters: chat UI vs background job vs batch

Cost floor:

Define cost per successful outcome, not per request: e.g., “≤ $0.08 per resolved ticket” or “≤ $2 per reviewed RFP section”

If stakeholders answer “as good as possible,” force a choice:

“Would you rather be wrong 2% of the time or be 2x slower?” “Would you rather pay 2x or answer 2x slower?”

Step 2: Choose a primary constraint (your “north star”)

Most use cases have one constraint that dominates. Name it explicitly.

Support / IT Helpdesk: Latency is primary (users leave), with an accuracy floor that prevents ticket creation.
Legal / Finance / Healthcare: Accuracy is primary (risk), with latency allowed to be slower.
Sales / Marketing drafts: Cost + latency are primary (volume), with hard guardrails against risky claims.

This step matters because it prevents contradictory requirements like: “instant + perfectly accurate + basically free” (aka: unicorn procurement).

Step 3: Map architecture choices to the triangle (pick your levers)

Now that you know your floors + primary constraint, pick the smallest set of levers that moves you toward that point — without violating the floors.

If you’re optimizing Accuracy + Latency (accept higher cost):

Hybrid retrieval + metadata filters
Reranking (top-K → top-N)
Fast path vs safe path routing
Caching for repeat intents
Confidence gates (ask/abstain instead of guessing)

If you’re optimizing Accuracy + Cost (accept higher latency):

Quote-first answers + strict source hierarchy
Deterministic extraction for numbers/thresholds
Higher abstention thresholds
Deeper evaluation + regression testing
More checks (versioning, conflict detection)

If you’re optimizing Latency + Cost (accept constrained accuracy):

FAQ-first routing + templates
Link/snippet-first answers
Aggressive caching (retrieval + final answers)
Smaller models for routing/classification
Hard token budgets + limited steps

Then lock it in with a simple rule:

If a change improves one corner but violates a floor, it’s not an improvement—it’s a regression.

What actually moves you around the triangle

Don't tweak prompts and hope, that’s like repainting a car to improve its fuel efficiency. The real levers are architectural.

Accuracy levers (you pay with latency/cost)

Better retrieval: hybrid search, metadata filters, domain-specific indexing
Reranking: cross-encoders, learned rerankers
Query rewriting / decomposition: turns vague input into retrievable intent
Chunking based on meaning: sections/clauses, not character counts
Confidence gating + abstention: prevents hallucinations (and forces clarifying questions)
Evaluation harness: you can’t optimize what you don’t measure

Latency levers (you pay with cost/accuracy if careless)

Routing: fast path vs safe path (FAQ vs deep retrieval)
Caching: embeddings, retrieval results, final answers
Precomputation: nightly summaries, pre-ranked bundles per intent
Token budgets: strict caps + better chunk selection
Reduce model calls: fewer steps, avoid agent chains unless justified

Cost levers (you pay with accuracy if naive)

Smaller models for the cheap parts: routing, rewriting, classification
Extraction over generation: for numbers, thresholds, policy values
Context compression: retrieve less, better; summarize offline
Reuse: approved answer libraries, canonical snippets
Measure cost per successful outcome: not per request

The hidden killer: “optimizing” without evals

If you’re not running systematic evals, you’re not optimizing. You’re just changing things and playing experimentation roulette. RAG systems degrade for reasons that are invisible day-to-day:

new documents shift retrieval
chunking changes alter recall/precision
a reranker update changes what “top” means
a prompt tweak causes longer answers → higher cost → latency spikes
permissions filters reduce retrieval and break coverage Without evals, you discover failures in production — where the cost is trust.

Minimum viable eval setup:

50–200 real questions from your users
expected answers or at least expected sources
metrics tracked weekly (see below)
regression tests after any pipeline change

Metrics that keep the triangle honest

Pick metrics that correspond to each corner: Accuracy (grounded correctness)

Grounded answer rate: % answers supported by retrieved sources
Source coverage: did you retrieve the right section/doc?
Hallucination / unsupported-claim rate
Abstention quality: abstain when you should

Latency (real user experience)

P95/P99 end-to-end latency
Time-to-first-token and time-to-final
Latency broken down by stage: retrieval, rerank, generation

Cost (scalability)

Cost per resolved conversation/task
Tokens per successful outcome
Model-call count per request
Cache hit rate (this is often your biggest lever)

We plan to provide typical “triangle profiles” by use case in later posts.

The punchline: you can’t escape the triangle, but you can stop suffering

Don’t lose months chasing a mythical “best RAG.” The winning teams do something more boring — and far more effective:

Define the point in the triangle for a specific use case
Choose the levers that move them toward that point
Measure continuously with evals and operational metrics
Iterate without breaking production trust RAG isn’t a single system. It’s a set of trade-offs you make explicit — and then automate. Because if you don’t choose your trade-offs deliberately, your users will choose them for you… by not using the product.

A simple closing rule you can steal

You can optimize two. The third will fight back. So pick your two, set floors for the third, and build accordingly. And if someone claims they built a RAG that’s perfectly accurate, instant, and basically free, ask them one question: “Cool — what did you stop measuring?”