How Onyx's RAG Engine Cuts Token Usage at Enterprise Scale (2026)

There are two ways an AI system can get enterprise context: read through the sources at query time, or query an index that was built ahead of time. The first approach pushes every document an agent touches through the LLM's context window, on every task, at full token price. The second pays for document understanding once, at indexing time, and answers every subsequent query with a retrieval call that costs a few thousand tokens instead of a few hundred thousand.

This is the difference between an agent that opens Confluence, pages through Slack, and reads ten documents to find one paragraph, and an agent that asks a RAG engine and gets the paragraph back. A June 2026 study measured the gap at 26x per-query token cost for document-grounded question answering, and for multi-step agentic workflows the gap compounds with every step. This guide explains how Onyx's RAG engine is architected around that difference: what the indexing pipeline does to each source, how the retrieval pipeline is designed for cheap and precise context, and how Onyx Craft agents tap the same engine mid-workflow instead of combing through sources themselves.

---

TL;DR: Reading sources through an LLM is the most expensive way to use them. Onyx indexes content from 40+ sources once, condenses it into a hybrid vector index (OpenSearch, dense embeddings + BM25), and serves any question from a few thousand tokens of ranked, permission-filtered passages. Retrieval itself costs zero LLM tokens. The same engine backs chat, search, deep research, and Craft agents: instead of manually combing through Slack, Drive, and Confluence with tool calls, a Craft agent invokes the company-search skill and gets grounded enterprise context in one cheap call. With per-token prices roughly flat in 2026 and inference consuming ~85% of enterprise AI budgets, this architecture is the difference between AI spend that scales with query volume and AI spend that explodes with it.

Pricing and market data verified against primary sources as of July 2026.

---

Why Token Efficiency Matters More in Mid-2026

For two years the advice was "don't optimize token costs, prices fall faster than you can engineer around them." That advice is out of date. Effective API token pricing declined only 6% in the first five months of 2026, enterprises keep routing spend to premium frontier models, and providers now charge explicit long-context premiums: Gemini 3.1 Pro doubles input pricing above 200K tokens, and GPT-5.5 jumps from $5 / $30 to $10 / $45 per million in the long-context band.

Meanwhile usage is exploding. Agentic workflows trigger 10-20 LLM calls per task, and inference now accounts for roughly 85% of enterprise AI budgets. Every one of those calls carries context, and where that context comes from is the architecture decision that sets your bill. You cannot negotiate your way out of it. You have to send fewer tokens, and that means retrieving context instead of reading for it.

The Expensive Way: Combing Through Sources at Query Time

Consider what "the agent finds the answer itself" actually costs. An agent asked to summarize the current state of a customer account, without a RAG engine, has to behave like a very fast, very expensive intern:

1. List recent Slack channels and read the plausible threads: tens of thousands of tokens.
2. Search Drive, open three candidate docs, read each one through the context window: 15K-50K tokens apiece.
3. Pull the CRM record, the last support tickets, the related Jira issues: more reads.
4. Reason over all of it, most of which turns out to be irrelevant.

Every document read is billed at full input-token price, the reads repeat on every task (nothing is remembered between queries), and each additional source multiplies the calls. Research on context relevance shows why this is structurally wasteful: when less than roughly 20% of provided context is relevant to the question, selective retrieval beats reading everything on both cost and reliability, and enterprise sources are almost always far below 20% relevance for any single question. Worse, more context actively degrades answers past a point: 48K well-chosen tokens outperform 117K tokens of full context by 13 F1 points, because relevant passages get diluted.

The manual-combing pattern pays the highest possible price for the lowest possible precision. The fix is to move the reading out of query time entirely.

What Is Onyx?

Onyx is an open-source (MIT-licensed) enterprise AI platform built around a RAG engine. 40+ connectors sync content from Slack, Confluence, Google Drive, SharePoint, Jira, GitHub, Salesforce, Zendesk, Gmail, and more into a hybrid OpenSearch index, kept current in near real time along with each document's metadata and access permissions. On top of that engine, Onyx serves permission-aware search, AI chat, multi-step deep research, and agents, with any LLM (OpenAI, Anthropic, Google, DeepSeek, Llama, Mistral, Qwen) via cloud APIs or local inference through Ollama, vLLM, or LM Studio.

It runs in production at Ramp (115K queries per month), Thales (82,000-employee organization), and UC San Diego (37K+ users, fully air-gapped), among 1,000+ enterprise customers. Everything below describes the pipeline those queries flow through.

The Indexing Pipeline: Condense Every Source Once

Onyx's answer to the combing problem starts before any question is asked. Each connector understands its source natively: a Slack connector pulls threads with their reply structure, a Confluence connector pulls pages with their hierarchy and labels, a Drive connector pulls documents with their sharing settings. From there, every piece of content flows through the same pipeline:

1. Extraction. Text is pulled out of the source format (pages, threads, tickets, PDFs) along with metadata: author, timestamps, source, tags.
2. Chunking. Documents are split into retrieval-sized passages so a query can pull back one relevant section instead of a whole document.
3. Contextual enrichment. Chunks are augmented with surrounding context so a passage remains meaningful when retrieved alone, which is what lets a 500-token chunk stand in for a 20K-token document.
4. Embedding and indexing. Each chunk is embedded into a dense vector and written to OpenSearch alongside a keyword (BM25) index and its permission entries from the source system.

The economics of this pipeline are the whole point:

• Understanding is paid once per document version. A 40-page spec is processed at index time and then serves every future query against it. In the combing model, that spec's tokens are re-billed every time an agent opens it.
• Embedding is not frontier-priced. Embedding models cost a small fraction of generation tokens, and self-hosted deployments run them locally at zero marginal API cost. UC San Diego runs the entire pipeline on its own GPUs.
• Sync is incremental. Connectors pull deltas in near real time, so keeping the index current means reprocessing what changed, not the corpus.

The result is your entire knowledge footprint, condensed into a form where finding the right 5K tokens is a database operation, not an LLM operation.

The Retrieval Pipeline: Precise Context, Zero LLM Tokens

At query time, Onyx's retrieval pipeline is designed to hand the LLM the smallest context that answers the question:

• Hybrid search. Dense vector similarity catches semantic matches ("how do we handle refunds" finds the returns policy); BM25 keyword search catches exact terms, acronyms, error codes, and project names that embeddings blur. The two result lists are fused, which matters on enterprise corpuses full of identifiers that pure vector search mishandles.
• Reranking. A small, focused reranking model reorders the top candidates by deeper relevance, so the chunks that reach the LLM are the best ones available, not just the closest vectors.
• Filters. Time ranges, authors, tags, and source types narrow the candidate set before ranking, so "what did engineering decide last quarter" searches the right slice of the index.
• Permission-aware filtering. Access permissions synced from each source are enforced at retrieval time, not at the UI. Users and agents only ever retrieve what they are allowed to read, and no tokens are spent generating over content that would have to be discarded.

Everything above happens in OpenSearch and a reranker. No generation-model tokens are spent finding context. The LLM receives a few thousand tokens of ranked, cited, permission-filtered passages and does the one thing only an LLM can do: compose the answer.

Precision here is itself a token optimization. When the right passage is in context on the first attempt, users do not re-ask the question three ways and agents do not loop. Onyx's published benchmarks show a 64-76% win rate on workplace-question quality versus ChatGPT, Claude, and Notion AI on a 220K-document corpus, and Thales saw the retrieval difference directly when comparing platforms on the same model: "The result given is better, even when they use the same underlying model."

The Token Math: One Retrieval Call vs Combing the Sources

Illustrative per-task economics at mid-2026 frontier pricing ($5 per million input tokens standard band):

Numbers are illustrative and depend on corpus, model, and task mix. The 26x per-query gap measured in controlled research sits inside this range, and multi-step agent workflows multiply it by the number of steps that need context.

Agentic Workflows: Craft Queries the Engine Instead of Combing

The retrieval-vs-combing distinction matters most for agents, because agents need context repeatedly, mid-task, across many steps. This is where most agent platforms quietly burn budgets: each "look something up" step becomes a fresh round of tool calls and document reads.

Craft, Onyx's agent workspace, is designed the other way. Craft agents run in a secure sandbox where they write code, build documents and web apps, and act in connected tools. When a Craft agent needs enterprise context, it does not open Slack and start reading. It invokes the built-in company-search skill, which gives the agent a CLI into Onyx's RAG engine from inside the sandbox. The call goes through scoped Craft APIs that enforce the signed-in user's document permissions, and what comes back is the same ranked, cited, few-thousand-token context that powers chat and search.

That one design choice changes the cost profile of complex agentic workflows:

• Grounding steps become cheap. Building an all-hands deck, Craft grounds each section with company-search calls: a handful of retrieval queries at a few thousand tokens each, instead of reading every source document the deck draws on through the context window.
• The index is shared infrastructure. Every Craft session, chat query, and deep research run hits the same continuously synced index. No agent ever pays to re-understand a document the pipeline already processed.
• Work stays out of the context window. Craft runs analysis as code in the sandbox, so large datasets flow through executed programs rather than the model. Retrieval brings in the knowledge; code handles the volume.
• Scheduled automations stay flat. A recurring Craft task re-queries the engine for fresh context on each run, so an always-on workflow costs the same per run instead of accumulating context.

The contrast with generic agent stacks is direct. An agent wired to raw MCP tools for Slack, Drive, and Confluence must comb: list, open, read, repeat, paying token price for every page it touches. A Craft agent asks a question and gets an answer, because someone already did the reading, once, at index time.

Recommended Setup by Scale

How to Choose

A RAG engine is the right cost architecture when your AI workload is grounded in a large, changing corpus where any single question needs a small fraction of it, when query or agent-task volume is high enough that per-task costs dominate, and when permissions must be enforced on what AI can retrieve. That describes nearly every enterprise deployment.

Reading full sources at query time is still the right tool in narrow cases: analyzing one contract end to end, reviewing a fixed bundle of files where every part matters. The research is consistent that full context buys some accuracy when relevance density is high. The mistake is letting that pattern become the default for enterprise questions where relevance density is a fraction of a percent, and paying long-context premium rates for it on every task.

Building the pipeline yourself is the option to be most skeptical of: production connectors, permission syncing, chunking, embedding, hybrid retrieval, and reranking are each real engineering projects, and every month of build time is a month of paying combing-model token bills in the interim.

Recommendation

If your AI bill is growing faster than your usage justifies, look at where context comes from. Systems that read sources at query time re-buy the same understanding on every task; systems built on a RAG engine buy it once. Onyx is the strongest fit when you want that engine with control over the rest of the cost structure: open-source (MIT) so you can audit the pipeline, model-agnostic so each query runs on an appropriately priced model, self-hostable so embedding and high-volume inference move off API billing, and one shared index behind search, chat, deep research, and Craft agents so nothing pays twice. Teams with tiny corpuses and low query volumes may not need the machinery; the economics diverge as volume grows.

Frequently Asked Questions

How does a RAG engine reduce token usage compared to an agent reading sources directly?

Retrieval replaces reading. Finding relevant content happens in the search index (OpenSearch hybrid search plus a reranker) at zero LLM token cost, and only 3K-8K tokens of ranked passages reach the model, versus 50K-200K+ tokens when an agent opens and reads documents itself. Controlled research measured semantic RAG at roughly 1/26th the per-query token cost of long-context prompting, and multi-step agent workflows multiply the gap.

What does Onyx's indexing pipeline actually do to my documents?

Connectors pull content from each source with its native structure, then the pipeline extracts text and metadata, chunks documents into retrieval-sized passages, enriches chunks with surrounding context, and embeds them into a hybrid OpenSearch index (dense vectors + BM25) along with permissions synced from the source. This runs once per document version, with incremental near-real-time sync afterward.

Does indexing itself cost a lot of tokens?

No. Indexing uses embedding models, which cost a small fraction of generation-model tokens, and the cost is paid once per document version rather than per query. Self-hosted Onyx deployments run embedding models locally at zero marginal API cost.

How do Onyx Craft agents use the RAG engine?

Through the built-in company-search skill, which gives the sandboxed agent a CLI into Onyx's search APIs. The agent makes a cheap retrieval call and receives ranked, cited, permission-filtered context, instead of combing through Slack, Drive, or Confluence with per-document tool calls. The APIs enforce the signed-in user's document permissions, so agents can only retrieve what that user can read.

Do 1M-token context windows make RAG obsolete?

No. Long context helps when everything in the prompt is relevant. Enterprise questions typically use under 1% of the corpus, where retrieval wins on cost, latency, and accuracy, and providers charge premium rates above 200K tokens (Gemini and GPT-5.5 both roughly double pricing in the long-context band). Research also shows well-chosen retrieved context outperforming much larger full contexts because irrelevant tokens dilute relevant ones.

Can I run the whole pipeline without paying API token costs at all?

Yes. Onyx supports local inference through Ollama, vLLM, and LM Studio for both embeddings and generation, alongside 100+ cloud LLMs. Fully air-gapped deployments run everything on your own hardware; UC San Diego serves 37,000+ users this way.

Is prompt caching a substitute for retrieval?

No. Caching discounts repeated identical context, which helps stable system prompts. Enterprise queries hit different documents per user and per question, and content changes continuously, so caches rarely hit. Retrieval reduces the tokens you send; caching discounts tokens you re-send. Use both.