Self-Hosted RAG in 2026: The Complete Guide to Running Retrieval-Augmented Generation On Your Own Infrastructure

---

TL;DR: Self-hosted RAG means running the entire retrieval-augmented generation pipeline (ingestion, embeddings, vector index, retrieval, reranker, LLM, and UI) inside your own network instead of a vendor's cloud. The motivation is some mix of data sovereignty, compliance, model freedom, and lower long-run cost. The 2026 landscape has split into three layers: open-source RAG platforms with a finished UI and connectors (Onyx, RAGFlow, AnythingLLM, Verba, Open WebUI, LibreChat), RAG frameworks you assemble into an app (LlamaIndex, LangChain, Haystack), and pure search infrastructure (Elastic, OpenSearch, Weaviate, Qdrant, Milvus). Most teams that want a workforce-facing AI search and chat product should start at the platform layer. This guide covers what self-hosted RAG actually involves, the options at each layer, recommended stacks for small, mid-market, regulated, and air-gapped deployments, and how to decide which path fits your team.

---

What Is Self-Hosted RAG?

Self-hosted RAG is the deployment pattern where every component of a retrieval-augmented generation system runs on infrastructure you control: your servers, your cloud account, your VPC, your air-gapped network, or your laptop. No documents leave your environment for embedding, no queries get sent to a third-party API, and no vendor manages the runtime.

A complete self-hosted RAG stack has seven components:

1. Connectors or ingestion pipelines that pull documents from Slack, Confluence, Google Drive, SharePoint, Jira, GitHub, file shares, or databases.
2. A document processing layer that parses files (PDFs, Office docs, HTML), chunks them, and extracts metadata.
3. An embedding model that converts chunks into vectors. Self-hosted RAG usually runs this locally with models like BGE, GTE, Nomic Embed, or E5.
4. A vector index (Qdrant, Milvus, Weaviate, PGVector, OpenSearch, Elasticsearch, Chroma) that stores embeddings and supports approximate nearest neighbor search, ideally alongside a keyword index for hybrid retrieval.
5. A retrieval and reranking pipeline that combines vector and keyword search, often with Reciprocal Rank Fusion, then reorders top candidates with a reranker such as BGE Reranker v2 or Cohere Rerank (Cohere offers an on-prem licensing path).
6. An LLM for generation, served by an inference engine you run yourself: Ollama, vLLM, SGLang, LM Studio, llama.cpp, or TGI.
7. A user interface and access layer with chat, search, citations, SSO, RBAC, and (ideally) permission inheritance from the source systems.

Most "local RAG" tutorials stop at step six with a single-script demo. Self-hosted enterprise RAG ships all seven, including the parts that take real engineering: connector sync, ACL propagation, evaluation, observability, and operations.

Why Teams Self-Host RAG

Self-hosting is more work than calling a managed RAG API. The teams that do it usually have at least one of these reasons:

• Data sovereignty and compliance. Defense (ITAR, CMMC), healthcare (HIPAA), public sector (FedRAMP), education (FERPA), and EU-regulated workloads often cannot send embeddings or queries to third-party clouds. Air-gapped deployments are the strictest version: zero outbound traffic, all models and indexes on-prem. UC San Diego runs Onyx fully air-gapped on local GPUs for 37,000+ users.
• Model freedom. A self-hosted stack lets you swap the underlying LLM as the frontier moves. Open weights have closed most of the quality gap in the past 18 months: Qwen3, DeepSeek V3, Llama 4, and Mistral models routinely score within range of frontier closed models on enterprise tasks, and they can be deployed on commodity GPUs.
• Predictable cost at scale. Cloud RAG services bill on indexed tokens, query volume, embedding calls, generation tokens, and storage. A 200-user deployment can swing from $30K to $300K/year on the same workload depending on the tier and provider. Self-hosted economics are dominated by GPU amortization, which typically breaks even against API costs in 6 to 12 months for moderately active teams.
• Auditability. Open-source RAG platforms let security teams read the code that handles retrieval, ACL filtering, and prompt construction. Closed cloud RAG products do not.
• Customization. Self-hosted stacks can be extended with custom connectors, custom rerankers, custom chunking strategies, and domain-specific evaluation pipelines without waiting for a vendor roadmap.

The trade-off is real: self-hosted RAG requires Docker or Kubernetes operational knowledge, GPU procurement or rental, and ongoing maintenance. That cost is acceptable to most enterprises with one of the drivers above, and a non-starter for teams that just want a private ChatGPT for 20 people.

What Is Onyx?

Onyx is an open-source (MIT-licensed) enterprise AI platform with self-hosted RAG built in. It bundles 40+ connectors, hybrid search over an OpenSearch-backed vector index, reranking and LLM-based knowledge graphs, permission-aware retrieval, AI chat, multi-step deep research, and custom agents with MCP tool use into a single deployment.

For a self-hosted RAG buyer, the relevant Onyx surface is:

• Deployment options: Docker Compose for small teams, Kubernetes Helm chart for production, Terraform modules for AWS/Azure/GCP, and fully air-gapped deployments with no internet dependency.
• Model choice: any LLM via LiteLLM, with first-class support for local inference through Ollama, vLLM, SGLang, and LM Studio. The same goes for embedding models, which can run locally or against a hosted endpoint.
• Permission inheritance: ACLs sync from each source system into the index, and filtering happens pre-retrieval rather than at the chat-UI layer.
• Extensibility: REST APIs, MCP server, agent SDK, and an embeddable chat widget. Customers like Ramp have built customer-facing AI features directly on Onyx.

Onyx is in production at Ramp (115K queries/month, 30-50x reported ROI), Thales (1,400 MAU across 82,000 employees with on-prem Kubernetes), Astranis (fully self-hosted inside the network), L3Harris, NASA, and UC San Diego (37K+ users, air-gapped, local GPUs), among 1,000+ enterprise customers.

How Self-Hosted RAG Actually Works

Three architectural decisions separate a serious self-hosted RAG stack from a weekend prototype.

Hybrid Retrieval (Vector + BM25)

Pure vector search fails on exact identifiers, acronyms, version numbers, and rare phrases. The production pattern combines dense vector search (semantic similarity) with sparse keyword search (BM25 or BM25F) and fuses the result lists, usually with Reciprocal Rank Fusion. Practitioner benchmarks consistently show hybrid retrieval delivering 15 to 30 percent better recall than either approach alone on enterprise corpuses. VentureBeat reported that enterprise intent to adopt hybrid retrieval tripled from 10.3% to 33.3% in a single quarter of 2025 as RAG programs hit the scale wall.

OpenSearch, Elasticsearch, Weaviate, Qdrant (with sparse-dense fusion), and Vespa all support hybrid retrieval natively. PGVector pairs cleanly with PostgreSQL full-text search for the same effect.

Reranking

After hybrid retrieval returns the top 100 to 200 candidates, a reranker reorders them by deeper relevance. The most-deployed self-hostable options are BGE Reranker v2, Jina Reranker, and Cohere Rerank 4 (which has on-prem licensing). Databricks reported a 15-percentage-point retrieval accuracy improvement after adding reranking to Mosaic AI Vector Search. ColBERT-style late-interaction retrievers offer a different trade-off: higher accuracy at higher index size and operational cost.

Permission-Aware Retrieval

This is the part most self-hosted RAG projects underestimate. Enterprise RAG must filter documents by who is allowed to read them, and the filter has to happen pre-retrieval. The two patterns are early binding (sync ACLs into the index, filter at query time) and late binding (retrieve broadly, filter post hoc). Late binding is brittle: a single missed filter and confidential content leaks into an LLM context window, where prompt injection or summary leakage can surface it.

Doing this correctly requires syncing access control lists from every connected source, propagating them into the vector and keyword indexes, and enforcing them on every query. Most DIY RAG stacks wave at "permissions" and only enforce them at the UI layer. That is not the same product.

Why This Matters

RAG reduces hallucinations but does not eliminate them. The 2024 Stanford RegLab study, published in 2025, found that production legal RAG systems still hallucinated on 17% to 33% of queries. They were still meaningfully better than a base LLM, but not error-free. Evaluation, citation, and (for high-stakes outputs) human review belong in the stack alongside retrieval and generation.

What to Look For in a Self-Hosted RAG Stack

Use these criteria in any platform-layer evaluation. They are ordered by how often they end up being the deciding factor in real deployments.

• Deployment surface. Docker Compose, Helm chart, Terraform, AWS/Azure/GCP marketplace listings, true air-gapped support.
• Connector breadth and depth. Native connectors for the SaaS apps your team uses, with continuous sync and permission preservation from the source.
• Permission inheritance. ACL syncing is hard. Verify the platform filters at retrieval time, not at the UI.
• Retrieval quality. Hybrid search, reranking, and ideally graph or agentic retrieval. Ask for evaluation results on your data.
• Model and inference flexibility. Any LLM via Ollama, vLLM, SGLang, LM Studio, or hosted API. Ability to swap embedding and reranker models.
• License. True open source (MIT, Apache 2.0) versus restrictive custom licenses that change pricing or branding requirements at certain scales.
• Generation surface area. Search-with-answers only, or chat, deep research, agents, and MCP tool use as well.
• Security and compliance. SSO (OIDC, SAML), SCIM, RBAC, audit logs, SOC 2, GDPR, and an answer for FedRAMP, ITAR, CMMC, FERPA, and HIPAA where relevant.
• Operational maturity. Helm chart quality, upgrade paths, observability hooks, evaluation tooling, and an active community or support contract.

The Self-Hosted RAG Landscape: Comparison Table

License notes: Open WebUI moved from BSD-3 to a restrictive custom license in 2025 that requires paid branding for 50+ user deployments. Elastic restored AGPLv3 in September 2024 alongside SSPL. Verify the current license at evaluation time.

Detailed Reviews

1. Onyx

What it is: A full-stack open-source self-hosted RAG platform combining enterprise search, AI chat, deep research, and custom agents in one product.

RAG architecture: Hybrid search (vector + BM25) over an OpenSearch-backed index with reranking, contextual chunking, multi-pass indexing, and optional Agent Search using StructRAG techniques. LLM-based knowledge graphs add cross-document context. Permission inheritance is enforced at retrieval time, not at the UI.

Connectors: 40+ native connectors with continuous sync and permission syncing: Slack, Confluence, Jira, Google Drive, SharePoint, Salesforce, GitHub, Notion, HubSpot, Zendesk, Linear, Gmail, Outlook, and more.

Models: Any LLM via LiteLLM. Local inference through Ollama, vLLM, SGLang, LM Studio. Local embedding models (BGE, E5, Nomic) or hosted endpoints. Local reranker support.

Deployment: Docker Compose, Kubernetes (Helm), Terraform on AWS, Azure, and GCP. Fully air-gapped deployments supported with local LLMs and zero outbound traffic.

Security: SOC 2 Type II, GDPR, ISO 27001 in progress. SSO via OIDC and SAML, SCIM, granular RBAC, audit trails. Deployed in FedRAMP, ITAR, CMMC, and FERPA environments.

Pricing: Free community edition (MIT). Cloud Business plan at $20/user/month (annual). Enterprise tier with tiered per-seat pricing for self-hosted enterprise installs (SSO, advanced RBAC, white-labeling, dedicated support).

Customer proof: 30K+ GitHub stars. 1,000+ enterprise customers including Ramp, Brex, Thales, L3Harris, NASA, Astranis, Roku, and UC San Diego.

The Head of Engineering at Thales, running both Microsoft Copilot and Onyx in production, said: "People are using both Copilot and Onyx. In the end, they are very happy with Onyx. The result given is better, even when they use the same underlying model."

Best for: Workforce-facing self-hosted RAG deployments where connector breadth, permission inheritance, model freedom, and air-gapped support all matter. Engineering-led teams that want extensibility (custom connectors, REST APIs, MCP server, agent SDK, embeddable widget).

Trade-offs: 40+ connectors is fewer than Glean's 100+ (Glean is not self-hostable in an open sense, but it is the connector benchmark). Self-hosted deployment requires Docker or Kubernetes operational knowledge. The managed cloud option removes the operational burden at the cost of some data-sovereignty advantages.

---

2. RAGFlow

What it is: An open-source RAG engine (Apache 2.0) from InfiniFlow focused on deep document understanding. RAGFlow's headline differentiator is its document parsing pipeline, which handles complex layouts (multi-column PDFs, tables, scanned documents) better than most open-source alternatives.

RAG architecture: DeepDoc layout-aware parser, configurable chunking templates per document type, hybrid retrieval with rerankers, and a chat UI with citation back to source layout coordinates. Recent releases added agent flows and GraphRAG support.

Connectors: Primarily file-based ingestion (local upload, S3, MinIO, web crawl). Native SaaS connectors are limited; most enterprise sources require custom ingestion.

Deployment: Docker Compose. Self-hostable in air-gapped environments with local models.

Best for: Document-heavy verticals (legal, life sciences, manufacturing, finance) where parsing quality on complex PDFs and tables is the binding constraint, and most ingestion is from a document repository rather than SaaS apps.

Trade-offs: Weak connector ecosystem versus Onyx or Glean. No first-class permission inheritance from source SaaS systems. Chat and agent features are less mature than dedicated chat platforms.

---

3. AnythingLLM

What it is: AnythingLLM from Mintplex Labs is an open-source (MIT) all-in-one desktop and Docker RAG application. It targets the broadest possible user base: individual users on laptops, small teams in Docker, and enterprises via a managed offering.

RAG architecture: Workspaces (each with its own vector context), broad vector DB support (LanceDB default, Pinecone, Chroma, Qdrant, Weaviate, Milvus, PGVector), pluggable embedding models, and broad LLM support including local models via Ollama, LM Studio, LocalAI, and OpenAI-compatible endpoints.

Connectors: File upload, web scraping, GitHub, Confluence, YouTube transcripts. Smaller native connector set than Onyx; no enterprise-grade permission syncing.

Deployment: Desktop apps (macOS, Windows, Linux) for individuals; Docker Compose for teams; AWS/GCP/DigitalOcean templates. Air-gappable.

Best for: Individuals and small teams that want a polished cross-platform desktop or Docker RAG app with broad model and vector DB support, especially when local-first usage matters.

Trade-offs: Workspace-level permissioning is not source-system permission inheritance. Smaller enterprise connector library. Less mature for large multi-tenant deployments.

---

4. Verba (Weaviate)

What it is: Verba, the "Golden RAGtriever," is Weaviate's open-source (BSD-3) reference RAG application. It is a thin, opinionated chat UI on top of Weaviate, designed to showcase Weaviate's hybrid search and agentic retrieval patterns.

RAG architecture: Weaviate-backed hybrid retrieval, configurable chunkers and embedders, support for local (Ollama, llama.cpp) and hosted LLMs, and a clean React UI.

Connectors: File upload (PDF, Office, Markdown), web URLs, GitHub. No native SaaS connectors.

Deployment: Docker Compose; cloud deploys easily on any container platform. Air-gappable with local models.

Best for: Teams already standardized on Weaviate that want a working reference RAG UI to build on, or developers prototyping retrieval strategies before committing to a fuller platform.

Trade-offs: Verba is a reference app, not an enterprise product. No permission inheritance, no enterprise connectors, no admin governance. Most teams use it to learn retrieval patterns, then move to a fuller platform or build on top of Weaviate directly.

---

5. Open WebUI

What it is: Open WebUI is the most-starred open-source chat UI (130K+ stars) with built-in document upload RAG. It has the polish of ChatGPT and broad community plugin support via its Pipelines architecture.

RAG architecture: Document upload to an embedded ChromaDB-style vector store, hybrid search (when configured correctly across versions), and OpenAI-compatible LLM backends including Ollama. RAG quality is uneven: users frequently report retrieval reliability issues, ChromaDB scaling pain under concurrent load, and inconsistent hybrid search across releases.

Connectors: None native for enterprise SaaS. Document upload only; live source syncing requires custom pipelines or community MCP servers.

Deployment: Docker, Helm, desktop. Air-gappable.

License: Moved from BSD-3 to a custom restrictive license in 2025 that mandates Open WebUI branding for 50+ user deployments unless you pay. No longer OSI-recognized open source.

Best for: Individuals and small teams that want a polished local chat UI on top of Ollama, with light document RAG.

Trade-offs: No enterprise connectors. No permission inheritance from source systems. RAG quality is hit-or-miss. License changes make 50+ user deployments effectively a paid product.

---

6. LibreChat

What it is: LibreChat is an open-source (MIT) multi-provider chat UI with the most mature authentication stack in open-source chat. ClickHouse acquired LibreChat in late 2025, signaling enterprise ambitions oriented toward data analytics.

RAG architecture: File upload RAG with a configurable vector store. No native enterprise connectors. MCP-based agent framework, multi-language code interpreter, and built-in support for OpenAI, Anthropic, Google, Azure, and local OpenAI-compatible endpoints.

Connectors: None native for enterprise SaaS. File upload only.

Deployment: Docker Compose, Kubernetes. Air-gappable with local models.

Best for: Teams that need a multi-provider chat UI with strong auth (OAuth, SAML, LDAP, 2FA) and MCP-based agent support, without enterprise search requirements.

Trade-offs: No enterprise connectors. No permission inheritance. Post-acquisition product direction is data analytics rather than enterprise knowledge.

---

7. PrivateGPT and LocalGPT

What they are: PrivateGPT and LocalGPT are open-source (Apache 2.0) reference RAG stacks focused on fully local document Q&A. They were two of the first projects to popularize self-hosted RAG and remain useful starting points for engineers learning the architecture.

RAG architecture: Local document ingestion, local embeddings (sentence-transformers, BGE), local vector store (Chroma, Qdrant), and local LLM via llama.cpp or Ollama.

Connectors: File-based only.

Deployment: Docker, native Python. Designed for air-gapped.

Best for: Engineering teams learning RAG architecture, or single-server deployments where document Q&A on a local corpus is the entire requirement.

Trade-offs: Reference implementations, not products. No multi-user UI, no permission system, no enterprise governance.

---

8. LlamaIndex

What it is: LlamaIndex is a widely deployed open-source (MIT) RAG framework with deep retrieval, indexing, and agent abstractions. The self-hosted path is the OSS library; LlamaCloud and LlamaParse are the hosted commercial layers.

RAG architecture: Composable retrievers (vector, keyword, hybrid, recursive, graph-based), reranking integrations, query routing, and an agent framework. Pairs with any vector DB and any LLM. LlamaHub contributes hundreds of community loaders.

Best for: Engineering teams building custom self-hosted RAG applications who want first-class abstractions for advanced retrieval patterns and complex document parsing.

Trade-offs: It is a framework, not a workforce product. You own the connectors, the permissions, the UI, the operations, and the upgrade churn.

---

9. LangChain and Haystack

What they are: LangChain (MIT, ~115K stars) and Haystack (Apache 2.0, by deepset) are the other two most-deployed open-source RAG frameworks. LangChain is the broadest ecosystem; Haystack is more opinionated and pipeline-oriented, with strong production tooling from deepset.

RAG architecture: Composable retrievers, document loaders, chunkers, rerankers, and prompt templates. LangChain pairs with LangSmith (hosted observability and evals) and LangGraph (stateful agent orchestration). Haystack pairs with deepset Studio for pipeline authoring and deepset Cloud for managed inference.

Best for: Engineering teams embedding RAG into a custom application or product. LangChain when you want the broadest integrations; Haystack when you want a tighter, more production-tested pipeline abstraction.

Trade-offs: Frameworks, not workforce products. You own the application end-to-end. LangChain has a large API surface and has historically had churn between versions.

---

10. Elastic + ESRE and OpenSearch

What they are: Elastic (AGPLv3 since September 2024) with the Elasticsearch Relevance Engine and OpenSearch (Apache 2.0, the AWS-led fork) are the two dominant open-source self-hostable search infrastructures with mature hybrid retrieval, vector support, and (in Elastic's case) ESRE for LLM integration primitives. Elastic was named a Leader in Forrester's Wave for Cognitive Search Platforms Q4 2025.

RAG architecture: Mature hybrid search (BM25 + dense vectors + Elastic's ELSER sparse vectors), strong reranking and learned-to-rank, broad relevance tooling. ESRE adds RAG primitives for chunking, embedding, and LLM calls. OpenSearch ships hybrid search, k-NN, and ML plugins for similar patterns. Both can underpin a custom self-hosted RAG application or sit beneath a platform (Onyx uses OpenSearch under the hood).

Best for: Engineering teams that already operate Elasticsearch or OpenSearch, or want maximum control over indexing, ranking, and relevance tuning. Also a strong fit as the retrieval layer beneath a higher-level platform.

Trade-offs: Search infrastructure, not a finished RAG product. Building an enterprise application on top requires substantial development. Elastic's standalone Enterprise Search product (App Search and Workplace Search) is end-of-life, which has created procurement friction over the past two years.

Recommended Self-Hosted RAG Stacks by Buyer Profile

Stack 1: Small Team Self-Hosted RAG (5 to 20 users)

Profile: Engineering team, lab, or small company that wants private AI chat and search over a small set of internal docs. No regulatory hard requirements.

Recommended stack:

• Platform: Onyx Community Edition (Docker Compose) for connectors + search + chat, or AnythingLLM if file-only ingestion is enough.
• Inference: Ollama on a single workstation with an RTX 4090/5090 or Mac Studio. Pull Qwen3 32B (quantized) or Mistral Small for general work.
• Embeddings: BGE-M3 or Nomic Embed, served by the same Ollama process.
• Vector index: Whatever the platform ships (OpenSearch for Onyx, LanceDB for AnythingLLM). Do not introduce a separate vector DB at this scale.
• Connectors: Three to five to start: Slack, Google Drive, GitHub.

Setup time: 1 to 2 hours from a clean machine to first query.

Stack 2: Mid-Market Self-Hosted RAG (20 to 200 users)

Profile: Growing company with multiple departments, real access-control needs, and a mixed SaaS stack (Slack + Google Drive + Confluence + Jira + GitHub + Salesforce or similar).

Recommended stack:

• Platform: Onyx Enterprise for SaaS connector breadth and permission inheritance, or a custom application on Elastic/ESRE if your team already operates Elastic and wants to build the UI in-house.
• Inference: vLLM or SGLang on a single GPU server (A100 80GB or 2x RTX 5090) for the local model tier, plus cloud API fallback (Claude, GPT, Gemini) for frontier tasks. Most platforms route per agent.
• Embeddings: Local BGE-M3 or hosted OpenAI text-embedding-3-large, depending on data sensitivity.
• Reranker: BGE Reranker v2 served locally, or Cohere Rerank 4 via API if data egress is acceptable.
• Vector index: Bundled with the platform (OpenSearch in Onyx's case), or BYO if building on Elastic.
• Connectors: All primary SaaS sources, with ACL syncing turned on per source.

Setup time: 2 to 4 hours for a platform install, plus a few days for full connector rollout and tuning. Building on Elastic from scratch is a multi-month engineering project.

Stack 3: Regulated or Air-Gapped Enterprise Self-Hosted RAG

Profile: Defense, aerospace, healthcare, finance, EU-regulated workloads, or any organization that cannot send queries or embeddings to a third-party cloud. ITAR, FedRAMP, CMMC, HIPAA, FERPA, GDPR.

Recommended stack:

• Platform: Onyx self-hosted on Kubernetes, fully air-gapped. UC San Diego runs this configuration for 37K+ users on internal GPUs.
• Inference: vLLM or SGLang cluster on internal NVIDIA H100/H200 or AMD MI300X GPUs. Open-weight models only: Qwen3, DeepSeek V3, Llama 4, Mistral. No external API calls.
• Embeddings: Local BGE-M3 or E5-Mistral. No external embedding APIs.
• Reranker: Local BGE Reranker v2.
• Vector index: Onyx-managed OpenSearch on internal infrastructure.
• Connectors: Whichever apply on-prem (Confluence Server, GitLab Self-Managed, internal file shares, internal Jira, internal Slack-equivalents).
• Security controls: SSO via internal IdP, SCIM for provisioning, audit log export to SIEM, encrypted secrets, RBAC, and egress blocked at the cluster network policy level.

Setup time: 1 to 2 weeks for the platform install and air-gapped LLM serving, plus a few weeks for full connector rollout, model evaluation, and security review.

Stack 4: Self-Hosted RAG Embedded in a Product

Profile: Engineering team building RAG into a SaaS product, internal app, or domain-specific workflow. End users are not the buyers of an enterprise AI platform.

Recommended stack:

• Framework: LlamaIndex or LangChain for retrieval orchestration. Haystack if you want a more opinionated pipeline.
• Vector index: Qdrant or Weaviate for hybrid search, or Milvus for very large indexes. PGVector if you already run Postgres and the corpus is small enough.
• Embeddings: BGE-M3 or E5-Mistral, served by Text Embeddings Inference or Ollama. OpenAI or Voyage if data egress is acceptable.
• Reranker: BGE Reranker v2, or Cohere Rerank 4.
• LLM: vLLM-served open-weight model for cost, or BYO API for frontier quality. Hybrid is the most common production pattern.
• Evaluation: RAGAS plus a small golden eval set built from your actual corpus.
• Alternative: Build on top of Onyx's REST APIs and MCP server if your product needs connector breadth, permission inheritance, and chat surface from day one. Ramp built their customer-facing "Ramp Assist" feature on Onyx this way.

How to Choose: A Decision Framework

Run through these questions in order. Each one narrows the field.

1. Is this for employees, or for a product I'm building? Employees: stay at the platform layer (Onyx, RAGFlow, AnythingLLM). Product: drop to frameworks (LlamaIndex, LangChain, Haystack) or build on Onyx APIs.
2. Do I need air-gapped deployment? Yes: Onyx, RAGFlow, AnythingLLM, or a framework stack assembled with fully open-source components. Anything that depends on a cloud-only vector DB, reranker, or embedding API is out.
3. How many enterprise SaaS connectors do I need? Many, with permission syncing: Onyx is the only open-source self-hostable platform that ships 40+ with ACLs. A handful, file-only: RAGFlow, AnythingLLM, Verba, Open WebUI, or LibreChat are simpler. Build-your-own: any framework plus custom ETL.
4. Does permission inheritance from source systems matter? Yes: Onyx, or self-built on Elastic/ESRE with disciplined ACL syncing. No: any open-source platform will do.
5. How important is open source and license stability? Critical: Onyx (MIT), LibreChat (MIT), LlamaIndex (MIT), AnythingLLM (MIT), Haystack (Apache 2.0). Watch the license history on Open WebUI before depending on it past 50 users.
6. What is my GPU budget? Tight (one workstation): Ollama plus a 7B-14B local model, with a hosted API for hard queries. Mid (one A100 or 2x RTX 5090): vLLM or SGLang with a 70B-class model. Large (multi-GPU H100/H200): full-stack air-gapped with Qwen3, DeepSeek V3, or Llama 4 at high concurrency.

Recommendation

Most enterprises that want self-hosted RAG should start at the platform layer. Stitching together a vector DB, an embedding service, a reranker, a chunking pipeline, a connector framework, a permission model, an auth layer, and a chat UI is technically possible and almost always a worse use of engineering time than self-hosting a finished product. The teams that try the framework route from scratch tend to spend most of a year on connectors and ACL syncing, not on the parts of RAG that are interesting.

Among self-hosted platform options, Onyx is the clearest fit for workforce-facing deployments that need connector breadth, permission inheritance, model freedom, and air-gapped support all in the same product. RAGFlow is a strong alternative for document-heavy verticals where parsing quality on complex PDFs is the binding constraint. AnythingLLM and Verba work well for single-team or single-workspace deployments without enterprise governance needs. Open WebUI and LibreChat are chat UIs first and light-RAG second; do not pick them if enterprise search is the goal.

For builders embedding RAG into a product, the right move is matching the layer to the team. Pick a framework (LlamaIndex, LangChain, Haystack) with a vector DB (Qdrant, Weaviate, Milvus) and a local inference engine (vLLM, SGLang) if you need maximum flexibility. Build on Onyx's APIs if your product needs connector breadth and permission inheritance from day one, the way Ramp did with "Ramp Assist."

Whatever path you pick, run a proof of concept on your actual corpus with your actual users. Self-hosted RAG quality is corpus-specific, public benchmarks rarely predict your results, and the differences between platforms become obvious within a week of real usage.

Try Onyx for free, self-host the open-source version and connect your first data source in under a day, or book a demo for a regulated or air-gapped deployment.

---

Frequently Asked Questions

What is self-hosted RAG?

Self-hosted RAG is retrieval-augmented generation deployed entirely on infrastructure you control. The connectors, document processing, embedding model, vector index, reranker, LLM, and UI all run inside your network or cloud account. No documents leave your environment for embedding and no queries get sent to a third-party API. Self-hosted RAG is the standard pattern for regulated industries (defense, healthcare, finance, public sector, education), for organizations with strict data residency requirements (EU AI Act, GDPR), and for teams that want predictable cost at scale.

Is self-hosted RAG production-ready in 2026?

Yes. Onyx (MIT, self-hosted) is in production at Ramp (115K queries/month), Thales (1,400 MAU across 82,000 employees on-prem Kubernetes), Astranis (fully self-hosted), L3Harris, NASA, and UC San Diego (37K+ users, fully air-gapped on local GPUs). The open-source frameworks (LangChain, LlamaIndex, Haystack) are deployed in production at thousands of companies. The tooling is mature; the deciding factor is usually how much of the stack a team wants to assemble themselves.

What is the difference between RAG and self-hosted RAG?

RAG is the architecture: retrieve relevant documents, then generate an answer grounded in them. Self-hosted RAG is the deployment pattern where every part of that architecture runs on your infrastructure rather than a vendor's cloud. Hosted RAG (Bedrock Knowledge Bases, Azure AI Search, Vertex AI Search, Vectara) is the opposite end of the spectrum, where the vendor runs everything and you call an API.

What's the best open-source RAG platform for self-hosting?

For workforce-facing deployments with enterprise connectors and permission inheritance, Onyx is the most-deployed open-source option. For document-heavy verticals where PDF and table parsing is the binding constraint, RAGFlow is strong. For small-team or single-workspace deployments without enterprise governance, AnythingLLM and Verba are simpler. For chat-only deployments with light file-upload RAG, Open WebUI and LibreChat dominate. Pick by what your users actually need.

Can I run RAG entirely offline or air-gapped?

Yes. Onyx, RAGFlow, AnythingLLM, and the open-source framework stacks (LlamaIndex, LangChain, Haystack) all support fully air-gapped deployments. The constraints are: open-weight LLMs only (Qwen3, DeepSeek V3, Llama 4, Mistral), local embedding models (BGE-M3, E5-Mistral), local rerankers (BGE Reranker v2), and connectors limited to on-prem systems (Confluence Server, GitLab Self-Managed, internal file shares). UC San Diego runs Onyx in exactly this configuration for 37,000+ users on internal GPUs.

What hardware do I need for self-hosted RAG?

For 5-20 users: a single workstation with one RTX 4090 or RTX 5090, or a Mac Studio, running Ollama plus a 7B-14B model. For 20-200 users: a GPU server with one A100 80GB or two RTX 5090s running vLLM or SGLang with a 70B-class model. For 200+ users or air-gapped enterprise: a multi-GPU cluster (4-8x H100/H200, or AMD MI300X) running vLLM or SGLang with quantized 200B-400B MoE models at high concurrency. Embedding and reranker workloads are CPU-tolerant but benefit from a small GPU slice.

How much does self-hosted RAG cost compared to cloud RAG services?

Self-hosting trades capex for opex. A 200-user deployment typically breaks even within 6 to 12 months on GPU hardware versus cloud API costs, depending on usage intensity and model tier. Cloud RAG services (AWS Bedrock Knowledge Bases, Azure AI Search, Vertex AI Search, Vectara) bill on indexed tokens, queries, embeddings, generation tokens, and storage. The platform layer can also be free: Onyx's MIT community edition is fully functional with no per-seat fees, so the only ongoing cost is infrastructure plus your time.

Does self-hosted RAG satisfy HIPAA, SOC 2, GDPR, or ITAR?

Self-hosting is necessary but not sufficient. Keeping data on your infrastructure checks one box, but auditors will also ask about audit logging, encryption at rest and in transit, retention policies, access controls, SSO, RBAC, and BAAs with every vendor in the chain. A common compliance gap: teams self-host the LLM but use a cloud embedding API, which means data still leaves the network. Onyx Enterprise includes SOC 2 Type II, GDPR compliance, and audit trails; for ITAR, FedRAMP, and CMMC environments, the air-gapped deployment with zero outbound traffic is the standard configuration. Onyx is deployed in all of these environments today.

Can I use OpenAI or Anthropic with self-hosted RAG?

Yes, in the deployment models where data egress is acceptable. Onyx, RAGFlow, AnythingLLM, LibreChat, and the OSS frameworks all support any LLM via OpenAI-compatible endpoints. Common patterns: route routine queries to a local open-weight model for cost and privacy, route hard queries (deep research, complex coding) to a frontier API for quality. Onyx supports this routing per agent. For truly sensitive workloads or air-gapped deployments, run open-weight models locally and disable hosted endpoints at the platform layer.

What vector database should I use for self-hosted RAG?

In platform deployments, the vector DB is bundled. Onyx uses OpenSearch under the hood. RAGFlow uses Elasticsearch. AnythingLLM defaults to LanceDB with broad alternatives. Verba uses Weaviate. You only pick a vector DB explicitly when building from frameworks: Qdrant and Weaviate for hybrid search with strong filtering, Milvus for very large indexes, PGVector if you already run Postgres, Chroma for development. OpenSearch and Elasticsearch are the natural choice when you need keyword + vector hybrid search in the same engine and already operate one of them.

Is building self-hosted RAG with LangChain and Qdrant cheaper than using a platform?

Almost never, for workforce-facing AI. Production-grade connectors, permission inheritance, evaluation, observability, governance, and a UI cost more to build than they cost to self-host with an open-source platform. Astranis connected all their knowledge sources with Onyx in under a day; the equivalent custom build is a multi-quarter engineering project. The framework approach is the right call when embedding RAG into a custom product with deep customization needs, or when the only requirement is document Q&A on a small fixed corpus.

How do I evaluate self-hosted RAG quality before committing?

Build an evaluation set of 50 to 100 questions with known correct answers from your actual corpus. Run each candidate platform against the same corpus and score on retrieval precision (was the right chunk retrieved?), answer faithfulness (does the answer match the source?), and citation correctness. RAGAS automates parts of this. The MTEB leaderboard is the public reference for embedding model quality. Public benchmarks rarely predict your results because RAG quality is highly corpus-specific.