Best Self-Hosted LLMs in 2026

Self-hosting an LLM used to mean accepting significantly worse performance than the proprietary APIs. In 2026, that's changed. The best self-hosted models now approach proprietary APIs much more closely on coding benchmarks, but they require real hardware to run them. The question isn't just capability anymore. It's whether the economics make sense for your team, and which model fits your GPU setup.

This guide covers the top 10 self-hosted LLMs from the Onyx Self-Hosted LLM Leaderboard, organized by the hardware tier they require. Data is updated as of July 20, 2026.

How this guide is sourced: Hardware, benchmark, and license data comes from the Onyx Self-Hosted LLM Leaderboard. The recommendations in each section interpret that data for specific deployment tiers.

---

TL;DR: The right self-hosted model depends almost entirely on what GPU you have. At the top of the leaderboard, DeepSeek-V4-Pro (80.6% SWE-bench Verified), Kimi K2.6 (80.2% SWE-bench Verified), and GLM-5.2 (91.2% GPQA Diamond) all need a 16x H100 80GB cluster. DeepSeek-V4-Flash brings that same DeepSeek-V4 family down to a 4x H100 cluster at 79% SWE-bench Verified, and Step-3.7-Flash runs on 2x H100 with 393.8 tokens/sec throughput. A single H100 gets you Nemotron 3 Super, a 120B MoE model with only 12B active parameters. On an RTX 4090, Qwen3.6-27B is the general-purpose pick, Qwen3.6-35B-A3B trades some benchmark score for MoE efficiency, and Gemma 4 31B is the fully free option at $0 per token. For budget GPUs, Gemma 4 12B runs on an RTX 3090.

---

What Is a Self-Hosted LLM?

A self-hosted LLM is a model you run on hardware you control rather than calling via an API. Your data stays on your infrastructure, you don't pay per token, and you can customize the model for your specific use case.

Reasons teams choose to self-host:

• Data privacy: Sensitive documents, code, and queries never leave your infrastructure
• Cost at scale: At high query volumes, self-hosting is cheaper than API pricing
• Air-gapped environments: Defense, healthcare, and regulated industries requiring offline deployment
• Model customization: Fine-tuning and LoRA adapters for domain-specific tasks
• Latency control: Predictable inference performance without rate limits

---

---

What Is Onyx?

Onyx is an open-source AI platform that connects self-hosted LLMs to company knowledge, permissions, and workplace workflows. The model runtime handles inference, while Onyx provides connectors, search, chat, agents, deep research, citations, and a governed user interface.

For self-hosted model deployments, Onyx is the layer that makes the model useful to a team instead of only to the infrastructure group. It can connect a local model endpoint to Slack, Confluence, Jira, Google Drive, SharePoint, GitHub, and other sources while preserving source-system access controls.

Hardware Tiers

---

Self-Hosted Models at a Glance

16x H100 (1,280GB): DeepSeek-V4-Pro, Kimi K2.6, GLM-5.2

4x H100 (320GB): DeepSeek-V4-Flash

Dual GPU (2x H100): Step-3.7-Flash

Single H100: Nemotron 3 Super

RTX 4090: Qwen3.6-27B, Qwen3.6-35B-A3B, Gemma 4 31B

RTX 3090: Gemma 4 12B

---

Best Self-Hosted LLMs by Hardware Tier

Multi-Node Cluster Tier (16x H100 80GB)

These are the frontier open-weight releases of this refresh, and they need real cluster infrastructure. Hardware requirements are steep at this tier: DeepSeek-V4-Pro, Kimi K2.6, and GLM-5.2 all list a 16x H100 80GB (1,280GB total) minimum, though INT4 quantization brings the working set down considerably. Recommended for enterprise teams with existing multi-node GPU infrastructure or the budget to build it.

DeepSeek-V4-Pro: MIT-licensed, 1.6T total / 49B active parameters. Leads this tier on SWE-bench Verified at 80.6% and LiveCodeBench at 93.5%, with 3,200GB VRAM at FP16 (800GB at INT4) and 64.1 tokens/sec throughput. The top pick when raw coding capability matters more than hardware footprint.

Kimi K2.6: Modified MIT license, 1T total / 32B active parameters. Close behind on SWE-bench Verified at 80.2% and ahead on GPQA Diamond at 90.5%, with strong agentic scores (73.1% OSWorld, 83.2% BrowseComp). Choose it for agentic and tool-use workloads at this tier.

GLM-5.2: MIT-licensed, 753B total / 40B active parameters. Leads on GPQA Diamond at 91.2% and runs at 193.3 tokens/sec, the fastest of the three. Choose it when reasoning benchmarks and throughput matter more than published SWE-bench numbers.

---

4x H100 Cluster Tier (320GB)

DeepSeek-V4-Flash brings the same model family down to a single 4x H100 80GB node, trading some benchmark score for dramatically lower hardware requirements.

DeepSeek-V4-Flash: MIT-licensed, 284B total / 13B active parameters. Scores 79% on SWE-bench Verified and 91.6% on LiveCodeBench, close to the 16x H100 tier at a quarter of the VRAM (568GB at FP16, 142GB at INT4). Runs at 120.4 tokens/sec, roughly double DeepSeek-V4-Pro's throughput. The practical choice for teams that want DeepSeek-V4 quality without a 16-GPU cluster.

---

Dual H100 Tier (2x H100 80GB)

Step-3.7-Flash: Apache 2.0, 198B total / 11B active parameters. Scores 76.5% on SWE-bench Verified and runs at 393.8 tokens/sec, the fastest model in this guide by a wide margin. Fits on 2x H100 80GB at 396GB VRAM (FP16) or 99GB at INT4. The pick when inference speed matters as much as capability.

---

Single H100 Tier (1x H100 80GB)

Single H100 deployments are the most common enterprise self-hosting configuration. At $15K-$25K for a used/lease H100, this tier represents strong cost-performance for teams running inference workloads.

Nemotron 3 Super: NVIDIA Open Model license, 120B total / 12B active parameters via MoE. Scores 60.47% on SWE-bench Verified and 90.21% on AIME 2025, with 240GB VRAM at FP16 (60GB at INT4) fitting comfortably on a single H100 80GB. Runs at 191.6 tokens/sec at $0.25 / $0.775 per 1M input/output tokens if served through NVIDIA's hosted endpoints, or free to self-host on your own hardware.

---

RTX 4090 Tier (24GB VRAM)

The RTX 4090 has become the benchmark consumer GPU for LLM enthusiasts and small teams. At 24GB VRAM, it runs 27B-35B dense and MoE models in INT4 quantization comfortably.

Qwen3.6-27B: Apache 2.0, 27B dense parameters, runs on a single RTX 4090 at 54GB VRAM (FP16) or 14GB at INT4. Scores 77.2% on SWE-bench Verified and 83.9% on LiveCodeBench. The default pick for RTX 4090 owners who want the strongest all-round dense model at this size.

Qwen3.6-35B-A3B: Apache 2.0, 35B total / 3B active parameters via MoE. Scores 73.4% on SWE-bench Verified while running at 153.2 tokens/sec, close to triple Qwen3.6-27B's throughput, at only 70GB VRAM (FP16) or 18GB (INT4). Choose it when inference speed matters more than the last few points of benchmark score.

Gemma 4 31B: Gemma license, 31B dense parameters, 1451 Chatbot Arena Elo and 88.4% MMMLU. Free to run at $0 per token, with 62GB VRAM at FP16 (16GB at INT4). The pick for teams that want a fully open, no-cost model with strong general reasoning.

---

Budget Tier (RTX 3090)

The most accessible tier for individual developers and small teams. Gemma 4 12B is the best-in-class pick here, offering strong instruction-following on consumer hardware that costs $400-$700.

Gemma 4 12B: Gemma license, 12B dense parameters, 97.2% IFEval and 83.4% MMMLU, at only 24GB VRAM (FP16) or 6GB at INT4. Runs at 108.8 tokens/sec and costs $0.1 / $0.3 per 1M input/output tokens on hosted endpoints, or free to self-host. The right pick when hardware is the hard constraint but instruction-following quality still matters.

---

Choosing the Right Self-Hosted Model

---

Connecting Self-Hosted LLMs to Your Company's Data

Self-hosting solves the inference problem, not the retrieval and permissions problem. Most teams still need a way to connect the model to internal knowledge sources and keep access controls intact.

Onyx is relevant here as the application layer around a self-hosted model. You can point Onyx at a vLLM or Ollama deployment, connect sources like Slack, Confluence, Jira, and GitHub, and keep permission-aware search and chat in front of users. That makes the hardware choices in this guide more actionable: the model handles inference, while Onyx handles retrieval, orchestration, and team-facing access.

---

Recommended Self-Hosted LLM Stack

The model runtime should not be the user experience. Put a governed layer like Onyx in front of the model so employees can search internal knowledge, see citations, and use the same permissions they already have in source systems.

Frequently Asked Questions

What is the best self-hosted LLM in 2026?

In this leaderboard snapshot, the best self-hosted LLM depends mostly on hardware tier. On enterprise GPU clusters, DeepSeek-V4-Pro leads SWE-bench Verified at 80.6%, Kimi K2.6 is close behind at 80.2% with the strongest agentic scores, and GLM-5.2 leads GPQA Diamond at 91.2%. DeepSeek-V4-Flash brings the same DeepSeek-V4 quality down to a 4x H100 cluster. On a single H100, Nemotron 3 Super is the top pick. On an RTX 4090, Qwen3.6-27B is the general-purpose pick, Qwen3.6-35B-A3B trades score for MoE throughput, and Gemma 4 31B is the free option. For budget hardware, Gemma 4 12B runs on an RTX 3090.

Can I run a frontier-class LLM on a single GPU?

Yes. Nemotron 3 Super achieves 60.47% SWE-bench Verified and 90.21% AIME 2025 on a single H100 80GB. On an RTX 4090, Qwen3.6-27B delivers 77.2% SWE-bench Verified and 83.9% LiveCodeBench, capability that would have required a multi-GPU cluster two years ago.

What tools do I use to run these models locally?

The most common inference runtimes are Ollama (easiest setup for consumer hardware), vLLM (best throughput for production deployments), LM Studio (GUI-based for Mac/Windows), and llama.cpp (CPU and low-VRAM support). Most models in this guide are available through Ollama or HuggingFace directly.

Which self-hosted LLMs support air-gapped deployment?

All models in this guide can run in fully air-gapped environments after the initial model weight download. MIT, Apache 2.0, and Gemma licensed models have no phone-home requirements or usage telemetry by default. For regulated environments (ITAR, HIPAA, FedRAMP), open-weight models on private infrastructure provide a compliant self-hosting path.

How do I estimate self-hosting cost versus API costs?

A single H100 80GB server node (cloud or on-premise) runs approximately $2-4/hour on cloud or $15K-$25K to purchase. Nemotron 3 Super running on that node can serve substantial daily token volume at 191.6 tokens/sec. At DeepSeek-V4-Flash's $0.14/M input price for hosted access, high-volume teams typically break even on self-hosting within a few months of steady utilization. For teams with consistent 24/7 inference demand, self-hosting is typically cheaper over a 2-year horizon than paying per-token. See Best Open Source LLMs 2026 for a full comparison of open-source model licenses, or Best LLMs for Coding 2026 for coding-focused rankings including API options.

What is the best model for my specific hardware?

It depends on your GPU's VRAM and whether you are using FP16 or INT4 quantization. The Onyx LLM Hardware Requirements Calculator lets you enter your hardware and instantly see which models fit, including VRAM requirements and recommended quantization settings for each model in this guide.