# AI Gateway & Model Evaluation Set 📊 > A professional, reproducible evaluation layer for [Awesome AI Gateway](README.md): how the **models** behind the gateways actually perform, what they **really cost** on concrete workloads, and how the **gateways themselves** score on compliance, price, security and stability. > > **Languages:** English · [简体中文](BENCHMARKS.zh-CN.md) · Last reviewed: **see footer** Every number here is **sourced and dated**. Cost cells are *computed* from a public pricing table by a unit-tested script ([`scripts/cost_calc.py`](scripts/cost_calc.py)), never hand-typed — re-run it and you get the same table. Model scores are copied from primary leaderboards with links. Gateway scores follow the published [rubric](#scoring-rubric-apply-consistently) below. ## Contents - [Part 1 — Authoritative model benchmarks](#part-1--authoritative-model-benchmarks) - [Part 2 — Pick a model by scenario](#part-2--pick-a-model-by-scenario) - [Part 3 — Real-world token cost (computed)](#part-3--real-world-token-cost-computed) - [Part 4 — Gateway scorecard: compliance · price · security · stability · observability](#part-4--gateway-scorecard-compliance--price--security--stability--observability) - [Part 5 — Real-world reviews: what production users report](#part-5--real-world-reviews-what-production-users-report) - [Part 6 — Gateway observability: the factors that matter](#part-6--gateway-observability-the-factors-that-matter) - [Methodology & caveats](#methodology--caveats) - [Sources](#sources) --- ## Part 1 — Authoritative model benchmarks How capable is each model? These are the most-cited public benchmarks as of the review date. **Read them with the [caveats](#methodology--caveats)** — leaderboards get gamed and contaminated; pair them with the human-preference Arena and the real-world cost tables below. Ranked by the **Artificial Analysis Intelligence Index** (the most-cited one-number composite). These are **v4.0**-scale numbers; AA has since shipped **v4.1**, which re-weights toward agentic work and now *absorbs* Terminal-Bench, τ³ and GDPval — so don't read those as independent of the Index. `♦` = GPQA Diamond. `—` = not verified at review time. | # | Model | Provider | Weights | Context | GPQA♦ | SWE-bench Verified | AIME | Arena Elo | AA Index | |---|---|---|---|---|---|---|---|---|---| | 1 | **Claude Fable 5** | Anthropic | Closed | 1M | 95.0% | 95.0% | — | —ᵗ | **65** 🥇 | | 2 | **Claude Opus 4.8** | Anthropic | Closed | 1M | 93.6% | 88.6% | — | —ᵗ | **61.4** | | 3 | **GPT-5.5** | OpenAI | Closed | ~1M | 93.6% | 88.7% | — | 1402ᵗ | **60.2** | | 4 | **Gemini 3.1 Pro** | Google | Closed | 1M | 94.3% | 80.6% | 98.2%¹ | 1406 | **57.2** | | 5 | **Qwen3.7 Max** | Alibaba | Closed | 1M | 92.4% | — | 75%² | — | **56.6** | | 6 | **Gemini 3.5 Flash** | Google | Closed | 1M | — | — | — | — | **55.3** | | 7 | **Kimi K2.6** | Moonshot | 🔓 Open | 256K | 90.5% | 80.2% | 96.4%¹ | — | **53.9** | | 8 | **Grok 4.3** | xAI | Closed | 1M | ~89%³ | ~75%³ | ~95%³ | — | **53.2** | | 9 | **Muse Spark** | Meta | Closed | 262K | — | — | — | — | **52.1** | | 10 | **DeepSeek V4 Pro** | DeepSeek | 🔓 Open · MIT | 1M | 90.1% | 80.6% | 89.3%² | — | **51.5** | | 11 | **GLM-5.1** | Z.ai (Zhipu) | 🔓 Open | 200K | 86.2% | — | 95.3%¹ | — | **51.4** | | 12 | **Claude Haiku 4.5** | Anthropic | Closed | 200K | — | 73.3% | — | — | — | | 13 | **Mistral Large 3** | Mistral | 🔓 Open | 256K | 43.9% | — | — | — | **22.8** | ᵗ Arena Elo shown is the prior GPT-5.2 snapshot; models released after May 2026 (Fable 5, Opus 4.8, GPT-5.5) are not yet settled on Arena — capabilities and human-preference ranking diverge, so don't read absence as weakness. ¹ AIME 2026 · ² AIME 2025 (different years and "with tools / no tools" variants are **not directly comparable**) · ³ Grok 4.3 figures extrapolated from Grok 4 reporting — approximate. > 🛡️ **Contamination-resistant cross-check.** On **SWE-bench Pro** (harder to game than Verified): Fable 5 **80.3%** 🥇 · Opus 4.8 69.2% · GPT-5.5 / Kimi K2.6 ~58.6% · GLM-5.1 58.4%. On **Humanity's Last Exam**: Fable 5 ~59% · Gemini 3.1 Pro 44.4%. The frontier clusters at 90–95% on GPQA — at that ceiling, 1–2 point gaps are noise. **What each column means** - **GPQA Diamond** — graduate-level science questions, Google-proof by design. - **SWE-bench Verified** — fixes real GitHub issues; the headline *agentic coding* score. - **AIME** — competition math (exact-answer reasoning under pressure). - **Arena Elo** — blind human preference on [Arena (ex-LMArena)](https://arena.ai/leaderboard); the hardest metric to game. - **AA Index** — [Artificial Analysis](https://artificialanalysis.ai) Intelligence Index, a composite across agentic/coding/reasoning/knowledge benchmarks. --- ## Part 2 — Pick a model by scenario Benchmarks rank capability in the abstract; most teams have one concrete job. This maps the common jobs to a *capability* pick and a *value* pick (good-enough, far cheaper). Cross-check the price in [Part 3](#part-3--real-world-token-cost-computed). | Your job | 🏆 Capability pick | 💸 Value pick (good-enough, far cheaper) | Why | |---|---|---|---| | **Agentic coding** (SWE-bench) | Claude Fable 5 / Opus 4.8 | Kimi K2.6 · DeepSeek V4 Pro | Open models hit ~80% SWE-bench Verified at a fraction of flagship cost | | **Long-context / RAG** (100K+) | Gemini 3.1 Pro (1M ctx) | DeepSeek V4-Flash (1M ctx) | Cost floor on input-heavy work; mind Gemini's >200K surcharge | | **Hard reasoning / math** | Gemini 3.1 Pro (98.2 AIME'26) | GLM-5.1 · Kimi K2.6 | Open models reach 95%+ AIME — math is the most commoditized frontier skill | | **Bulk generation** (emails, content) | Claude Haiku 4.5 | DeepSeek V4-Flash · GPT-5.4 nano | Output-heavy → output price dominates; see [3.1](#31-write-a-100k-token-report-generation-heavy) | | **Cheapest acceptable chat** | GPT-5.4 nano | DeepSeek V4-Flash | ~$0.21 per 1M-token chatbot month vs $17.50 for GPT-5.5 | | **Open-ended chat** (human pref) | Gemini 3.1 Pro (Arena 1406) · GPT-5.5 | — | Arena Elo is the metric that tracks "feels good to use" | | **On-prem / data-sovereign** | DeepSeek V4 Pro (MIT) · GLM-5.1 | Kimi K2.6 | Open weights you can run inside your own VPC — zero data egress | | **Compliance-bound enterprise** | Claude Opus 4.8 / GPT-5.5 via Azure / Bedrock / Vertex | — | Route flagships through a [first-party cloud](#part-4--gateway-scorecard-compliance--price--security--stability--observability) with HIPAA/FedRAMP | > A **gateway** is what lets you act on this table without rewriting code: set the capability pick as primary and the value pick as fallback, or route per-request by task. That's the whole point of the [list](README.md). --- ## Part 3 — Real-world token cost (computed) > "Benchmarks tell you what's *best*. Your invoice tells you what's *affordable*." These tables price four concrete workloads across a representative model set, computed from the pricing in [`data/models.json`](data/models.json) by [`scripts/cost_calc.py`](scripts/cost_calc.py). Pricing is USD per 1M tokens; reasoning models bill hidden thinking tokens at the output rate. ### 3.1 Write a 100K-token report (generation-heavy) **Write a 100K-token report** (input 2,000 tok · output 100,000 tok) | # | Model | Provider | Cost | |---|---|---|---| | 1 | DeepSeek V4-Flash | DeepSeek | $0.028 | | 2 | GPT-5.4 nano | OpenAI | $0.13 | | 3 | Mistral Large 3 | Mistral | $0.15 | | 4 | Kimi K2.6 | Moonshot | $0.40 | | 5 | GLM-5.1 | Z.ai (Zhipu) | $0.44 | | 6 | Claude Haiku 4.5 | Anthropic | $0.50 | | 7 | Gemini 3.5 Flash | Google | $0.90 | | 8 | Gemini 3.1 Pro | Google | $1.20 | | 9 | Grok 4 | xAI | $1.51 | | 10 | Claude Opus 4.8 | Anthropic | $2.51 | | 11 | GPT-5.5 | OpenAI | $3.01 | > 📊 Cheapest is **~106×** less than the most expensive for this task. ### 3.2 Summarize a 100K-token document (input-heavy) **Summarize a 100K-token document** (input 100,000 tok · output 2,000 tok) | # | Model | Provider | Cost | |---|---|---|---| | 1 | DeepSeek V4-Flash | DeepSeek | $0.015 | | 2 | GPT-5.4 nano | OpenAI | $0.023 | | 3 | Mistral Large 3 | Mistral | $0.053 | | 4 | Kimi K2.6 | Moonshot | $0.10 | | 5 | Claude Haiku 4.5 | Anthropic | $0.11 | | 6 | GLM-5.1 | Z.ai (Zhipu) | $0.15 | | 7 | Gemini 3.5 Flash | Google | $0.17 | | 8 | Gemini 3.1 Pro | Google | $0.22 | | 9 | Grok 4 | xAI | $0.33 | | 10 | Claude Opus 4.8 | Anthropic | $0.55 | | 11 | GPT-5.5 | OpenAI | $0.56 | > 📊 Cheapest is **~38×** less than the most expensive for this task. ### 3.3 Coding-agent session (mixed + reasoning tokens)

> **Capability *and* cost on one axis.** Every model with both a published SWE-bench Verified score and a price, plotted on the shared coding-agent session. Open weights (green) hit ~80% — flagship-*tier* coding — for a fraction of the spend: **DeepSeek V4 Pro ties Gemini 3.1 Pro (80.6%) at ~11× less**, and the 95% ceiling (Fable 5) costs ~46× the cheapest model that still clears 80%. The cost axis reuses the unit-tested engine below; capability is the dated `swe_bench_verified` figure. Rendered by [`scripts/make_coding_chart.py`](scripts/make_coding_chart.py) — re-run it and you get the same picture. **Coding-agent session** (input 50,000 tok · output 20,000 tok · +30,000 thinking for reasoning models) | # | Model | Provider | Cost | |---|---|---|---| | 1 | DeepSeek V4-Flash | DeepSeek | $0.021 | | 2 | Mistral Large 3 | Mistral | $0.055 | | 3 | GPT-5.4 nano | OpenAI | $0.073 | | 4 | Kimi K2.6 | Moonshot | $0.13 | | 5 | Claude Haiku 4.5 | Anthropic | $0.15 | | 6 | GLM-5.1 | Z.ai (Zhipu) | $0.29 | | 7 | Gemini 3.5 Flash | Google | $0.53 | | 8 | Gemini 3.1 Pro | Google | $0.70 | | 9 | Grok 4 | xAI | $0.90 | | 10 | Claude Opus 4.8 | Anthropic | $1.50 | | 11 | GPT-5.5 | OpenAI | $1.75 | > 📊 Cheapest is **~83×** less than the most expensive for this task. ### 3.4 1M-token chatbot month (balanced) **1M-token chatbot month** (input 500,000 tok · output 500,000 tok) | # | Model | Provider | Cost | |---|---|---|---| | 1 | DeepSeek V4-Flash | DeepSeek | $0.21 | | 2 | GPT-5.4 nano | OpenAI | $0.72 | | 3 | Mistral Large 3 | Mistral | $1.00 | | 4 | Kimi K2.6 | Moonshot | $2.48 | | 5 | GLM-5.1 | Z.ai (Zhipu) | $2.90 | | 6 | Claude Haiku 4.5 | Anthropic | $3.00 | | 7 | Gemini 3.5 Flash | Google | $5.25 | | 8 | Gemini 3.1 Pro | Google | $7.00 | | 9 | Grok 4 | xAI | $9.00 | | 10 | Claude Opus 4.8 | Anthropic | $15.00 | | 11 | GPT-5.5 | OpenAI | $17.50 | > 📊 Cheapest is **~83×** less than the most expensive for this task. **Pricing gotchas a gateway buyer must know** 1. **Reasoning tokens are billed as output.** A "cheap" reasoning model can cost more than a flagship once it thinks for 30K tokens. The coding table above includes them. 2. **Cached input is 5–10× cheaper.** Reusing a long system prompt? The cached-input rate, not the headline input rate, is your real cost. 3. **Batch APIs are ~50% off** for non-interactive work (Anthropic, OpenAI, Google all offer this). 4. **China models are priced in RMB** and often have off-peak discounts (DeepSeek) — the USD figures here are conversions and move with the exchange rate. --- ## Part 4 — Gateway scorecard: compliance · price · security · stability · observability This is the part buyers actually lose sleep over. Models are interchangeable; the gateway is where your keys, prompts, and audit trail live. Each gateway is scored ★1–5 on five axes using the rubric below, so scores are comparable rather than vibes. The observability axis grades what a gateway actually exposes (per-gateway evidence in [`data/gateways_eval.json`](data/gateways_eval.json)); [Part 6](#part-6--gateway-observability-the-factors-that-matter) explains what each pillar means in practice. ### Scoring rubric (apply consistently) | ★ | Compliance | Security | Stability / reliability | Observability | |---|---|---|---|---| | ★5 | SOC 2 Type II **+** ISO 27001 **+** HIPAA BAA **+** EU residency **+** ZDR | Guardrails + PII redaction + RBAC + SSO/SAML + audit logs + key vault | Public uptime SLA ≥99.9%, status page, multi-provider failover, sub-ms overhead | All 5 pillars: metrics export + trace export + per-key token/cost attribution + log export + dashboard | | ★4 | SOC 2 + one of {ISO, HIPAA, residency} + ZDR option | Most of the above, missing one enterprise control | SLA or strong failover + healthy maintenance | 4 of 5 pillars (typically missing trace export or a bundled UI) | | ★3 | SOC 2 **or** GDPR posture, ZDR on request | RBAC + audit logs + keys encrypted | Failover/fallback, active releases, no public SLA | Cost/usage accounting + dashboard, no standards-based (Prometheus/OTel) export | | ★2 | Privacy policy only, no third-party audit | Basic auth + key storage, few controls | Best-effort, community-maintained | Basic request logs/stats only | | ★1 | None stated | Known unpatched issues / minimal controls | Sporadic maintenance or unproven | Little beyond the provider invoice | | 🏠 | *Self-hosted: **you** own these. Score reflects the controls the software gives you to comply.* | | | | **Markup** = what the gateway charges on top of provider token cost. Self-hosted = $0 markup, you pay infra + ops. #### Hosted multi-provider gateways | Gateway | Compliance | Markup | Security | Stability | Obsv | One-line | |---|---|---|---|---|---|---| | **Cloudflare AI Gateway** | ★★★★½ | **0%** | ★★★★ | ★★★★½ | ★★★★★ | CF holds SOC 2 II / ISO 27001 / PCI; free DLP + fallback; 100% SLA at Business+ | | **Portkey** (cloud) | ★★★★½ | usage-based | ★★★★½ | ★★★★ | ★★★★★ | SOC 2 II + ISO + HIPAA; 50+ guardrail marketplace, RBAC/SSO; 99.99% SLA | | **Vercel AI Gateway** | ★★★★ | **0%** | ★★★½ | ★★★★ | ★★★★ | SOC 2 II + 99.99% SLA (Enterprise); true 0% even on BYOK | | **Helicone** (cloud) | ★★★½ | **0%** passthrough | ★★★½ | ★★★ | ★★★★½ | SOC 2 + HIPAA (Team); PII detection; OSS core → VPC/self-host option | | **Requesty** | ★★★½ | ~5% | ★★★½ | ★★★ | ★★★ | EU residency + PII masking + ZDR; SOC 2 "in progress Q2'26" (not yet Type II) | | **OpenRouter** | ★★★½ | ~5.5% credit fee | ★★★ | ★★★ | ★★★★½ | ~90 providers, auto-failover, free ZDR; **no public SLA** (enterprise only) | | **Eden AI** | ★★★½ | ~5.5% platform fee | ★★★ | ★★★½ | ★★★ | France-based, EU-default residency, GDPR-first; SOC 2 UNVERIFIED | | **Martian** | ★★★ | volume (undisclosed) | ★★★½ | ★★★ | ★★½ | "Airlock" compliance vetting + cost-routing; certs UNVERIFIED | #### First-party clouds (single-vendor, strongest certs) | Gateway | Compliance | Markup | Security | Stability | Obsv | One-line | |---|---|---|---|---|---|---| | **Azure OpenAI** | ★★★★★ | N/A | ★★★★★ | ★★★★½ | ★★★★½ | SOC 2 / ISO / HIPAA-BAA / **FedRAMP High**, region pinning, ZDR endpoints | | **AWS Bedrock** | ★★★★★ | N/A | ★★★★★ | ★★★★½ | ★★★★ | ISO / SOC / CSA STAR / HIPAA / FedRAMP High; multi-model within Bedrock | | **Google Vertex AI** | ★★★★½ | N/A | ★★★★★ | ★★★★½ | ★★★★ | First GenAI platform to FedRAMP High (2025); SOC 2 / ISO / HIPAA | | **OpenAI** (direct) | ★★★★ | N/A | ★★★★ | ★★★★ | ★★★★ | SOC 2 II, HIPAA-BAA, ZDR; but single-vendor = no cross-provider failover | > ⚠️ First-party clouds win compliance but **can't survive a provider outage** — that cross-vendor failover is exactly what a gateway in front of them buys you. #### Open-source self-hosted (🏠 you own compliance; $0 markup, you pay infra) | Gateway | Compliance | Security | Stability | Obsv | One-line | |---|---|---|---|---|---| | **Portkey Gateway** (OSS) | ★★★🏠 | ★★★★ | ★★★★ | ★★ | Apache-2.0; full guardrails, MCP OAuth, fallbacks free; <1ms overhead | | **Kong AI Gateway** | ★★★½ | ★★★★½ | ★★★★ | ★★★½ | PII sanitization (20+ types), Prompt Guard, RBAC on mature Kong lineage | | **Envoy AI Gateway** | ★★★🏠 | ★★★★ | ★★★★ | ★★★★ | Multi-provider + MCP gateway w/ OAuth+CEL authz; native K8s/Istio | | **Bifrost** (Maxim) | ★★★🏠 | ★★★½ | ★★★★½ | ★★★★★ | Go; ~11µs overhead benchmark, cluster mode; no known CVEs | | **TensorZero** | ★★★🏠 | ★★★ | ★★★★ | ★★★★½ | Rust; <1ms p99 at 10k+ QPS; routing + built-in observability; ⚠️ archived Jun 2026 | | **Higress** | ★★★🏠 | ★★★½ | ★★★★ | ★★★★½ | Istio/Envoy AI-native, Wasm plugins, console; Alibaba-backed | | **Apache APISIX** | ★★★🏠 | ★★★ | ★★★★ | ★★★½ | ai-proxy / ai-prompt-guard plugins on mature ASF gateway | | **LiteLLM** | ★★★🏠 | ★★½ ⚠️ | ★★★★ | ★★★★★ | SOC 2 I + ISO (Enterprise); **patch to ≥v1.83.7** — 2 serious 2026 CVEs (1 RCE on CISA KEV), both fixed | | **GPT-Load** | ★★🏠 | ★★½ | ★★★½ | ★½ | Go key-pool rotation + encrypted key store + dual auth; proxy-level only | | **new-api** | ★★🏠 | ★½ ⚠️ | ★★★ | ★★½ | ~38k★ & active, but **cluster of 2026 CVEs** (IDOR/SSRF/SQLi) — sandbox + patch fast | | **one-api** | ★★🏠 | ★★ | ★★½ | ★★ | The MIT original; maintenance slowed — new-api is the more active successor | > ⚠️ **CVE honesty.** Popularity makes OSS gateways targets. LiteLLM (pre-auth SQLi + unauth RCE) and new-api (IDOR/SSRF/SQLi) both had serious 2026 advisories — *patched*, but the lesson is: pin to current stable, restrict egress, and don't expose the admin panel publicly. Absence of found CVEs (Bifrost, TensorZero, Higress, Envoy, GPT-Load) ≠ proven-secure; it can mean less scrutiny. > 📊 **Obsv column** = the five-pillar observability score from the rubric; per-gateway evidence (which pillars, which docs) is machine-readable in [`data/gateways_eval.json`](data/gateways_eval.json). Standouts: **LiteLLM / Bifrost / Cloudflare / Portkey cloud** cover all five pillars; **Portkey OSS v1.x ships near-zero observability** (its telemetry lands in the unreleased 2.0 branch); **Envoy AI Gateway** is the strongest standards-first pick (OTel GenAI-semconv, no UI); CN-panel gateways (new-api/one-api/GPT-Load) invert — strong billing UI, no Prometheus/OTel. > ⏱️ **Gateway overhead, independently measured.** Vendors market conflicting overhead claims (µs-level vs ms-level) with no third-party numbers — so this project measures it: a reproducible harness (mock OpenAI upstream, interleaved rounds, median-of-medians; no API keys) runs monthly in CI on the same neutral runner. 2026-07 results, added latency per request: **Bifrost 0.56 ms** (IQR 0.51–0.58) · **Portkey Gateway OSS v1.15.2 2.69 ms** (2.56–2.82) · **LiteLLM v1.91.0 5.41 ms** (5.38–5.60). Read against marketing: Bifrost's "fastest" holds directionally (~10× vs LiteLLM — not the marketed 50×, which refers to loaded throughput); Portkey's "<1 ms" didn't reproduce on shared CI hardware (it did on a fast desktop: 0.47 ms). Data: [`llm-gateway-bench/data/overhead.json`](https://github.com/cuihuan/llm-gateway-bench/blob/main/data/overhead.json) · [methodology](https://github.com/cuihuan/llm-gateway-bench/blob/main/docs/methodology.md) — PRs adding Kong/Envoy/Higress are welcome. > 🔌 **Protocol-translation fidelity, independently measured (2026-07).** The most-reported gateway failure in real bug trackers isn't routing — it's the gateway **corrupting tool-calls / streaming / usage in translation** ("claude code" appears in 400+ LiteLLM issues; the top-commented issues on Portkey/OpenRouter/Bifrost are all this). So this project measures it: a mock upstream returns spec-correct responses (a tool_call, a genuine multi-chunk SSE stream with `usage`), and the harness checks what actually arrives **through** each gateway (no keys, reproducible in CI). Result, self-hosted at a custom upstream: **LiteLLM v1.91.0 — 3/3 · Bifrost — 3/3 · Portkey Gateway OSS v1.15.2 — 1/3.** All three relay a tool_call intact; but **Portkey OSS in custom-host mode threw an internal error on *every* streaming request** (client got 0 chunks, no `usage`) while non-streaming worked — reproduced on a clean CI runner. Fair caveats: this is the OSS gateway's *self-host custom-host path* (Portkey's hosted product / standard provider integrations may stream fine, and v2.0 is unreleased), and the test is OpenAI-format **passthrough** only — cross-format (Anthropic `tool_use` id-remapping) translation, where the hardest bugs live, is future work. **Takeaway: before committing, run your actual agent — tools + streaming — through the gateway, not a hello-world.** Data: [`llm-gateway-bench/data/fidelity.json`](https://github.com/cuihuan/llm-gateway-bench/blob/main/data/fidelity.json). > 🏠 **Self-hosted shifts the burden to you.** LiteLLM/Bifrost/Kong score on the *controls they hand you* (RBAC, audit logs, key vaulting, on-prem) — but SOC 2 / HIPAA compliance of the *deployment* is yours to earn. That's the trade for $0 markup and full data control. --- ## Part 5 — Real-world reviews: what production users report Benchmarks rank capability; this ranks **what actually breaks once a gateway is in production**. Sourced from incident postmortems, status pages, security research and acquisition news — every dated event below links to a primary or recognized source, summarized fairly (what users praise *and* complain about). Read it alongside the [scorecard](#part-4--gateway-scorecard-compliance--price--security--stability--observability): stars measure popularity; this measures the 3am pager. | Gateway | Praised for | Recurring gripe | Dated event worth knowing | |---|---|---|---| | **LiteLLM** | Default OpenAI-compatible multi-provider proxy; unmatched model breadth | Latency/memory overhead and degradation at high RPS | ⚠️ **PyPI supply-chain attack (Mar 2026)** — v1.82.7/1.82.8 were backdoored with a credential stealer after a Trivy CI-token compromise (TeamPCP); PyPI quarantined the project in ~3h. Pin to a clean release. [Trend Micro](https://www.trendmicro.com/en/research/26/c/inside-litellm-supply-chain-compromise.html) · [Datadog](https://securitylabs.datadoghq.com/articles/litellm-compromised-pypi-teampcp-supply-chain-campaign/) | | **OpenRouter** | 400+ models behind one key and one bill; hard spend caps | ~5.5% credit fee at scale; provider quality/quantization varies; no public SLA | **~50-min database outage (Aug 28 2025)**, then two more on **Feb 17 & 19 2026** — all DB-layer failures, not upstream providers (the Feb pair: the caching layer dropped every DB connection, returning 401s with ~80–90% request failures); ~99.96% uptime over 8 months, but no SLA or downtime credits. [StatusGator](https://statusgator.com/services/openrouter) · [Feb 2026 postmortem](https://openrouter.ai/blog/announcements/openrouter-outages-on-february-17-and-19-2026/) | | **Portkey** | Genuinely production-grade; strong observability, fallbacks, prompt management | Much "review" content is SEO — weigh G2/Gartner over listicles | **Acquired by Palo Alto Networks (closed May 2026)** — now the control plane for Prisma AIRS; a neutrality/roadmap question if you want vendor-independent infra. [Palo Alto Networks](https://www.paloaltonetworks.com/company/press/2026/palo-alto-networks-completes-acquisition-of-portkey-to-secure-ai-agents) | | **Vercel AI Gateway** | One endpoint, 0% markup on BYOK, native to the AI SDK | Durable independent sentiment still thin | **Vercel breach (Apr 2026)** — a Context.ai OAuth supply-chain compromise exposed environment variables for a subset of customers. Not the gateway product itself, but it bears on trusting Vercel with your keys. [TechCrunch](https://techcrunch.com/2026/04/20/app-host-vercel-confirms-security-incident-says-customer-data-was-stolen-via-breach-at-context-ai/) · [Vercel](https://vercel.com/kb/bulletin/vercel-april-2026-security-incident) | | **Cloudflare AI Gateway** | Zero-infra observability / caching / rate-limiting on CF's edge; dollar spend limits | Limited request-placement / routing control | Newer entrant; independent production-review depth is still thin. | | **Kong AI Gateway** | Inherits a battle-tested data plane + broad plugin ecosystem | Open-core: several important plugins are Enterprise-only (some prefer Apache APISIX for predictable OSS) | The AI Gateway is newer; most Kong-vs-LiteLLM/Portkey benchmarks are vendor-published — reproduce before trusting. | | **TensorZero** | Ambitious OSS gateway + observability + optimization "flywheel" | — | ⚠️ **Archived June 2026** — company wound down; Apache-2.0 code and community forks remain, but plan for self-support, not a vendor roadmap. [GitHub](https://github.com/tensorzero/tensorzero) | | **Helicone** | ~2-minute onboarding; fast cost/usage and token-limit debugging | As a proxy it sits in the request path (a SPOF unless you use async logging) | **Acquired by Mintlify (Mar 2026), now maintenance mode** — security fixes, bug fixes and new models keep shipping; Mintlify supports migration. [Mintlify](https://www.mintlify.com/blog/mintlify-acquires-helicone) · [Helicone](https://www.helicone.ai/blog/joining-mintlify) | | **Bifrost** | Markets itself as a fast Go drop-in for LiteLLM | Independent production evidence is still thin | Its 50–90× lower-p99 claims are **vendor-reported** — reproduce on your own workload before betting on them. | > **How to read this.** Every gateway here is a *legitimate* project with a real maintainer or company — the kind this list includes. We surface their warts on purpose: a gateway with *no* public criticism usually just has no public users. These notes are point-in-time and dated; sources move fast, so verify before you sign. --- ## Part 6 — Gateway observability: the factors that matter *Why this is its own axis — separate from the [scorecard](#part-4--gateway-scorecard-compliance--price--security--stability--observability) and from generic APM: a gateway sits between many internal consumers and metered, $-per-token providers, so the unit of analysis is **tokens and dollars attributed per key / team / user / model** — and the gateway's own value-add (retries, fallback, caching, guardrails) actively **masks** cost and failure unless it's instrumented. The cross-vendor standard is the [OpenTelemetry GenAI semantic conventions](https://github.com/open-telemetry/semantic-conventions-genai) (`gen_ai.*` spans + metrics), now natively consumed by Datadog/Honeycomb/Grafana — but most `gen_ai.*` attributes are still **"Development"** status in 2026, and several heavily-marketed capabilities (online evals, drift detection) are real **product features, not standards**. This rubric grades what a gateway actually exposes, flags standardized vs aspirational, and stays neutral. The premise is survey-backed: cost is the **#2 most-monitored production metric** after quality/task success, with nearly half of teams actively monitoring cost in prod ([Amplify 2026 AI Engineering Report](https://www.amplifypartners.com/blog-posts/the-2026-ai-engineering-report), n>1,000).* ### Table-stakes — miss these and it's barely instrumented (mostly maps to Required/Stable OTel) | Factor | What a well-instrumented gateway exposes | How to verify | |---|---|---| | **Core inference telemetry** | `gen_ai.operation.name` + `gen_ai.provider.name` on every span (**both Required**), span named `{op} {model}`, and **both** `gen_ai.request.model` (alias requested) **and** `gen_ai.response.model` (concrete model that answered) — so a silent reroute / alias-swap is visible. | Call two upstreams via a model alias; confirm distinct `provider.name`, and `request.model`=alias vs `response.model`=resolved version. | | **Token & $ cost attribution** | input/output token counts + the `gen_ai.client.token.usage` histogram split by `gen_ai.token.type`; cost read from the **provider usage payload** (not just model-name inference), broken down by token type (prompt / completion / cached-read / reasoning) and rolled up per key/team/user/model. | Two identical calls under different keys → separate cost; a cache-read costs less than a miss; an **unknown model still yields token counts**, not a silent $0. | | **Decomposed latency** | total **and** upstream-provider **and** gateway-overhead as separate numbers, plus **TTFT** on streams, as **histograms** (p95/p99) — `gen_ai.client.operation.duration` is the **only** metric the spec marks Required. | One trace shows total/upstream/overhead separately; a streaming call records TTFT distinctly; the metric is a histogram, not an average gauge. | | **Error taxonomy by origin** | failures carry the **Stable** `error.type` + provider status + class, separated by origin (client/gateway rejection vs upstream failure vs guardrail block) — not one flat 5xx counter. | Trigger a provider 429, an oversized-prompt 400, a budget/auth rejection, and a guardrail block → each labeled distinctly. | | **Open export / no lock-in** | emits **and** ingests OTLP (`gen_ai.*` / OpenInference), a Prometheus `/metrics` endpoint, webhooks, and bulk raw export to S3/warehouse — not a dashboard-only silo (a real risk after the 2026 shakeout: TensorZero archived, Helicone→Mintlify maintenance, Portkey→Palo Alto). | Point its exporter at a throwaway OTel Collector; `curl /metrics`; configure a test webhook; ask for a documented warehouse export. | | **Cardinality discipline** | metrics use only **bounded** labels (model / provider / region / status / deployment); unbounded ids (prompt text, conversation/request id, raw user id) live in trace/log attributes, **never** labels — else the monitoring bill exceeds the inference bill. | Scrape `/metrics` and look for any unbounded label; ask "what's my metrics bill at 10k RPS with 1M distinct users?" | ### Differentiating — instruments the gateway's own value-add (which otherwise hides cost & failure) | Factor | What a well-instrumented gateway exposes | How to verify | |---|---|---| | **Reliability visibility** (retry / fallback / failover / cooldown) | counters for successful vs failed fallbacks labeled `requested→fallback` model, retry-attempt counts, circuit-breaker/cooldown events — because a request that succeeds after 3 retries + a fallback otherwise looks like a clean 200. | Force the primary to 500 → trace shows retry spans, the fallback source→target, and which deployment finally served. | | **Cache visibility** | per-request HIT/MISS + hit-rate % + cost/time saved; exact **and** semantic caching; provider prompt-cache read/write tokens tracked **separately** from the gateway's own response cache. | Send the same request twice → 2nd flagged HIT with lower cost/latency; dashboard shows hit rate + cumulative savings. | | **Budget / quota / rate-limit telemetry** | live remaining-budget gauges per key **and** team ($ + hours-to-reset), remaining RPM/TPM per model, upstream provider rate-limit headroom, graduated alerts, and a **hard cutoff** at the limit (not just an alert). | Set a small team budget, drive spend → gauge ticks down, alert fires, **and traffic is actually cut off**. | | **Streaming lifecycle** | a `gen_ai.request.stream` flag, TTFT + per-output-token latency, and **mid-stream failure** detected as a failure/partial (a stream can send 200 OK then stall or error). | Stream a request → TTFT distinct from total; cut the connection mid-stream → recorded as failure/partial, not a clean success. | | **Prompt/response capture *with* redaction** | content capture **off by default** (the OTel default, toggled via `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT`), landing in `gen_ai.input.messages` / `output.messages`; paired with **PII redaction before storage and before the provider call**. Raw logging without redaction can breach GDPR + the EU AI Act (high-risk provisions enforceable **2026-08-02**). | With capture unset, only metadata appears; a planted SSN/email is masked on both input **and** output. | ### Advanced — the 2026 frontier, mostly **non-standard product features** (grade as bonus, verify claims) | Factor | What a well-instrumented gateway exposes | Honest caveat | |---|---|---| | **Retention + tail/eval-driven sampling** | per-data-class retention (keep metrics long, expire raw prompts fast) + sampling that keeps **100% of errors** and high-cost/anomalous traces while sampling the boring bulk — head-only sampling drops the rare hallucination you needed. | Only OTLP transport is standardized; the **sampling policy is product-specific**. | | **Eval / quality & drift signals** | online evals (LLM-as-judge or programmatic) attachable to live traces and chartable over time, + capture of the **resolved** `gen_ai.response.model` so a silent provider re-point is visible, + trace→eval behind a CI gate. | Online evals / LLM-as-judge are **vendor product features, NOT the OTel standard** — only `gen_ai.response.model` is standardized. Treat vendor eval/latency claims as vendor-reported until reproduced. | **Exemplars among listed gateways** (illustrative, not endorsements): **LiteLLM** ships the reference self-hosted label set (`litellm_spend_metric`, cached/reasoning token splits, distinct request/upstream/overhead latency, fallback + budget metrics, Prometheus + Grafana). **Helicone**, **Langfuse**, **Arize Phoenix**, **Pydantic Logfire** and **Braintrust** are observability-first (OTLP-native, cost-by-segment, evals). **Portkey** exposes OTLP export + hard budget enforcement + cache/guardrail telemetry. **Kong AI Gateway** maps the `gen_ai.*` span set and PII sanitization. **Cloudflare AI Gateway** adds spend limits + free DLP/PII scan. *Verify against your own workload — most performance/eval figures are vendor-reported.* > **Questions to ask before you trust a gateway's dashboard** > - Show me a sample exported span: do you **emit and ingest OTLP** with `gen_ai.*` / OpenInference names, or is telemetry proprietary-only? > - Is there a Prometheus `/metrics` endpoint, and are all labels **bounded**? (Any unbounded label — prompt text, conversation/request id, raw user id — is a walk-away.) > - Can cost be attributed **simultaneously per virtual-key / team / user / model AND by token type**, read from the provider usage payload (not just inferred from the model name, which breaks on new/renamed models)? > - Can I read **total, upstream, and gateway-overhead latency separately**, with TTFT on streams, as histograms (p95/p99) — not one average gauge? > - When a request succeeds only after retries + a fallback, does the trace show the **retry spans and the fallback source→target**, or just a 200? > - Do budgets **hard-cut-off** at the limit or only alert, and where do alerts route? > - Is content capture **off by default**, with PII redacted **before storage and before the provider sees it**? > - Do you record the **resolved** `gen_ai.response.model`, so a silent provider re-point or model re-tune is detectable? > - Under ZDR / self-host, **what observability do I lose** (typically you keep metadata/metrics, drop prompt bodies)? > **Grounded in** the [OpenTelemetry GenAI semantic conventions](https://github.com/open-telemetry/semantic-conventions-genai) ([spans](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-spans.md) · [metrics](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-metrics.md)). Most `gen_ai.*` attributes are still **Development** status in 2026 (`error.type` / `server.*` are Stable) — pin `OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental` so dashboards don't silently break across releases. Reference label sets: [LiteLLM Prometheus](https://docs.litellm.ai/docs/proxy/prometheus), [OpenLLMetry/Traceloop](https://github.com/traceloop/openllmetry), [OpenInference](https://github.com/Arize-ai/openinference). Last reviewed **2026-06-25**. --- ## Methodology & caveats - **Benchmarks are necessary, not sufficient.** Public sets leak into training data (contamination), and vendors optimize for the leaderboard. We therefore show *several* benchmarks + blind human preference (Arena) + real cost, and weight none of them alone. - **"Verified" matters.** We prefer SWE-bench **Verified** over the raw set, and official model cards / [Artificial Analysis](https://artificialanalysis.ai) independent runs over vendor press releases. Vendor-reported numbers are labeled. - **Cost ≠ price.** Headline $/token hides reasoning-token inflation, cached-input discounts and batch pricing. Part 3 prices *workloads*, not tokens, and the script is open for you to plug your own token mix. - **Gateway scores are point-in-time estimates** from public trust pages, status pages and docs. Compliance certs lapse and ship; verify before you sign. Corrections welcome via PR — see [CONTRIBUTING](CONTRIBUTING.md). - **No affiliation.** This list takes no money from any vendor listed. Self-hosted and commercial options are scored on the same axes. ## Sources Primary leaderboards and pricing references (verify live — these move weekly): - [LMArena](https://lmarena.ai) — blind human-preference Elo - [Artificial Analysis](https://artificialanalysis.ai) — Intelligence Index, price & speed - [SWE-bench](https://www.swebench.com) — agentic coding leaderboard - [Vellum LLM Leaderboard](https://www.vellum.ai/llm-leaderboard), [OpenRouter rankings](https://openrouter.ai/rankings) - **Agentic & tool-calling:** [Terminal-Bench](https://www.tbench.ai/leaderboard) (shell/CLI agents), [τ²-bench](https://github.com/sierra-research/tau2-bench) (tool-agent policy adherence), [BFCL v4](https://gorilla.cs.berkeley.edu/leaderboard.html) (Berkeley function-calling), [Aider polyglot](https://aider.chat/docs/leaderboards/) (multi-language code edits) - **Contamination-resistant / frontier:** [LiveBench](https://livebench.ai) & [LiveCodeBench](https://livecodebench.github.io/leaderboard.html) (questions refreshed monthly), [FrontierMath](https://epoch.ai/frontiermath) (research-level math) - Official pricing: [Anthropic](https://www.anthropic.com/pricing), [OpenAI](https://openai.com/api/pricing/), [Google](https://ai.google.dev/pricing), [DeepSeek](https://api-docs.deepseek.com/quick_start/pricing) - Observability standard (Part 6): [OpenTelemetry GenAI semantic conventions](https://github.com/open-telemetry/semantic-conventions-genai) — `gen_ai.*` spans & metrics Per-cell sources are listed in [`data/models.json`](data/models.json) and the gateway research notes. --- *Maintained as part of [Awesome AI Gateway](README.md). Model scores and prices change fast; this set is reviewed on a published cadence and every figure is dated at its source.* **Last reviewed: 2026-07-06** · benchmark & pricing snapshot in [`data/models.json`](data/models.json), gateway scores in [`data/gateways_eval.json`](data/gateways_eval.json).