--- title: "How to Evaluate Any MCP Gateway" description: "A practical guide to evaluating MCP gateways (Kong, Envoy, APISIX): a 5-point rubric covering enterprise fit, integration, reliability, cost, docs." source: "https://agntc.work/blog/mcp-gateways-verified/" pubDate: "2026-09-16T00:00:00.000Z" category: "platforms-tools" author: "Erez Eden" proficiencyLevel: "Advanced" tags: ["MCP","Gateway","Enterprise","Review","Benchmark"] --- # Executive Summary A practical 5-point rubric (enterprise fit, integration, reliability, cost clarity, documentation) for evaluating any MCP gateway — Kong, Envoy Gateway, Apache APISIX, or newer — mapped from Anthropic's evaluation framework graders. Not marketing claims. # Key Takeaways - No single gateway covers SSO, RBAC, and audit out of the box — expect to build integration layers regardless of vendor. - Test the error contract before buying: typed errors (`not_found`, `rate_limited`) beat thrown exceptions for agent reliability. - Measure gateway cost as `cost per task` (Artificial Analysis, Sep 2026) — total verified outcome cost, not advertised query price. --- Every team deploying agents hits the same procurement question: which gateway connects agents to our internal systems safely? Not just "does it support MCP" — does it enforce permissions, return typed errors, log audit events, and expose cost clearly? That is what a practical evaluation measures, not what a vendor's landing page claims. As of September 2026, the vocabulary is settling: MCP is the standard plug (`https://modelcontextprotocol.io`), but the gateway — Kong Gateway (`docs.konghq.com/gateway`, v3.9+), Envoy Gateway (`envoyproxy.io/docs/gateway`), Apache APISIX (`apisix.apache.org`) — is still the decision point. Anthropic's evaluation framework (2026) separates graders by dimension (`deterministic_tests`, `llm_rubric`, `state_check`, `transcript`, `latency`); the same separation applies here. Enterprise fit (SSO, RBAC, audit) is a gate, integration effort (typed errors, docs) is a rubric, reliability (observed failures) is a contract, cost clarity is its own grader (`cost per task`, Artificial Analysis, Sep 2026 — see [Intelligence vs Cost per Task chart](https://artificialanalysis.ai/models?cost=intelligence-vs-cost-per-task#cost-tabs)), and documentation quality rounds out the rubric. No single score captures any gateway. A builder in a community forum put it plainly: you are really evaluating two things at once — the protocol (does it connect?) and the gateway (does it enforce, verify, and explain?). This guide tests both. ## What to check before buying (the 5 graders) Use this rubric for any gateway you are considering — Kong, Envoy, APISIX, or a newer MCP-specific option. Score each dimension 1-5 based on what you can verify, not what the docs claim. **1. Enterprise fit (gate)** — Does the gateway enforce `auth_blocked` before routing? Does it support your SSO provider (SAML, OIDC, or LDAP)? Is RBAC enforced at the gateway layer, or delegated to the upstream service? A gateway that skips the audit log (`audit_event`) is not a gateway — it is a protocol adapter that skips the harness. **2. Integration effort (rubric)** — Are errors typed (`not_found`, `rate_limited`) or thrown exceptions? Does the schema reference cover all endpoints with versions? How long does "first working call" take with the docs alone? Kong's docs (`docs.konghq.com`) are versioned and searchable; Envoy Gateway's (`envoyproxy.io`) cover the core but leave some MCP-specific endpoints undocumented; APISIX (`apisix.apache.org`) requires plugin documentation for full coverage. **3. Reliability (contract)** — After `create_user()`, does the audit log show the event? Does retry corrupt state? Does the gateway handle errors deterministically? This is the same `state_check` grader from the evaluation framework — verify it with a test script (`generate_invoice()` + `audit_event`), not a marketing benchmark. **4. Cost clarity (grader)** — Not "$ per query" but `cost per task`: the weighted average cost per verified outcome (`artificialanalysis.ai/models`, Sep 2026), including integration time and missing audit hooks. A gateway with $0 advertised price that requires a full-time engineer for audit integration is not free. **5. Documentation (rubric)** — Versioned docs (`v3.9`, `v2.3` etc.), searchable error reference, complete schema, and no missing endpoints. Check the docs for the version you will deploy — not the "latest" page, which may reference unreleased features. ## Real gateway categories (as of Sep 2026) These are the categories you will actually compare — not fictional scores, but real architectures with documented trade-offs: - **[Kong Gateway](https://docs.konghq.com/gateway/) (`konghq.com`, v3.9+)** — Full plugin ecosystem (SSO, RBAC, audit, rate limiting). Integration takes a senior engineer. Audit hooks (`security_logs`) exist but require custom plugin configuration. Cost: transparent licensing; integration cost is the hidden variable. - **[Envoy Gateway](https://envoyproxy.io/docs/gateway) (`envoyproxy.io`)** — Fast setup; typed errors (`not_found`) are first-class. RBAC is weaker at the gateway layer than Kong — filter at retrieval, not after routing. Documentation covers core well; MCP-specific endpoints need upstream confirmation. - **[Apache APISIX](https://apisix.apache.org) (`apisix.apache.org`)** — Open-source; strong plugin support for auth and audit. Schema docs require plugin-level verification. Integration effort varies by plugin version — verify the exact plugin version (`v2.14` etc.) against the gateway version. - **Newer MCP-specific gateways** — Emerging; often skip native SSO or audit. Treat them as protocol adapters until the harness contracts (`auth_blocked`, `audit_event`) pass deterministically. No gateway covers all five dimensions well out of the box. That is not a vendor failure — it is the architecture: the harness (gateway + agent + permission model) must match your security review, not the vendor's marketing page. A framework you can run in a week and apply continuously is more valuable than a perfect comparison that takes six months. **Note:** This review applies the rubric above to the gateway category, not a specific tested version. Product-level scores with version stamps (`v3.9`, `v2.3` etc.), observed failure modes, and `last-verified` dates require a separate `T4` comparison — see site workflow. **Last verified:** Sep 2026. References checked: Kong docs (`konghq.com/gateway`), Envoy Gateway docs (`envoyproxy.io/docs/gateway`), Apache APISIX docs (`apisix.apache.org`), Anthropic eval framework (2026), Artificial Analysis (`artificialanalysis.ai/models`). ## Building the framework for your team Start with the same three questions the evaluation framework asks: 1. What hard failures must never pass? (enterprise fit gate — `auth_blocked`, SSO match) 2. What state must exist after the agent acts? (`state_check` — `audit_event` in log, no duplicate records) 3. How does behavior drift? (`transcript` — retry patterns, latency, error frequency over time) Then aggregate: score each gateway against the 5 graders using the contracts you can actually test (`auth_blocked` event, `create_user()` state, `not_found` error). A gateway that passes contracts 1 and 3 but scores low on 2 or 4 is a governance success but an engineering burden. A gateway scoring low on 1 or 3 is not a gateway — it skips the harness. A framework that takes one week to build and runs for months is worth more than a perfect vendor comparison that takes half a year. The rubric is the operating model. ## Further reading - [Agent evaluation patterns (Enterprise Context Engine)](/blog/enterprise-context-engine): how evaluation contracts protect the harness - [How to Measure Cost Per Agent Outcome](/blog/agent-cost-per-outcome): cost per verified outcome, not per query - [More on AI-first operations](/category/ai-first-operations/): the operating model where evaluation is the control plane