For the complete documentation index, see llms.txt. Markdown versions of all docs pages are available by appending .md to any docs URL.

vLLM Semantic Router

Route OpenAI-compatible requests by prompt semantics with vLLM Semantic Router and agentgateway.

vLLM Semantic Router (vSR) classifies OpenAI-compatible LLM requests and selects a model based on prompt content. With agentgateway, you can make this semantic decision before routing while continuing to apply gateway policies and record model, token, latency, and cost telemetry.

This integration is distinct from using vLLM as an inference provider. vSR provides the model-selection policy. Your configured provider, Kubernetes Service, or InferencePool serves the selected model.

How the integration works

The following diagram shows the cost-based routing example. A coding agent requests the stable auto model, vSR selects a lower-cost or higher-capability model, and agentgateway forwards the request and records the result.

The request follows these component boundaries:

1. A client sends an OpenAI-compatible request to agentgateway.
2. An AgentgatewayPolicy calls vSR as an external processor during the PreRouting phase.
3. vSR evaluates its semantic, complexity, keyword, context, and structure signals. It returns the selected model in its processing response.
4. Agentgateway applies the routing decision and forwards the request to the configured provider or inference workload.
5. Agentgateway records the requested and selected models alongside usage, latency, and optional catalog-priced cost data.

PreRouting is important when the vSR decision changes the model or adds a header that an HTTPRoute uses for matching. It makes the result available before agentgateway evaluates the route.

Choose an integration path

The vSR and agentgateway projects provide complementary guides. Choose the one that matches the models and outcome that you want to evaluate.

The vSR deployment guide owns the installation, Helm values, and semantic-router configuration. The agentgateway example owns the cost-routing policy and runnable gateway resources. Keeping those details with their projects avoids version drift in this integration overview.

Integration considerations

• Client model selection: The cost-based example uses model: "auto" to opt in to semantic selection. This value is an example policy convention, not a reserved agentgateway model. You can let clients request model tiers directly or validate the request body to require the automatic path.
• Backend choice: vSR can select models served by hosted providers or Kubernetes inference workloads. Configure the corresponding LLM provider or routing backend in agentgateway.
• Model names: Keep the names returned by vSR aligned with the models in your agentgateway routes, provider configuration, and cost catalog. You can expose stable client-facing names with model aliases.
• Cost and observability: vSR makes the semantic decision. Agentgateway remains the source for completed-request telemetry and can calculate realized cost when you configure a model cost catalog. Use LLM metrics and logs or the OpenTelemetry stack to evaluate the result.

Before a broad rollout, compare routed traffic with a fixed higher-capability-model baseline. Confirm that the policy uses both tiers, then evaluate task completion, user feedback, retries, and escalation rates alongside cost and latency.