📄️ Overview
This guide shows how to serve several models behind a single OpenAI-compatible endpoint on SageMaker HyperPod, and how the gateway decides where each request goes.
📄️ First Gateway
This page deploys one model, puts a single-scheduler gateway in front of it, and sends the first completion through it. Start here before adding a second model.
📄️ Gateway Anatomy
Applying one InferenceGatewayConfig caused the controller to create a number of resources across two namespaces. This page maps each of them back to the architecture, so that later troubleshooting is not guesswork.
📄️ Multi-Model Routing
This page adds a second model and turns on body-based routing (BBR), so that a single endpoint serves both models and routes each request by the model field in its body. You will then prove from logs that routing was correct.
📄️ Endpoint Picking
Body-based routing decides which pool serves a request. Within that pool, the endpoint picker decides which pod. Everything so far ran at one replica per model, so that second decision had only one candidate. This page gives it a real choice.
📄️ Cleanup and Troubleshooting
1. Cleanup