📄️ Overview
This guide shows how to run open source inference frameworks on SageMaker HyperPod and reuse HyperPod managed tiered KV cache through LMCache.
📄️ Managed KV Cache
All framework examples in this section use the same LMCache configuration to connect to the HyperPod managed tiered-storage daemon.
📄️ Dynamo
This example runs a Dynamo frontend and Dynamo vLLM worker on HyperPod. The vLLM worker uses LMCache to read and write KV cache through the HyperPod managed tiered-storage daemon.
📄️ llm-d
This example uses llm-d as the serving framework and configures its vLLM model server to use LMCache with HyperPod managed tiered storage.
📄️ SGLang
This example runs SGLang on HyperPod and uses LMCache to read and write KV cache through the HyperPod managed tiered-storage daemon.
📄️ Validate Cache Reuse
The core test is the same for every framework:
📄️ Troubleshooting
LMCache Looks For shared_memory