Overview
This guide shows how to run open source inference frameworks on SageMaker HyperPod and reuse HyperPod managed tiered KV cache through LMCache.
Managed KV Cache
All framework examples in this section use the same LMCache configuration to connect to the HyperPod managed tiered-storage daemon.
Dynamo
This example runs a Dynamo frontend and Dynamo vLLM worker on HyperPod. The vLLM worker uses LMCache to read and write KV cache through the HyperPod managed tiered-storage daemon.
llm-d
This example uses llm-d as the serving framework and configures its vLLM model server to use LMCache with HyperPod managed tiered storage.
SGLang
This example runs SGLang on HyperPod and uses LMCache to read and write KV cache through the HyperPod managed tiered-storage daemon.
Validate Cache Reuse
The core test is the same for every framework:
Troubleshooting
LMCache Looks For shared_memory