vLLM on Neuron¶
A collection of guides for serving LLMs on AWS Trainium using the vLLM Neuron Plugin.
-
Model Onboarding
The full process for adding a new model architecture to the vLLM Neuron Plugin. Includes the 2-phase approach (NF assembly → NKI optimization), the 5-step workflow, and an architecture diff analysis script.
-
vLLM Neuron Plugin Internals · Coming soon
Plugin architecture, Model Executor structure, prefill/decode separation, and how continuous batching works.
Coming soon
-
vLLM Upstreaming · Coming soon
How vLLM Neuron integrates with upstream vLLM, and the plugin structure and separation strategy.
Coming soon
-
Serving Recipes by Model · Coming soon
Serving configurations and benchmarks for validated models such as Llama 3.1 70B, Qwen2.5 72B, and Qwen3 32B.
Coming soon
References¶
| Resource | Link |
|---|---|
| Official documentation | Neuron Docs: vLLM Neuron |
| GitHub | vllm-project/vllm-neuron |
| NF API source | vllm_neuron/functional |
| Model Recipes | Neuron Docs: Model Recipes |