Examples¶

Concise walkthroughs for each deployment configuration. Every example follows the same pattern: generate, build, deploy, test. For a full end-to-end tutorial, see Getting Started.

HTTP: sklearn + Flask¶

Deploy a scikit-learn model with Flask serving on a CPU instance.

ml-container-creator sklearn-flask-demo \
  --deployment-config=http-flask \
  --model-format=pkl \
  --include-sample \
  --instance-type=ml.m5.large \
  --build-target=codebuild \
  --region=us-east-1 \
  --skip-prompts

Copy your own model into the project (or use the generated sample):

cp /path/to/model.pkl code/model.pkl

Build, push, deploy:

./do/build
./do/push
./do/deploy
./do/test

HTTP: XGBoost + FastAPI¶

Deploy an XGBoost model with FastAPI serving.

ml-container-creator xgboost-fastapi-demo \
  --deployment-config=http-fastapi \
  --model-format=json \
  --include-sample \
  --instance-type=ml.m5.large \
  --build-target=codebuild \
  --region=us-east-1 \
  --skip-prompts

./do/build && ./do/push && ./do/deploy && ./do/test

HTTP: TensorFlow + Flask¶

Deploy a TensorFlow SavedModel with Flask serving.

ml-container-creator tf-flask-demo \
  --deployment-config=http-flask \
  --model-format=SavedModel \
  --include-sample \
  --instance-type=ml.m5.large \
  --build-target=codebuild \
  --region=us-east-1 \
  --skip-prompts

./do/build && ./do/push && ./do/deploy && ./do/test

Transformers: vLLM¶

Deploy an LLM with vLLM. GPU instance required.

ml-container-creator vllm-demo \
  --deployment-config=transformers-vllm \
  --model-name=openai/gpt-oss-20b \
  --instance-type=ml.g6.12xlarge \
  --build-target=codebuild \
  --region=us-east-1 \
  --skip-prompts

LLM containers are large. Use CodeBuild for the image build:

./do/submit    # Build and push via CodeBuild
./do/deploy
./do/test

For gated models (e.g., Llama), store the token in Secrets Manager and pass --hf-token-arn <secret-arn> (or set it once with mcc bootstrap add-secret hfToken <secret-arn>). See Secrets and HuggingFace Authentication.

Transformers: SGLang¶

Deploy an LLM with SGLang. Same workflow as vLLM with a different deployment config.

ml-container-creator sglang-demo \
  --deployment-config=transformers-sglang \
  --model-name=openai/gpt-oss-20b \
  --instance-type=ml.g6.12xlarge \
  --build-target=codebuild \
  --region=us-east-1 \
  --skip-prompts

./do/submit && ./do/deploy && ./do/test

Transformers: TensorRT-LLM¶

Deploy an LLM with NVIDIA TensorRT-LLM. Requires NGC authentication for the base image and A10G or newer GPUs (ml.g5 instances, not ml.g6).

ml-container-creator trtllm-demo \
  --deployment-config=transformers-tensorrt-llm \
  --model-name=meta-llama/Llama-3.2-3B-Instruct \
  --instance-type=ml.g5.12xlarge \
  --build-target=codebuild \
  --region=us-east-1 \
  --skip-prompts

Set your NGC API key before building:

export NGC_API_KEY='your-ngc-api-key'
./do/submit && ./do/deploy && ./do/test

The generated container runs TensorRT-LLM on port 8081 behind an Nginx reverse proxy on port 8080 for SageMaker compatibility. Key environment variables for tuning:

Variable	Default	Description
`TRTLLM_TP_SIZE`	`1`	Tensor parallelism (set to GPU count)
`TRTLLM_MAX_BATCH_SIZE`	`256`	Maximum batch size
`TRTLLM_MAX_INPUT_LEN`	`2048`	Maximum input token length
`TRTLLM_MAX_OUTPUT_LEN`	`512`	Maximum output token length

Transformers: LMI (Large Model Inference)¶

Deploy an LLM with AWS Large Model Inference (DJL-based). Uses serving.properties for configuration instead of environment variables.

ml-container-creator lmi-demo \
  --deployment-config=transformers-lmi \
  --model-name=openai/gpt-oss-20b \
  --instance-type=ml.g5.12xlarge \
  --build-target=codebuild \
  --region=us-east-1 \
  --skip-prompts

./do/submit && ./do/deploy && ./do/test

Transformers: DJL¶

Deploy an LLM with Deep Java Library serving.

ml-container-creator djl-demo \
  --deployment-config=transformers-djl \
  --model-name=openai/gpt-oss-20b \
  --instance-type=ml.g5.12xlarge \
  --build-target=codebuild \
  --region=us-east-1 \
  --skip-prompts

./do/submit && ./do/deploy && ./do/test

Transformers: llama.cpp (GGUF, CPU or GPU)¶

Deploy a quantized GGUF model with the AWS DLC for llama.cpp. The container serves an OpenAI-compatible API via llama-server and reads SM_LLAMA_CPP_* environment variables, which it maps to llama.cpp arguments itself (MLCC does not translate them). Works on CPU instances for cost-sensitive or background workloads, or on GPU with full layer offload for low latency.

CPU (the default DLC image). Point it at a HuggingFace GGUF repo and the specific .gguf file, and tune CPU threads:

ml-container-creator llamacpp-cpu-demo \
  --deployment-config=transformers-llama-cpp \
  --model-name=Qwen/Qwen3-4B-GGUF \
  --server-env=HF_FILE=Qwen3-4B-Q4_K_M.gguf \
  --engine-feature=threads=32 \
  --instance-type=ml.c6i.8xlarge \
  --build-target=codebuild \
  --region=us-east-1 \
  --skip-prompts

GPU — select the CUDA DLC image and offload every layer (gpu_layers=-1):

ml-container-creator llamacpp-gpu-demo \
  --deployment-config=transformers-llama-cpp \
  --model-name=Qwen/Qwen3-4B-GGUF \
  --server-env=HF_FILE=Qwen3-4B-Q4_K_M.gguf \
  --base-image=763104351884.dkr.ecr.us-east-1.amazonaws.com/llama-cpp:server-sagemaker-cuda-v1 \
  --engine-feature=gpu_layers=-1 \
  --instance-type=ml.g6e.xlarge \
  --build-target=codebuild \
  --region=us-east-1 \
  --skip-prompts

./do/submit && ./do/deploy && ./do/test

--engine-feature knobs for llama.cpp: gpu_layers (layers to offload to GPU; -1 = all, 0 = CPU-only), threads (CPU generation threads), and flash_attn (enable the Flash Attention kernel). You can also package a .gguf directly in the SageMaker model artifact under /opt/ml/model instead of downloading from HuggingFace.

Triton: FIL (Tree Models)¶

Deploy XGBoost or LightGBM models on NVIDIA Triton Inference Server using the Forest Inference Library backend.

ml-container-creator triton-fil-demo \
  --deployment-config=triton-fil \
  --model-format=json \
  --instance-type=ml.g5.xlarge \
  --build-target=codebuild \
  --region=us-east-1 \
  --skip-prompts

The generator creates a Triton model repository layout with config.pbtxt. Place your model file in the generated model_repository/ directory before building.

./do/build && ./do/push && ./do/deploy && ./do/test

Triton: ONNX Runtime¶

Deploy ONNX models on Triton.

ml-container-creator triton-onnx-demo \
  --deployment-config=triton-onnxruntime \
  --instance-type=ml.g5.xlarge \
  --build-target=codebuild \
  --region=us-east-1 \
  --skip-prompts

./do/build && ./do/push && ./do/deploy && ./do/test

Triton: Python Backend¶

Deploy custom Python models on Triton. The Python backend gives full control over preprocessing, inference, and postprocessing logic.

ml-container-creator triton-python-demo \
  --deployment-config=triton-python \
  --instance-type=ml.g5.xlarge \
  --build-target=codebuild \
  --region=us-east-1 \
  --skip-prompts

Edit the generated model.py in the model repository to implement your inference logic, then build and deploy.

CodeBuild CI/CD¶

Any of the examples above can use CodeBuild for the image build instead of building locally. Set --build-target=codebuild during generation, then use ./do/submit instead of ./do/build + ./do/push:

./do/submit    # Creates CodeBuild project, uploads source, builds, pushes to ECR
./do/deploy    # Deploy the CodeBuild-built image
./do/test      # Validate the endpoint

./do/submit automatically creates the CodeBuild project, IAM service role, and S3 source bucket on first run. All projects share a single ECR repository (ml-container-creator) with project-specific image tags.

HyperPod EKS Deployment¶

Any of the examples above can deploy to HyperPod EKS instead of real-time inference. The target is chosen at deploy time, so generate the project as usual and pass the cluster details to ./do/deploy:

ml-container-creator hyperpod-demo \
  --deployment-config=transformers-vllm \
  --model-name=openai/gpt-oss-20b \
  --instance-type=ml.g5.12xlarge \
  --build-target=codebuild \
  --region=us-east-1 \
  --skip-prompts

./do/submit
./do/deploy --target hyperpod-eks --cluster-name my-cluster --namespace ml-serving
./do/test hyperpod

Full fine-tune on HyperPod EKS (`do/add-pod`)¶

On a hyperpod-eks deployment, a LoRA adapter hot-loads with do/adapter, and a full fine-tuned model is served next to the base as a second InferenceEndpointConfig with do/add-pod:

./do/tune --technique sft --dataset s3://my-bucket/train.jsonl --training-type full-rank
./do/add-pod sft-full --from-tune sft
./do/add-pod --list
./do/add-pod --remove sft-full

See Serving a full fine-tune next to the base.

Plain EKS (bring-your-own cluster)¶

The eks target deploys standard Kubernetes objects (Deployment + Service + ConfigMap) to a cluster you already run, with no HyperPod operator and no SageMaker endpoint. Generate as usual, then pick the target at deploy time:

ml-container-creator eks-demo \
  --deployment-config=transformers-vllm \
  --model-name=Qwen/Qwen3-4B \
  --instance-type=ml.g5.xlarge \
  --skip-prompts

cd eks-demo
./do/submit
./do/deploy --target eks --cluster-name my-eks-cluster --namespace ml-serving
./do/test eks
./do/benchmark --workload multi-turn-chat   # local AIPerf via port-forward (pip install aiperf)

./do/deploy --target eks checks the cluster contract (pod identity, CSI driver, volume backing) before applying anything. See Plain EKS.

Benchmarking: Generate → Deploy → Benchmark → Results¶

Measure LLM endpoint performance using SageMaker AI Benchmarking (NVIDIA AIPerf). Benchmarking is available for transformer and diffusor architectures only.

Generate a project with benchmarking enabled:

ml-container-creator vllm-benchmark-demo \
  --deployment-config=transformers-vllm \
  --model-name=meta-llama/Llama-3.1-8B-Instruct \
  --instance-type=ml.g5.2xlarge \
  --build-target=codebuild \
  --region=us-east-1 \
  --skip-prompts

Build, deploy, and wait for the endpoint to reach InService:

cd vllm-benchmark-demo
./do/submit    # Build and push via CodeBuild
./do/deploy

Run the benchmark against the deployed endpoint with a named workload profile (it sets the token shape, streaming and concurrency levels):

./do/benchmark --workload multi-turn-chat

Example output:

✓ Endpoint is InService
✓ Created workload config: vllm-benchmark-demo-benchmark-config
✓ Created benchmark job: vllm-benchmark-demo-benchmark-20250115-143022
⏳ Polling for completion (every 30s, up to 30 min)...
✓ Benchmark completed

┌─────────────────────────────────┬───────────┐
│ Metric                          │ Value     │
├─────────────────────────────────┼───────────┤
│ Request throughput (req/s)      │ 8.42      │
│ Output token throughput         │ 1,263     │
│ Request latency P50             │ 1.12s     │
│ Request latency P90             │ 1.58s     │
│ Request latency P99             │ 2.34s     │
│ TTFT P50                        │ 85ms      │
│ TTFT P90                        │ 142ms     │
│ TTFT P99                        │ 298ms     │
│ ITL P50                         │ 7.2ms     │
│ ITL P90                         │ 12.1ms    │
│ ITL P99                         │ 24.8ms    │
└─────────────────────────────────┴───────────┘

Clean up benchmark resources when done:

./do/benchmark --clean    # Delete workload config and benchmark jobs
./do/clean all            # Delete endpoint, ECR images, and benchmark resources

For parameter tuning and interpreting results, see Benchmarking.

Cleanup¶

Tear down resources when done to stop incurring charges:

./do/clean endpoint   # Delete SageMaker endpoint, config, and inference component
./do/clean ecr        # Delete ECR images
./do/clean codebuild  # Delete CodeBuild project and IAM role
./do/clean benchmark  # Delete workload configs and benchmark jobs
./do/clean all        # All of the above