Getting Started

This guide covers installation and two end-to-end walkthroughs: deploying a predictive model (sklearn + Flask) and deploying an LLM (vLLM). Both deploy to a SageMaker AI managed inference endpoint.

Prerequisites

Tool	Version	Purpose
Node.js	24+	Runs the CLI
Python	3.10+	do/ lifecycle scripts and model serving
Docker	20+	Local container builds
AWS CLI	2+	AWS resource management

You also need an AWS IAM role with permissions for ECR, SageMaker AI, and (optionally) CodeBuild.

Service quota for large model staging

If you plan to stage models >15B parameters via do/stage (SageMaker Processing Job), your account's default EBS volume quota for Processing Jobs may be too low (100-500 GB). The staging job requests 2048 GB of attached storage for model downloads.

Recommended: Request a quota increase to 4096 GB via: AWS Console → Service Quotas → SageMaker → "Processing job maximum EBS volume size in GB"

This is only needed for large model staging — deployments and training use separate quotas.

Python dependencies (boto3, sagemaker>=3.0, huggingface_hub, pyarrow, hf_transfer, packaging, PyYAML) are installed automatically when you run npm install. If you use a virtual environment or pyenv, activate it before running npm install so dependencies land in the correct location. The full list is in requirements.txt.

Verify your setup:

node --version
python --version
docker --version
aws --version
aws sts get-caller-identity

Installation

From npm (recommended)

npm install -g @aws/ml-container-creator

Zero-install with npx

npx @aws/ml-container-creator

From source

git clone https://github.com/awslabs/ml-container-creator.git
cd ml-container-creator
npm install && npm link

Verify the CLI is available:

ml-container-creator --version

Example 1: Predictive Model (sklearn + Flask)

This walkthrough generates a project that serves a scikit-learn model behind Flask on a SageMaker AI real-time endpoint. It uses the built-in Abalone sample model so you can follow along without providing your own model file.

Generate the project

mkdir sklearn-demo && cd sklearn-demo
ml-container-creator sklearn-demo \
  --deployment-config=http-flask \
  --model-format=pkl \
  --deployment-target=realtime-inference \
  --instance-type=ml.m5.large \
  --region=us-east-1 \
  --skip-prompts

The generator creates the project directory and trains the sample model automatically.

Project structure

sklearn-demo/
├── Dockerfile
├── requirements.txt
├── nginx-predictors.conf
├── code/
│   ├── model_handler.py          # Model loading and inference
│   ├── serve.py                  # Flask server
│   ├── start_server.py
│   └── flask/
│       ├── gunicorn_config.py
│       └── wsgi.py
├── sample_model/
│   ├── train_abalone.py          # Training script
│   ├── test_inference.py         # Local inference test
│   └── abalone_model.pkl         # Trained model artifact
├── do/                           # Lifecycle scripts
│   ├── config                    # Project configuration
│   ├── build                     # Build Docker image
│   ├── push                      # Push to ECR
│   ├── submit                    # Submit build to CodeBuild
│   ├── deploy                    # Deploy to SageMaker AI
│   ├── validate                  # Validate config against AWS service models
│   ├── test                      # Test local or deployed endpoint
│   ├── clean                     # Tear down resources
│   ├── register                  # Log to deployment registry
│   ├── manifest                  # Asset manifest operations
│   ├── run                       # Run container locally
│   ├── logs                      # Tail CloudWatch logs
│   └── export                    # Export config as JSON
└── test/
    ├── test_local_image.sh
    ├── test_model_handler.py
    └── test_endpoint.sh

Build, push, and deploy

If you haven't already, run ml-container-creator bootstrap to set up shared AWS infrastructure. The bootstrap provisions modular CDK stacks (core, registry, benchmark, training, ci) that you can select interactively or via --with:

# Interactive: select modules from a menu
ml-container-creator bootstrap add my-profile

# Non-interactive: provision core + registry (default)
ml-container-creator bootstrap --non-interactive --profile my-profile --region us-west-2

# Non-interactive with extra modules
ml-container-creator bootstrap --non-interactive --profile my-profile --region us-west-2 --with benchmark,training

# Preview what would be provisioned without creating resources
ml-container-creator bootstrap add my-profile --dry-run

Add or remove modules after initial setup:

ml-container-creator bootstrap add-module training
ml-container-creator bootstrap add-module training --dry-run
ml-container-creator bootstrap remove-module benchmark --dry-run

Optionally sync AWS service models for pre-deploy validation:

ml-container-creator bootstrap sync-schemas

Tip

bootstrap automatically chains post-setup steps: AWS resources → mcp init → sync-architectures → sync-schemas. You don't need to run sync-schemas separately after a fresh bootstrap. Use --skip-post-setup to bypass the chain if you only need the core AWS resources.

./do/build        # Build the Docker image
./do/push         # Push to Amazon ECR
./do/validate     # Validate config against AWS service models (optional)
./do/deploy       # Deploy to SageMaker AI (requires IAM role ARN)

./do/deploy creates a SageMaker AI endpoint configuration, endpoint, and inference component. It waits for the endpoint to reach InService status.

Test

# Test the deployed endpoint
./do/test

Output:

🧪 Testing SageMaker AI endpoint: sklearn-demo-endpoint-<TIMESTAMP>

🔍 Test 1: Health check
   Checking endpoint status...
✅ Endpoint is InService

🔍 Test 2: Inference request
   Payload: Sample feature vector
   Invoking SageMaker AI endpoint...
✅ Inference request successful
   Response preview: {"predictions": [12.86]}

✅ All tests passed!

You can also test locally before deploying:

./do/run          # Start container on localhost:8080
./do/test         # Test against local container (no argument = local mode)

Bring your own model

To use your own model instead of the sample, edit the Dockerfile COPY directive:

# Replace the sample model line:
# COPY sample_model/abalone_model.pkl /opt/ml/model/
COPY path/to/your/model.pkl /opt/ml/model/

Example 2: LLM (vLLM)

This walkthrough deploys Qwen/Qwen3-4B to a SageMaker AI real-time endpoint using vLLM. The model runs on a single ml.g5.xlarge GPU instance and supports LoRA adapter hot-swapping out of the box.

Time estimate

~30 minutes end-to-end (most of that is waiting for the container build and endpoint deployment).

Generate the project

ml-container-creator qwen3-demo \
  --deployment-config=transformers-vllm \
  --model-name=Qwen/Qwen3-4B \
  --deployment-target=realtime-inference \
  --instance-type=ml.g5.xlarge \
  --region=us-east-1 \
  --skip-prompts

Project structure

qwen3-demo/
├── Dockerfile
├── IAM_PERMISSIONS.md
├── code/
│   ├── serve                     # Entrypoint script launching vLLM
│   └── serving.properties        # Server configuration (model ID, LoRA settings)
├── do/
│   ├── config                    # Project configuration
│   ├── build                     # Build Docker image locally
│   ├── push                      # Push to Amazon ECR
│   ├── submit                    # Submit build to CodeBuild (alternative)
│   ├── deploy                    # Deploy to SageMaker AI
│   ├── test                      # Test inference
│   ├── tune                      # Fine-tune (SageMaker AI managed customization)
│   ├── adapter                   # LoRA adapter management (add/remove/list)
│   ├── benchmark                 # Latency + throughput measurement
│   ├── clean                     # Tear down resources
│   ├── register                  # Log to deployment registry
│   ├── manifest                  # Asset manifest operations
│   ├── logs                      # Tail CloudWatch logs
│   └── export                    # Export config as JSON
└── test/
    └── test_endpoint.sh

Build and push

Build the container image locally and push to Amazon ECR:

cd qwen3-demo
./do/build        # Build the Docker image (~10 minutes for vLLM base)
./do/push         # Push to Amazon ECR

Tip

If your local machine doesn't have enough disk space or you prefer a cloud build, use ./do/submit instead. This submits the build to AWS CodeBuild (requires bootstrap with CodeBuild enabled).

Deploy

./do/deploy       # Deploy to SageMaker AI (GPU endpoint, 5-10 minutes)

The deploy script creates an endpoint configuration, endpoint, and inference component. It waits until the endpoint reaches InService status — vLLM needs time to download the model weights from HuggingFace and load them onto the GPU.

Test

./do/test

Output:

🧪 Testing SageMaker AI endpoint: qwen3-demo-endpoint-<TIMESTAMP>

🔍 Test 1: Health check
   Checking endpoint status...
✅ Endpoint is InService

🔍 Test 2: Inference request
   Payload: OpenAI-compatible chat completion request
   Invoking SageMaker AI endpoint...
✅ Inference request successful
   Response preview: {"choices": [{"message": {"content": "Hello! I'm Qwen, a large language model created by Alibaba Cloud..."}}]}

✅ All tests passed!

The endpoint serves an OpenAI-compatible API — you can use it with any OpenAI SDK client by pointing to your SageMaker AI endpoint URL.

Next: Fine-Tune and Add an Adapter

Your endpoint supports LoRA adapter hot-swapping (enabled by default). You can fine-tune Qwen3-4B and attach the resulting adapter without redeploying:

# Fine-tune with SageMaker AI managed customization
./do/tune --technique sft --dataset s3://my-bucket/train.jsonl

# Attach the trained adapter to the running endpoint
./do/adapter add my-sft --from-tune

# Test the adapter
./do/test

The adapter is served as a separate inference component on the same endpoint — no downtime, no container rebuild. See Fine-Tuning for the full guide on training data format, all supported techniques (SFT, DPO, RLAIF, RLVR), and adapter lifecycle management. For full control over the training code (custom trainers, custom techniques), see Custom Training.

Cleanup

Tear down deployed resources to stop incurring charges:

./do/clean endpoint   # Delete SageMaker AI endpoint, config, and inference component
./do/clean ecr        # Delete ECR images
./do/clean codebuild  # Delete CodeBuild project and IAM role (if applicable)
./do/clean all        # All of the above

CI Integration (Optional)

The bootstrap command can provision a CI module that automatically tests your deployment configurations end-to-end on a recurring schedule. This is useful for validating that the generator continues to produce working containers across all supported configurations.

To include CI during bootstrap:

# Interactive: select the "ci" module when prompted
ml-container-creator bootstrap add my-profile

# Non-interactive
ml-container-creator bootstrap --non-interactive --profile my-profile --region us-west-2 --with benchmark,ci

Or add CI to an existing bootstrap:

ml-container-creator bootstrap add-module ci
ml-container-creator bootstrap add-module ci --dry-run  # preview first

Once provisioned, register any generated project for automated testing:

./do/register --ci

The harness will regenerate, build, deploy, test, and tear down the project hourly, reporting results via ./do/ci report.

For full details, see the CI Integration Guide.

Next Steps

• Supported Models — Check if your model is validated end-to-end
• Configuration — CLI flags, environment variables, config files, and MCP servers
• Deployment & Inference — All deployment targets and lifecycle scripts
• Fine-Tuning — Fine-tune supported models with do/tune and deploy adapters
• Custom Training — Bring your own training code with do/train
• Examples — Walkthroughs for other architectures (Triton, diffusors, async, batch transform)
• MCP Servers — Configure intelligent defaults for instance sizing, region selection, and more
• Benchmarking — Measure latency and throughput with SageMaker AI Benchmarking
• CI Integration — Automated lifecycle testing for all deployment configurations
• Troubleshooting — Common issues and solutions
• Advisory Agent — Get help from a conversational AI advisor: ml-container-creator hey

---

Quick Start: DLC-Direct Mode (v1.2+)

For the fastest path to deployment when you don't need container customization:

# 1. Bootstrap (one-time setup)
ml-container-creator bootstrap add my-profile

# 2. Generate a DLC-direct project (no Docker build needed)
ml-container-creator my-model --no-build \
  --deployment-config transformers-vllm \
  --model-name Qwen/Qwen3-0.6B \
  --instance-type ml.g5.xlarge

# 3. Stage model weights to S3
cd my-model
./do/stage

# 4. Deploy directly (uses stock AWS DLC image)
./do/deploy

This skips the Dockerfile/build/push steps entirely — saving 10-20 minutes on first deployment. The resolved DLC image is CUDA-driver-compatible with your chosen instance type.

See Configuration: DLC-Direct Mode for details and limitations.