Skip to content

Deployment & Inference

MCC supports four deployment targets and two build paths, all managed through standardized do/ scripts inspired by the do-framework. Every generated project contains scripts for all four targets — you select which target to deploy to at deploy time, not at generation time. See Interactive Deployment UX for the full deploy-time workflow.

Build Paths

Local Build

Run ./do/build to create the Docker image and ./do/push to upload it to Amazon ECR. This two-step approach lets you test locally with ./do/run before pushing.

Local containers may produce exec errors when deployed to a different architecture (e.g., building on ARM, deploying on x86). Use CodeBuild for production builds to avoid this.

./do/run starts the container on localhost:8080 for local testing. This works well for predictive ML containers (small images, no GPU dependency). LLM containers are large and typically require GPU resources, so local deployment may not be practical for those.

AWS CodeBuild

./do/submit creates an AWS CodeBuild project that builds the Docker image and pushes it to ECR in a single step. This is the preferred method for production containers, as it avoids architecture mismatches and provides fast network access to base image registries.

Deployment Targets

MCC supports four deployment targets. Select the active target at deploy time using ./do/deploy --target <mode>. The active target determines how ./do/test, ./do/clean, and ./do/logs behave.

SageMaker AI Real-Time Inference (realtime-inference)

The default deployment target. ./do/deploy provisions resources using the SageMaker AI Inference Components API:

  1. Create endpoint configuration -- specifies the instance type and count
  2. Create endpoint -- provisions the compute infrastructure
  3. Create inference component -- associates the ECR container image with the endpoint

The inference component model decouples compute provisioning from model deployment, allowing multiple models to share a single endpoint. Once the inference component reaches InService status, the endpoint is accessible via the SageMaker AI Runtime API for real-time inference requests.

The generated do/config file stores the INSTANCE_TYPE and optionally INFERENCE_AMI_VERSION for controlling the CUDA driver version on the instance.

After deployment, ./do/test validates the endpoint by invoking inference through the inference component, ./do/logs tails CloudWatch logs, and ./do/clean endpoint tears down the inference component, endpoint, and endpoint configuration.

For real-time inference, async inference, and batch transform deployment patterns, see the target-specific sections below.

SageMaker AI HyperPod EKS (hyperpod-eks)

For existing SageMaker AI HyperPod clusters running on Amazon EKS, MCC deploys through the SageMaker HyperPod inference operator using an InferenceEndpointConfig custom resource (rather than raw Kubernetes manifests):

  • ./do/deploy retrieves the underlying EKS cluster from the HyperPod cluster, configures kubectl, and applies a single InferenceEndpointConfig custom resource to the specified namespace. The amazon-sagemaker-hyperpod-inference operator reconciles it — creating the serving Deployment, Service, pods, and a SageMakerEndpointRegistration that registers a SageMaker AI endpoint named after the project. The deploy driver polls the resource's status.state until DeploymentComplete, then waits for the SageMaker endpoint to reach InService (up to 15 minutes) before recording ENDPOINT_NAME in do/config.
  • The model source is derived automatically: a model staged to S3 (STAGED_MODEL_PATH in do/config) renders modelSourceConfig.s3Storage (bucket + region parsed from the staged URI); otherwise the model is pulled from Hugging Face via MODEL_NAME, using a hf-token-secret Kubernetes Secret when a token is configured.
  • ./do/test hyperpod port-forwards the operator-created Kubernetes service (resolved from the project name) and runs the same /ping and /invocations health checks used for managed inference.
  • ./do/logs tails serving-pod logs via kubectl.
  • ./do/benchmark targets the registered SageMaker endpoint directly (no inference component) once ENDPOINT_NAME is set.
  • ./do/clean hyperpod deletes the InferenceEndpointConfig, waits for the SageMakerEndpointRegistration to be removed, and clears ENDPOINT_NAME.

The generated do/config file stores HyperPod-specific variables: HP_CLUSTER_NAME, HP_NAMESPACE, HP_REPLICAS, and — after a successful deployment — ENDPOINT_NAME.

Speculative decoding

Speculative decoding is supported for vLLM and SGLang images on HyperPod EKS. See the dedicated guide: Speculative Decoding on HyperPod EKS.

Prerequisites:

  • An existing SageMaker AI HyperPod cluster with EKS orchestrator
  • The amazon-sagemaker-hyperpod-inference EKS add-on installed (via mcc bootstrap add-module hyperpod-cluster), which provisions the hyperpod-inference service account in the target namespace
  • kubectl installed locally
  • IAM permissions for sagemaker:DescribeCluster, eks:DescribeCluster, and sagemaker:DescribeEndpoint
  • Sufficient node capacity (especially GPU nodes for LLM workloads)

Async Inference (async-inference)

For workloads with large payloads or long processing times (> 60s). ./do/deploy creates an async endpoint with an S3 output location:

  • Requests are submitted and return immediately with an output location
  • Results are written to S3 when processing completes
  • Optional SNS notifications on success/failure
  • Endpoint auto-scales to zero when idle (no cost when not in use)
ml-container-creator my-async-project \
  --deployment-target=async-inference \
  --async-s3-output-path=s3://my-bucket/async-output/ \
  ...

Batch Transform (batch-transform)

For offline batch processing of large datasets. ./do/deploy submits a SageMaker AI Transform Job:

  • Input: S3 path containing request payloads (one per file or line)
  • Output: S3 path where predictions are written
  • Compute is provisioned on-demand and released after the job completes
  • No persistent endpoint — pay only for processing time

Limitations

Batch transform does not support do/tune or do/adapter (no running endpoint to attach adapters to).

Lifecycle Scripts Reference

All generated projects include these do/ scripts:

Command Description
./do/build Build Docker image locally
./do/push Push image to Amazon ECR
./do/run Run container locally on port 8080
./do/test Test local container or deployed endpoint
./do/validate Validate configuration against AWS service models (requires schema sync)
./do/deploy Deploy to the configured deployment target. Flags: --optimize (run do/benchmark --recommend --apply before deploying), --no-optimize (skip optimization), --force-ic (force new IC even if one exists), --dry-run (validate only)
./do/tune Fine-tune using SageMaker AI Managed Model Customization (serverless)
./do/train Custom training jobs with your own scripts and hyperparameters
./do/adapter LoRA adapter lifecycle (add, list, remove, update)
./do/add-ic Add an inference component to an existing endpoint
./do/benchmark Run latency and throughput benchmarks via SageMaker AI Benchmarking
./do/status Check endpoint and inference component status
./do/logs Tail logs (CloudWatch for realtime-inference, kubectl for HyperPod)
./do/clean <target> Clean up resources (local, ecr, endpoint/hyperpod, codebuild, all)
./do/config Centralized configuration for all scripts (sourced, not executed)
./do/export Export current configuration as a reproducible CLI command
./do/register Capture deployment to the deployment registry
./do/ci CI pipeline integration (report, status, trigger, dashboard)
./do/submit Submit build to AWS CodeBuild (CodeBuild build target only)

See the generated do/README.md for detailed documentation on each command.

Pre-Deploy Validation

Run ./do/validate before deploying to catch configuration issues that would cause AWS API failures:

./do/validate                # Text output, exit 1 on errors
./do/validate --format=json  # JSON output for CI pipelines
./do/validate --smart        # Include smart-mode advisory findings

This validates your do/config values against the AWS service model, checking enum constraints, type correctness, required fields, and cross-cutting consistency (GPU counts, tensor parallelism, CUDA compatibility). See Configuration — Schema-Driven Validation for setup instructions.

The ./do/deploy --dry-run flag also runs schema validation as part of its pre-flight checks and blocks deployment if errors are found.

Pre-Deploy Optimization

Pre-deploy optimization

do/deploy --optimize runs do/benchmark --recommend --apply before deploying, writing any Athena-proven serving-config improvements to do/ic/default.conf. Non-fatal — if no benchmark history exists or Athena is unavailable, deploy proceeds with the existing config.

# Apply proven config improvements, then deploy
./do/deploy --optimize

# Explicit opt-out (e.g., for pinned configs in CI)
./do/deploy --no-optimize

Already-live endpoints

If the endpoint is already InService, do/deploy prints "Deployment is already live. Nothing to do." even after optimization runs. To apply the updated config to a running endpoint, force a new inference component: 

./do/deploy --optimize --force-ic
Or use the explicit workflow: do/benchmark --recommend --apply → do/clean endpoint → do/deploy

For hardware recommendations (which instance type to use), see do/optimize.

Benchmarking

For transformer and diffusor architectures, MCC generates a do/benchmark script by default that measures endpoint performance using the SageMaker AI Benchmarking service (NVIDIA AIPerf). Disable with --include-benchmark=false during project generation.

See the dedicated Benchmarking guide for prerequisites, parameter tuning, and interpreting results.

AWS Marketplace Model Packages

For pre-built models from AWS Marketplace vendors (AI21, Cohere, etc.), MCC generates a thin project with only lifecycle scripts — no Dockerfile, no code/ directory, no build/push steps.

How It Works

Marketplace model packages include the vendor's container image and model weights. MCC deploys them using the SageMaker AI CreateModel API with ModelPackageName instead of a custom ECR image:

ml-container-creator my-marketplace-model \
  --deployment-config=marketplace \
  --model-name='marketplace://arn:aws:sagemaker:us-east-1:aws:model-package/vendor-model/1' \
  --instance-type=ml.g5.xlarge \
  --region=us-east-1

Generated Project Structure

my-marketplace-model/
├── do/
│   ├── config          ← MODEL_PACKAGE_ARN, instance type, region
│   ├── deploy          ← CreateModel(ModelPackageName=ARN) → endpoint
│   ├── test            ← invoke endpoint
│   ├── benchmark       ← benchmark endpoint (same as BYOC)
│   ├── logs            ← CloudWatch logs
│   ├── clean           ← delete model + endpoint
│   ├── status          ← endpoint status
│   └── register        ← register deployment
├── (NO Dockerfile)
├── (NO code/)
├── (NO do/build, do/push, do/submit)

What Doesn't Apply

  • No Dockerfile (vendor provides the container)
  • No do/build, do/push, do/submit (nothing to build)
  • No LoRA adapters (can't modify vendor's model)
  • No do/tune (can't fine-tune proprietary weights)
  • No local testing (no container to run locally)

What Still Works

  • do/deploy / do/test / do/clean lifecycle
  • do/benchmark (benchmarks the endpoint regardless of who built the container)
  • do/status / do/logs / do/register
  • Async inference and batch transform (if supported by the model package)

Prerequisites

  1. Subscribe to a model on AWS Marketplace
  2. Note the Model Package ARN from your subscription
  3. Ensure your IAM role has permission to deploy the model package