Skip to content

Troubleshooting

Common issues and solutions when using ML Container Creator.

Quick Reference

Issue Fix
SyntaxError: Unexpected token 'export' nvm use node (requires Node.js 24.11.1+)
Generator not found npm link in the project directory
Docker build fails: package not found Pin versions in requirements.txt
Container exits immediately docker logs <id> to check for errors
/ping returns connection refused Verify container is running and port 8080 is exposed
ECR authentication failed aws ecr get-login-password (see ECR Auth)
Endpoint stuck in Creating Check CloudWatch logs for health check or model loading failures
Model file not found in container Verify COPY directive in Dockerfile targets /opt/ml/model/
Adapter "Not Found" on first call Wait 60s after do/adapter add (see LoRA Adapter Issues)
Adapter read timeout Re-run ./do/test --adapter <name> after ~1 minute (see Read timeout)
Adapter wrong model name Use base model name, not adapter name (see Wrong Model Name)
HuggingFace API timeout Use --offline flag
HuggingFace access denied Verify token and model license agreement

Generator Issues

Generator Not Found

Error: @aws/ml-container-creator generator not found
cd ml-container-creator
npm link
ml-container-creator --help   # Should show available commands

Node.js Version Error

SyntaxError: Unexpected token 'export'
SyntaxError: Cannot use import statement outside a module

MCC requires Node.js 24.11.1+ for ES module support:

nvm install node
nvm use node
node --version    # Must be 24.11.1+

CLI Not Found in CI

npm install -g @aws/ml-container-creator
ml-container-creator --help

Docker Build Issues

Package Not Found

ERROR: Could not find a version that satisfies the requirement scikit-learn

Pin versions in requirements.txt:

scikit-learn==1.3.0
numpy==1.24.0

Or rebuild without cache: docker build --no-cache -t my-model .

Permission Denied

ERROR: failed to copy files: permission denied
chmod 644 code/*
chmod 755 code/*.sh
docker build -t my-model .

Local Testing Issues

Container Exits Immediately

docker logs <container-id>           # Check for startup errors
docker run -it my-model /bin/bash    # Debug interactively

Common causes: missing model file at /opt/ml/model/, missing Python dependencies, syntax errors in serve.py.

Health Check Fails

curl: (7) Failed to connect to localhost port 8080
docker ps                            # Is the container running?
docker logs <container-id>           # Check for errors

If the container is running but not responding, the server may have failed to bind to port 8080. Check that the Dockerfile exposes port 8080 and the server is configured to listen on 0.0.0.0:8080.

Inference Returns Error

docker logs <container-id>           # Check for the full traceback

Common causes:

  • Wrong input format -- ensure Content-Type: application/json header is set
  • Feature count mismatch -- input must have the same number of features as training data
  • Wrong data types -- ensure numeric values are floats, not strings

AWS Deployment Issues

ECR Authentication Failed

Error: no basic auth credentials
aws ecr get-login-password --region us-east-1 | \
  docker login --username AWS --password-stdin \
  <account-id>.dkr.ecr.us-east-1.amazonaws.com

Verify credentials: aws sts get-caller-identity

IAM Permission Denied

Error: User is not authorized to perform: ecr:CreateRepository

Required permissions for deployment: ecr:CreateRepository, ecr:PutImage, ecr:InitiateLayerUpload, ecr:UploadLayerPart, ecr:CompleteLayerUpload, sagemaker:CreateModel, sagemaker:CreateEndpointConfig, sagemaker:CreateEndpoint, iam:PassRole. See the generated IAM_PERMISSIONS.md for the full policy document.

Endpoint Creation Failed

aws logs tail /aws/sagemaker/Endpoints/<endpoint-name> --follow

Common causes:

  • Invalid IAM role -- verify with aws iam get-role --role-name <role>
  • Image not found in ECR -- verify with aws ecr describe-images --repository-name <repo>
  • Insufficient instance capacity -- try a different instance type or region

Endpoint Stuck in Creating

If the endpoint stays in Creating status for more than 15 minutes, check CloudWatch logs. Common causes:

  • Container fails to start -- check Dockerfile CMD/ENTRYPOINT and verify port 8080 is exposed
  • Health check fails -- /ping must return 200 within 2 seconds
  • Model loading fails -- verify model file exists in the container and format matches the handler code

To recover: delete the endpoint and redeploy.

./do/clean endpoint
./do/deploy

Endpoint Returns 500

aws logs tail /aws/sagemaker/Endpoints/<endpoint-name> --follow

Common causes: exception in prediction code, wrong input format, model not loaded. Test locally first:

./do/run
curl -X POST http://localhost:8080/invocations \
  -H "Content-Type: application/json" \
  -d '{"instances": [[1.0, 2.0, 3.0]]}'

Model Loading Issues

Server rejects an argument / won't start (junk --build-* args)

vLLM engine args: [... --build-url https://buildkite.com/... --image-tag vllm/vllm-openai:v0.23.0
                   --build-pipeline ... --build-commit ...]
api_server.py: error: unrecognized arguments: --build-url ...

Cause: vLLM v0.21+ bakes build-provenance env vars (VLLM_BUILD_URL, VLLM_IMAGE_TAG, VLLM_BUILD_PIPELINE, VLLM_BUILD_COMMIT) into the image. A pre-fix serve script forwarded every VLLM_* env var as a CLI flag, so these leaked in as --build-url etc. and the engine rejected them — the IC never reaches InService.

Fix: current serve scripts use a --help introspection whitelist — they forward only args the running engine actually accepts. If you see junk --build-* args, your generated project predates this fix:

  1. Regenerate the project (picks up the fixed code/serve.d/vllm/vllm.ejs), or hand-patch the serve script (it's EJS-free — safe to edit directly).
  2. ./do/build && ./do/push — the serve script is baked into the image, so a rebuild is required.
  3. Redeploy. The vLLM engine args: log line should now be clean.

This applies to SGLang too (SGLANG_* → sglang serve). The model-server-version MCP is not at fault — it selects the image correctly; the issue is purely serve-script arg harvesting. Full mechanism + a troubleshooting matrix: Serving Engine Arguments & Version Troubleshooting.

Model File Not Found

FileNotFoundError: No such file or directory: '/opt/ml/model/model.pkl'
# Verify model is in the container
docker run my-model ls -la /opt/ml/model/

# Check the COPY directive in Dockerfile
grep "COPY.*model" Dockerfile

For predictive models, the model file must be copied into the container at build time. For transformer models, the serving framework downloads the model at runtime from HuggingFace Hub.

Model Format Mismatch

ValueError: Model format not recognized

Verify the model file format matches what you selected during generation:

Framework Expected formats
sklearn .pkl, .joblib
xgboost .json, .model, .ubj
tensorflow SavedModel/ directory, .keras, .h5

The authoritative per-framework format list lives in each framework's descriptor, templates/code/predictors.d/<framework>/manifest.json (model_formats). The table above mirrors it for convenience.

Pickle Version Mismatch

ValueError: unsupported pickle protocol: 5

The Python version used to save the model must match the version in the container. Either re-save the model with a compatible protocol (pickle.dump(model, f, protocol=4)) or update the Python version in the Dockerfile.

HuggingFace Issues

API Timeout

Warning: HuggingFace API timeout, checking local registry

This is expected behavior -- the generator falls back to local registry data. To skip HuggingFace API calls entirely:

ml-container-creator --offline

Access Denied or Repository Not Found

Warning: Model 'my-org/my-model' not found on HuggingFace

For private or gated models:

  1. Verify the model ID at https://huggingface.co/<model-id>
  2. Accept the model's license agreement on HuggingFace (for gated models like Llama)
  3. Store the token in Secrets Manager and reference it: ml-container-creator secrets create --type hf-token --name <label> --secret-value hf_***, then mcc bootstrap add-secret hfToken <secret-arn> (see the plaintext-token entry below)

See HuggingFace Authentication for details.

Rate Limit Exceeded

Warning: HuggingFace API rate limit exceeded, using cached data

Use --offline to skip API calls, or set HF_TOKEN for higher rate limits:

export HF_TOKEN=hf_your_token_here
ml-container-creator

do/deploy refuses: "HF_TOKEN is set as a plaintext value"

❌ HF_TOKEN is set as a plaintext value and no HF_TOKEN_ARN is configured.

do/deploy won't write a plaintext token into the SageMaker container Environment. SageMaker Describe* APIs return that map and CloudTrail records it. Store the token in Secrets Manager and deploy by reference:

ml-container-creator secrets create --type hf-token --name production --secret-value hf_***
mcc bootstrap add-secret hfToken <secret-arn>    # or: export HF_TOKEN_ARN=<secret-arn> in do/config
./do/deploy

If the model isn't gated, run unset HF_TOKEN (it may come from your shell profile). The same applies to NGC_API_KEY, IC_ENV_HF_TOKEN and IC_ENV_NGC_API_KEY. ./do/deploy --allow-plaintext-secret keeps the old behavior with a warning. See Secrets: deploy-time.

Container exits: "failed to resolve HF_TOKEN_ARN"

Error: failed to resolve HF_TOKEN_ARN (arn:aws:secretsmanager:...) from Secrets Manager.

The serving container resolves the secret at startup as the SageMaker execution role. Check that:

  1. The role allows secretsmanager:GetSecretValue on the secret. The bootstrap role covers mlcc/*, ml-container-creator/* and mlcc-hf-token-*. For any other name, or for your own ROLE_ARN, add a grant. do/deploy warns when the name is outside those prefixes.
  2. The role allows kms:Decrypt if the secret uses a customer-managed KMS key. Re-run mcc bootstrap to pick up the SecretsManagerKmsDecrypt statement.
  3. The endpoint can reach Secrets Manager: from a VPC with no internet path, add a com.amazonaws.<region>.secretsmanager interface endpoint.

LMI/DJL, llama.cpp and Triton images don't run code/serve, so they never resolve *_ARN. For gated models, stage the weights with ./do/stage.

Endpoints deployed before the secrets-by-reference fix still expose the token

Endpoints created by earlier releases have HF_TOKEN / NGC_API_KEY in plaintext in their container Environment. List the keys without printing values:

aws sagemaker describe-inference-component --inference-component-name <ic> \
    --query 'keys(Specification.Container.Environment)'
aws sagemaker describe-model --model-name <model> --query 'keys(PrimaryContainer.Environment)'

If a secret key is listed, rotate the token at the provider, update the secret, redeploy, and delete the superseded models and inference components.

LoRA Adapter Issues

"Not Found" error on first adapter invocation

Symptoms: After do/adapter add, the first do/test returns {"detail":"Not Found"} but a second attempt succeeds.

Root cause: The adapter inference component reports InService before vLLM finishes loading LoRA weights into GPU memory. SageMaker AI's readiness check passes (base model /ping returns 200) but the adapter isn't actually ready to serve requests yet.

Workaround: Wait 30–60 seconds after do/adapter add reports success before testing. A future release will add a post-attach probe loop to confirm the adapter is serving.

# Wait, then test
./do/adapter add my-sft --from-tune
sleep 60
./do/test

Read timeout on first adapter inference

Symptoms: First inference after adapter load returns a "Read timeout" error, but the response body contains valid JSON.

Root cause: The first inference triggers JIT compilation of the adapter code path. Combined with thinking-mode tokens (for reasoning models like DeepSeek R1), this can exceed the default CLI read timeout.

Fix: do/test has no read-timeout flag. Wait about a minute and re-run the test; subsequent calls are fast, only the first invocation is slow:

./do/test --adapter <name>

Wrong model name in adapter test

Symptoms: do/test sends the adapter name (e.g., "val-sft") as the model field in the request, but vLLM returns "Not Found" because it only recognizes the base model name.

Root cause: When adapter config is detected, do/test uses ADAPTER_MODEL_NAME in the request's "model" field. However, SageMaker AI handles adapter routing at the inference component layer — vLLM doesn't need to know the adapter name. The correct value is the base MODEL_NAME (e.g., "Qwen/Qwen3-0.6B").

Workaround: Invoke directly with the base model name:

aws sagemaker-runtime invoke-endpoint \
  --endpoint-name <endpoint> \
  --inference-component-name <adapter-ic-name> \
  --body '{"model": "Qwen/Qwen3-0.6B", "messages": [{"role": "user", "content": "Hello"}]}' \
  --content-type application/json \
  output.json

This will be fixed in a future release so do/test automatically uses the base model name when testing adapters.

Adapter IC stuck in "Creating" state

Symptoms: do/adapter add hangs waiting for the inference component to reach InService.

Common causes:

  1. Insufficient GPU memory — The base model + adapter don't fit in the instance's GPU memory. Try a smaller adapter rank or a larger instance.
  2. Invalid S3 path — The adapter weights URI doesn't exist or isn't accessible by the SageMaker AI execution role.
  3. Incompatible adapter — The adapter was trained with a different base model or rank than configured.

Debug:

# Check IC status and failure reason
aws sagemaker describe-inference-component \
  --inference-component-name <adapter-ic-name> \
  --query '[InferenceComponentStatus, FailureReason]'

# Check endpoint logs for vLLM errors
./do/logs

Kubernetes: "No adapter found" when loading a LoRA adapter (hyperpod-eks / eks)

Symptoms: load_lora_adapter fails with No adapter found for <path>, or vLLM tries to download the adapter from Hugging Face.

Root cause: vLLM loads adapters from a LOCAL path. An s3:// URI (or a path that is not on the pod's model mount) is treated as a Hub repo id. The pod has no AWS credentials.

Fix: use ./do/adapter add <name> --from-tune <t> (or --from-weights): it promotes the adapter to its canonical version models/<org>/<base>/.mlcc-adapters/<project>/<name>/v<hash12>/ in the core bucket and loads /opt/ml/model/.mlcc-adapters/<project>/<name>/v<hash12>. If it refuses:

  • model is not served from S3 — ./do/stage and ./do/deploy --target hyperpod-eks --force;
  • prefetchEnabled: true — redeploy so MLCC turns prefetch off for LoRA + S3: ./do/deploy --target hyperpod-eks --force (add --surge-node on a single-node group);
  • no S3 model mount (eks) — set EKS_MODEL_MOUNT=s3 with a staged model and redeploy;
  • incompatible base — the adapter was trained on a different base than the deployment serves.

See LoRA delivery on Kubernetes.

"rank X > max_lora_rank Y" when adding an adapter

Symptoms: ./do/adapter add refuses with the adapter's rank and the serving limit, or vLLM rejects the load with a rank error.

Root cause: vLLM fixes max_lora_rank at start. Managed-tune adapters are often r32/r64; the engine default is 16.

Fix: redeploy so the limit is derived from the recorded adapters (./do/deploy --target <target> --force), or, if VLLM_MAX_LORA_RANK is set explicitly in do/config, raise it to at least the adapter's rank (an allowed value: 1, 8, 16, 32, 64, 128, 256, 320, 512) first. On realtime-inference the fix is the base IC's IC_ENV_VLLM_MAX_LORA_RANK and ./do/deploy --target realtime-inference --force-ic. A rank whose LoRA buffer exceeds 10% of the free GPU memory is refused at deploy with the numbers.

hyperpod-eks: rollout stuck Pending / deploy stops with "rollout cannot schedule"

Symptoms: after a pod-template change the new pod stays Pending (insufficient nvidia.com/gpu), or ./do/deploy --target hyperpod-eks --force stops before applying.

Root cause: the instance group has one node and the serving pod holds all its GPUs, so a surge pod cannot schedule. MLCC never scales to zero to make room.

Fix: ./do/deploy --target hyperpod-eks --force --surge-node adds one node for the rollout and removes the vacated one afterwards (the count is restored on any failure). Capacity errors (InsufficientCapacity) are printed verbatim — retry later. If nothing changed in the pod template, no rollout is needed and none happens.

hyperpod-eks: deploy stops with "no node has a free pod slot"

Symptoms: a first ./do/deploy --target hyperpod-eks prints The serving pod cannot start on <instance-type>: no node has a free pod slot and Nothing was applied. (./do/add-pod prints No node has room for <name> (GPUs and pod slots, …); it also accepts --surge-node.)

Root cause: the nodes have free GPUs, but no free pod slots (the node's allocatable pods minus its running pods). The serving pod needs one slot, plus one for its Mountpoint-S3 pod when the model is mounted from S3. Without that slot the pod would schedule and then hang in ContainerCreating (FailedMount, "Too many pods").

Fix: free pod slots on a node of that instance type (remove idle pods), add a node to the instance group, or deploy on another instance type. The message prints the per-node counts and a suggested fix. A GPU shortage alone only warns: the pod is applied and waits Pending for a GPU.

do/stage Processing Job fails for large models

Symptom: Processing Job exits with code 1. CloudWatch logs may show Killed or out-of-memory errors during model download. No useful error message from do/stage itself.

Cause: The default Processing Job instance (ml.m5.xlarge, 16 GB RAM) doesn't have enough memory to download and checkpoint large models (>15B parameters).

Fix: Use a larger instance:

./do/stage --instance-type ml.m5.4xlarge   # 64 GB RAM — for 15B-70B models
./do/stage --instance-type ml.m5.12xlarge  # 192 GB RAM — for 70B+ models
Model Size Required Instance
≤15B ml.m5.xlarge (default)
15B–70B ml.m5.4xlarge
70B+ ml.m5.12xlarge

Also check: Your account's EBS volume quota for Processing Jobs. New accounts default to 100-500 GB, but do/stage requests 2048 GB. Request an increase to 4096 GB via:

Service Quotas → SageMaker → "Processing job maximum EBS volume size in GB"

The agent (ml-container-creator hey) checks this automatically in its health report.

vLLM container logs go dark after "engine args" on multi-GPU

Symptoms: Container starts, logs show CUDA compat: driver X < Y, adding compat libs and vLLM engine args: [...], then no further output. IC reports "InService" but inference returns InternalFailure. Benchmark jobs fail with 400 Bad Request.

Root cause: The vLLM image was compiled against a newer CUDA toolkit than the instance's GPU driver supports. The CUDA forward compatibility layer loads partially but fails silently during NCCL initialization for multi-GPU tensor-parallel deployments. No Python exception is raised — the process hangs or is killed by the container runtime.

Example: vllm/vllm-openai:v0.23.0 (CUDA 12.9, requires driver ≥580) on ml.g5.24xlarge (driver ~550).

Fix:

  1. Downgrade to a compatible vLLM version:

    # Check driver compatibility table in MCP Servers docs
    # For g5 (driver ~550): use vLLM ≤v0.21.x
    sed -i '' 's/vllm-openai:v0.23.0/vllm-openai:v0.20.2/' Dockerfile
    ./do/build && ./do/push
    

  2. Or use base-image-picker with instanceType context — it automatically excludes incompatible versions:

    ml-container-creator mcp --bundled add base-image-picker
    # During generation, base-image-picker filters by driver compatibility
    

Key diagnostic: The CUDA compat log line confirms the mismatch. If you see this followed by silence (no "Loading model..." or error), it's always the driver compatibility issue.

Multi-GPU NCCL fails on plain EKS: "unhandled system error" at ncclCommInitRank

Symptoms: On a plain eks deploy of a tensor-parallel model (--tp-size > 1), every TP rank crashes at startup with RuntimeError: NCCL error: unhandled system error (run with NCCL_DEBUG=INFO for details) in ncclCommInitRank. Weights may load first; the failure is at communicator init.

Root cause: Kubernetes defaults a pod's /dev/shm to 64Mi. NCCL uses /dev/shm for intra-node, cross-GPU transport; 64Mi is far too small, so the communicator can't form. (hyperpod-eks doesn't hit this — the operator sizes shared memory.)

Fix: The eks Deployment template mounts a RAM-backed /dev/shm sized HP_GPU_COUNT × 8Gi. If you still see the error, confirm the mount took and raise it:

kubectl exec <pod> -- df -h /dev/shm      # should show GiB, not 64M
# raise it in do/config, then redeploy:
echo 'export HP_SHM_SIZE=64Gi' >> do/config

Diagnostic: Set export NCCL_DEBUG=INFO in do/config and read the NCCL INFO/WARN lines just before the failure — they name the subsystem (shm, P2P, NVLS). A shm/shmOpen error is this issue; a P2P/IPC or NVLS error is a different, pod-capability/driver problem.

Server start fails with "Argument list too long" on every command (E2BIG)

Symptoms: After an S3 model download completes on plain eks, the serve log shows env, grep, python, and the server binary all failing with Argument list too long, and the engine never starts.

Root cause: aws s3 cp/sync write a progress meter to stdout. The serve script captures resolve_model's stdout as the model path and exports it into the engine's model env var; the leaked progress blob can exceed the kernel's per-string limit (MAX_ARG_STRLEN, 128 KiB), after which every execve fails with E2BIG. It is model-size probabilistic — surfaces on large, many-shard models.

Fix: Fixed in the generated code/serve (aws calls use --no-progress --only-show-errors and the downloader's stdout is isolated from the captured path). If you see this on an older generated project, mcc regenerate to pick up the fix.

SGLang won't start a quantized MoE model with LoRA enabled

Symptoms: Weights load, then NotImplementedError: Mxfp4MoEMethod does not expose quant info for 'flashinfer_mxfp4' from init_lora_manager.

Root cause: LoRA was enabled (HP_LORA_ENABLED=true) on an MXFP4-quantized MoE model. SGLang (as of 0.5.x) does not implement LoRA over MXFP4-quantized MoE expert modules.

Fix: If you are serving the base model, disable LoRA — set export HP_LORA_ENABLED=false in do/config and redeploy. (You can hot-patch a live pod by removing SGLANG_ENABLE_LORA/SGLANG_MAX_LORA_RANK/SGLANG_LORA_TARGET_MODULES from its ConfigMap and restarting.) If you genuinely need LoRA on such a model, it is blocked upstream in SGLang — no MLCC setting changes it.

Custom-tokenizer model refuses to start without trust-remote-code

Symptoms: A model whose tokenizer ships custom Python (e.g. certain MoE/multimodal models) fails at tokenizer load unless remote code is trusted.

Fix: Set the engine's trust-remote-code flag in do/config — export SGLANG_TRUST_REMOTE_CODE=true (or VLLM_TRUST_REMOTE_CODE=true). The same flag also feeds do/benchmark's tokenizer loading on the HyperPod/SageMaker path.

Adapter IC "InService" but inference returns "Failed to download model data"

Symptoms: do/test --adapter <name> returns ValidationError: Failed to download model data (bucket: ..., key: .../adapter-name). The IC shows InService in describe-inference-component.

Root cause: The adapter's ArtifactUrl is missing the trailing slash. SageMaker Inference Components expect S3 directory prefixes to end with / — without it, SageMaker looks for a single object at that exact key (which doesn't exist).

Fix:

  1. Delete the IC and re-create with the corrected URL:

    aws sagemaker delete-inference-component \
      --inference-component-name <ic-name> --region <region>
    sleep 45
    
    # Fix the conf file
    sed -i '' 's|adapters/my-adapter"|adapters/my-adapter/"|' do/adapters/<name>.conf
    
    # Re-add
    ./do/adapter add <name> --from-weights "s3://bucket/prefix/"
    

  2. For adapters from --from-registry: the template fix (2026-06-29) automatically re-adds the trailing slash for non-tar.gz adapter URIs. Regenerate the project or update the do/adapter script.

Additive deploy: "Contract violation: artifact-kind" or "has no load mechanism"

Symptoms: do/adapter add or do/add-ic exits with code 3 and one of:

  • Contract violation: artifact-kind: "This source is a 'full_model', but this verb accepts 'lora_adapter'" (or the reverse).
  • Contract violation: target: "do/add-ic (accepts: full_model) has no load mechanism on hyperpod-eks … not servable on hyperpod-eks yet".

Root cause: Additive deploy follows one contract (ADR-013). The source's artifact kind picks the verb, and the target picks the load mechanism. do/adapter hot-loads LoRA adapters only. do/add-ic deploys full models only, and only on realtime-inference. On hyperpod-eks a full model goes through do/add-pod. Plain eks has no full-model verb yet (planned post-2.0).

Fix:

  • A LoRA adapter: ./do/adapter add <name> --from-tune (or --from-weights s3://..., --from-registry <arn>) on realtime-inference, hyperpod-eks, or eks.
  • A full model: ./do/add-ic <name> --from-tune (or --from-weights, --from-registry) on realtime-inference, or ./do/add-pod <name> --from-tune (same --from-* flags) on hyperpod-eks.
  • An "artifact_kind unknown … falling back to a content check" notice is informational. Raw S3 URIs, Hub repos, and older registry versions carry no recorded kind, so the verb proceeds after an advisory adapter_config.json check.
  • "--weights is deprecated" / "--model-data is deprecated" notices: switch to --from-weights. The old flags still work.

Getting Help

Collect diagnostics:

# Container logs
docker logs <container-id>

# SageMaker AI endpoint logs
aws logs tail /aws/sagemaker/Endpoints/<endpoint-name> --follow

# Generator debug output
DEBUG=* ml-container-creator