Troubleshooting¶
Common issues and solutions when using ML Container Creator.
Quick Reference¶
| Issue | Fix |
|---|---|
SyntaxError: Unexpected token 'export' |
nvm use node (requires Node.js 24.11.1+) |
| Generator not found | npm link in the project directory |
| Docker build fails: package not found | Pin versions in requirements.txt |
| Container exits immediately | docker logs <id> to check for errors |
/ping returns connection refused |
Verify container is running and port 8080 is exposed |
| ECR authentication failed | aws ecr get-login-password (see ECR Auth) |
| Endpoint stuck in Creating | Check CloudWatch logs for health check or model loading failures |
| Model file not found in container | Verify COPY directive in Dockerfile targets /opt/ml/model/ |
| Adapter "Not Found" on first call | Wait 60s after do/adapter add (see LoRA Adapter Issues) |
| Adapter read timeout | Re-run ./do/test --adapter <name> after ~1 minute (see Read timeout) |
| Adapter wrong model name | Use base model name, not adapter name (see Wrong Model Name) |
| HuggingFace API timeout | Use --offline flag |
| HuggingFace access denied | Verify token and model license agreement |
Generator Issues¶
Generator Not Found¶
Node.js Version Error¶
MCC requires Node.js 24.11.1+ for ES module support:
CLI Not Found in CI¶
Docker Build Issues¶
Package Not Found¶
Pin versions in requirements.txt:
Or rebuild without cache: docker build --no-cache -t my-model .
Permission Denied¶
Local Testing Issues¶
Container Exits Immediately¶
docker logs <container-id> # Check for startup errors
docker run -it my-model /bin/bash # Debug interactively
Common causes: missing model file at /opt/ml/model/, missing Python dependencies, syntax errors in serve.py.
Health Check Fails¶
If the container is running but not responding, the server may have failed to bind to port 8080. Check that the Dockerfile exposes port 8080 and the server is configured to listen on 0.0.0.0:8080.
Inference Returns Error¶
Common causes:
- Wrong input format -- ensure
Content-Type: application/jsonheader is set - Feature count mismatch -- input must have the same number of features as training data
- Wrong data types -- ensure numeric values are floats, not strings
AWS Deployment Issues¶
ECR Authentication Failed¶
aws ecr get-login-password --region us-east-1 | \
docker login --username AWS --password-stdin \
<account-id>.dkr.ecr.us-east-1.amazonaws.com
Verify credentials: aws sts get-caller-identity
IAM Permission Denied¶
Required permissions for deployment: ecr:CreateRepository, ecr:PutImage, ecr:InitiateLayerUpload, ecr:UploadLayerPart, ecr:CompleteLayerUpload, sagemaker:CreateModel, sagemaker:CreateEndpointConfig, sagemaker:CreateEndpoint, iam:PassRole. See the generated IAM_PERMISSIONS.md for the full policy document.
Endpoint Creation Failed¶
Common causes:
- Invalid IAM role -- verify with
aws iam get-role --role-name <role> - Image not found in ECR -- verify with
aws ecr describe-images --repository-name <repo> - Insufficient instance capacity -- try a different instance type or region
Endpoint Stuck in Creating¶
If the endpoint stays in Creating status for more than 15 minutes, check CloudWatch logs. Common causes:
- Container fails to start -- check Dockerfile
CMD/ENTRYPOINTand verify port 8080 is exposed - Health check fails --
/pingmust return 200 within 2 seconds - Model loading fails -- verify model file exists in the container and format matches the handler code
To recover: delete the endpoint and redeploy.
Endpoint Returns 500¶
Common causes: exception in prediction code, wrong input format, model not loaded. Test locally first:
./do/run
curl -X POST http://localhost:8080/invocations \
-H "Content-Type: application/json" \
-d '{"instances": [[1.0, 2.0, 3.0]]}'
Model Loading Issues¶
Server rejects an argument / won't start (junk --build-* args)¶
vLLM engine args: [... --build-url https://buildkite.com/... --image-tag vllm/vllm-openai:v0.23.0
--build-pipeline ... --build-commit ...]
api_server.py: error: unrecognized arguments: --build-url ...
Cause: vLLM v0.21+ bakes build-provenance env vars (VLLM_BUILD_URL,
VLLM_IMAGE_TAG, VLLM_BUILD_PIPELINE, VLLM_BUILD_COMMIT) into the image.
A pre-fix serve script forwarded every VLLM_* env var as a CLI flag, so
these leaked in as --build-url etc. and the engine rejected them — the IC
never reaches InService.
Fix: current serve scripts use a --help introspection whitelist —
they forward only args the running engine actually accepts. If you see junk
--build-* args, your generated project predates this fix:
- Regenerate the project (picks up the fixed
code/serve.d/vllm/vllm.ejs), or hand-patch the serve script (it's EJS-free — safe to edit directly). ./do/build && ./do/push— the serve script is baked into the image, so a rebuild is required.- Redeploy. The
vLLM engine args:log line should now be clean.
This applies to SGLang too (SGLANG_* → sglang serve). The
model-server-version MCP is not at fault — it selects the image correctly;
the issue is purely serve-script arg harvesting. Full mechanism + a
troubleshooting matrix: Serving Engine Arguments & Version Troubleshooting.
Model File Not Found¶
# Verify model is in the container
docker run my-model ls -la /opt/ml/model/
# Check the COPY directive in Dockerfile
grep "COPY.*model" Dockerfile
For predictive models, the model file must be copied into the container at build time. For transformer models, the serving framework downloads the model at runtime from HuggingFace Hub.
Model Format Mismatch¶
Verify the model file format matches what you selected during generation:
| Framework | Expected formats |
|---|---|
| sklearn | .pkl, .joblib |
| xgboost | .json, .model, .ubj |
| tensorflow | SavedModel/ directory, .keras, .h5 |
The authoritative per-framework format list lives in each framework's descriptor,
templates/code/predictors.d/<framework>/manifest.json(model_formats). The table above mirrors it for convenience.
Pickle Version Mismatch¶
The Python version used to save the model must match the version in the container. Either re-save the model with a compatible protocol (pickle.dump(model, f, protocol=4)) or update the Python version in the Dockerfile.
HuggingFace Issues¶
API Timeout¶
This is expected behavior -- the generator falls back to local registry data. To skip HuggingFace API calls entirely:
Access Denied or Repository Not Found¶
For private or gated models:
- Verify the model ID at
https://huggingface.co/<model-id> - Accept the model's license agreement on HuggingFace (for gated models like Llama)
- Store the token in Secrets Manager and reference it:
ml-container-creator secrets create --type hf-token --name <label> --secret-value hf_***, thenmcc bootstrap add-secret hfToken <secret-arn>(see the plaintext-token entry below)
See HuggingFace Authentication for details.
Rate Limit Exceeded¶
Use --offline to skip API calls, or set HF_TOKEN for higher rate limits:
do/deploy refuses: "HF_TOKEN is set as a plaintext value"¶
do/deploy won't write a plaintext token into the SageMaker container Environment. SageMaker Describe* APIs return that map and CloudTrail records it. Store the token in Secrets Manager and deploy by reference:
ml-container-creator secrets create --type hf-token --name production --secret-value hf_***
mcc bootstrap add-secret hfToken <secret-arn> # or: export HF_TOKEN_ARN=<secret-arn> in do/config
./do/deploy
If the model isn't gated, run unset HF_TOKEN (it may come from your shell profile). The same applies to NGC_API_KEY, IC_ENV_HF_TOKEN and IC_ENV_NGC_API_KEY. ./do/deploy --allow-plaintext-secret keeps the old behavior with a warning. See Secrets: deploy-time.
Container exits: "failed to resolve HF_TOKEN_ARN"¶
The serving container resolves the secret at startup as the SageMaker execution role. Check that:
- The role allows
secretsmanager:GetSecretValueon the secret. The bootstrap role coversmlcc/*,ml-container-creator/*andmlcc-hf-token-*. For any other name, or for your ownROLE_ARN, add a grant.do/deploywarns when the name is outside those prefixes. - The role allows
kms:Decryptif the secret uses a customer-managed KMS key. Re-runmcc bootstrapto pick up theSecretsManagerKmsDecryptstatement. - The endpoint can reach Secrets Manager: from a VPC with no internet path, add a
com.amazonaws.<region>.secretsmanagerinterface endpoint.
LMI/DJL, llama.cpp and Triton images don't run code/serve, so they never resolve *_ARN. For gated models, stage the weights with ./do/stage.
Endpoints deployed before the secrets-by-reference fix still expose the token¶
Endpoints created by earlier releases have HF_TOKEN / NGC_API_KEY in plaintext in their container Environment. List the keys without printing values:
aws sagemaker describe-inference-component --inference-component-name <ic> \
--query 'keys(Specification.Container.Environment)'
aws sagemaker describe-model --model-name <model> --query 'keys(PrimaryContainer.Environment)'
If a secret key is listed, rotate the token at the provider, update the secret, redeploy, and delete the superseded models and inference components.
LoRA Adapter Issues¶
"Not Found" error on first adapter invocation¶
Symptoms: After do/adapter add, the first do/test returns {"detail":"Not Found"} but a second attempt succeeds.
Root cause: The adapter inference component reports InService before vLLM finishes loading LoRA weights into GPU memory. SageMaker AI's readiness check passes (base model /ping returns 200) but the adapter isn't actually ready to serve requests yet.
Workaround: Wait 30–60 seconds after do/adapter add reports success before testing. A future release will add a post-attach probe loop to confirm the adapter is serving.
Read timeout on first adapter inference¶
Symptoms: First inference after adapter load returns a "Read timeout" error, but the response body contains valid JSON.
Root cause: The first inference triggers JIT compilation of the adapter code path. Combined with thinking-mode tokens (for reasoning models like DeepSeek R1), this can exceed the default CLI read timeout.
Fix: do/test has no read-timeout flag. Wait about a minute and re-run the test; subsequent calls are fast, only the first invocation is slow:
Wrong model name in adapter test¶
Symptoms: do/test sends the adapter name (e.g., "val-sft") as the model field in the request, but vLLM returns "Not Found" because it only recognizes the base model name.
Root cause: When adapter config is detected, do/test uses ADAPTER_MODEL_NAME in the request's "model" field. However, SageMaker AI handles adapter routing at the inference component layer — vLLM doesn't need to know the adapter name. The correct value is the base MODEL_NAME (e.g., "Qwen/Qwen3-0.6B").
Workaround: Invoke directly with the base model name:
aws sagemaker-runtime invoke-endpoint \
--endpoint-name <endpoint> \
--inference-component-name <adapter-ic-name> \
--body '{"model": "Qwen/Qwen3-0.6B", "messages": [{"role": "user", "content": "Hello"}]}' \
--content-type application/json \
output.json
This will be fixed in a future release so do/test automatically uses the base model name when testing adapters.
Adapter IC stuck in "Creating" state¶
Symptoms: do/adapter add hangs waiting for the inference component to reach InService.
Common causes:
- Insufficient GPU memory — The base model + adapter don't fit in the instance's GPU memory. Try a smaller adapter rank or a larger instance.
- Invalid S3 path — The adapter weights URI doesn't exist or isn't accessible by the SageMaker AI execution role.
- Incompatible adapter — The adapter was trained with a different base model or rank than configured.
Debug:
# Check IC status and failure reason
aws sagemaker describe-inference-component \
--inference-component-name <adapter-ic-name> \
--query '[InferenceComponentStatus, FailureReason]'
# Check endpoint logs for vLLM errors
./do/logs
Kubernetes: "No adapter found" when loading a LoRA adapter (hyperpod-eks / eks)¶
Symptoms: load_lora_adapter fails with No adapter found for <path>, or vLLM tries to
download the adapter from Hugging Face.
Root cause: vLLM loads adapters from a LOCAL path. An s3:// URI (or a path that is not on
the pod's model mount) is treated as a Hub repo id. The pod has no AWS credentials.
Fix: use ./do/adapter add <name> --from-tune <t> (or --from-weights): it promotes the
adapter to its canonical version models/<org>/<base>/.mlcc-adapters/<project>/<name>/v<hash12>/
in the core bucket and loads /opt/ml/model/.mlcc-adapters/<project>/<name>/v<hash12>. If it refuses:
- model is not served from S3 —
./do/stageand./do/deploy --target hyperpod-eks --force; - prefetchEnabled: true — redeploy so MLCC turns prefetch off for LoRA + S3:
./do/deploy --target hyperpod-eks --force(add--surge-nodeon a single-node group); - no S3 model mount (eks) — set
EKS_MODEL_MOUNT=s3with a staged model and redeploy; - incompatible base — the adapter was trained on a different base than the deployment serves.
See LoRA delivery on Kubernetes.
"rank X > max_lora_rank Y" when adding an adapter¶
Symptoms: ./do/adapter add refuses with the adapter's rank and the serving limit, or vLLM
rejects the load with a rank error.
Root cause: vLLM fixes max_lora_rank at start. Managed-tune adapters are often r32/r64; the
engine default is 16.
Fix: redeploy so the limit is derived from the recorded adapters
(./do/deploy --target <target> --force), or, if VLLM_MAX_LORA_RANK is set explicitly in
do/config, raise it to at least the adapter's rank (an allowed value: 1, 8, 16, 32, 64, 128,
256, 320, 512) first. On realtime-inference the fix is the base IC's IC_ENV_VLLM_MAX_LORA_RANK
and ./do/deploy --target realtime-inference --force-ic. A rank whose LoRA buffer exceeds 10% of
the free GPU memory is refused at deploy with the numbers.
hyperpod-eks: rollout stuck Pending / deploy stops with "rollout cannot schedule"¶
Symptoms: after a pod-template change the new pod stays Pending (insufficient
nvidia.com/gpu), or ./do/deploy --target hyperpod-eks --force stops before applying.
Root cause: the instance group has one node and the serving pod holds all its GPUs, so a surge pod cannot schedule. MLCC never scales to zero to make room.
Fix: ./do/deploy --target hyperpod-eks --force --surge-node adds one node for the rollout
and removes the vacated one afterwards (the count is restored on any failure). Capacity errors
(InsufficientCapacity) are printed verbatim — retry later. If nothing changed in the pod
template, no rollout is needed and none happens.
hyperpod-eks: deploy stops with "no node has a free pod slot"¶
Symptoms: a first ./do/deploy --target hyperpod-eks prints
The serving pod cannot start on <instance-type>: no node has a free pod slot and
Nothing was applied. (./do/add-pod prints No node has room for <name> (GPUs and pod slots, …);
it also accepts --surge-node.)
Root cause: the nodes have free GPUs, but no free pod slots (the node's allocatable pods
minus its running pods). The serving pod needs one slot, plus one for its Mountpoint-S3 pod when
the model is mounted from S3. Without that slot the pod would schedule and then hang in
ContainerCreating (FailedMount, "Too many pods").
Fix: free pod slots on a node of that instance type (remove idle pods), add a node to the
instance group, or deploy on another instance type. The message prints the per-node counts and a
suggested fix. A GPU shortage alone only warns: the pod is applied and waits Pending for a GPU.
do/stage Processing Job fails for large models¶
Symptom: Processing Job exits with code 1. CloudWatch logs may show Killed or out-of-memory errors during model download. No useful error message from do/stage itself.
Cause: The default Processing Job instance (ml.m5.xlarge, 16 GB RAM) doesn't have enough memory to download and checkpoint large models (>15B parameters).
Fix: Use a larger instance:
./do/stage --instance-type ml.m5.4xlarge # 64 GB RAM — for 15B-70B models
./do/stage --instance-type ml.m5.12xlarge # 192 GB RAM — for 70B+ models
| Model Size | Required Instance |
|---|---|
| ≤15B | ml.m5.xlarge (default) |
| 15B–70B | ml.m5.4xlarge |
| 70B+ | ml.m5.12xlarge |
Also check: Your account's EBS volume quota for Processing Jobs. New accounts default to 100-500 GB, but do/stage requests 2048 GB. Request an increase to 4096 GB via:
Service Quotas → SageMaker → "Processing job maximum EBS volume size in GB"
The agent (ml-container-creator hey) checks this automatically in its health report.
vLLM container logs go dark after "engine args" on multi-GPU¶
Symptoms: Container starts, logs show CUDA compat: driver X < Y, adding compat libs and vLLM engine args: [...], then no further output. IC reports "InService" but inference returns InternalFailure. Benchmark jobs fail with 400 Bad Request.
Root cause: The vLLM image was compiled against a newer CUDA toolkit than the instance's GPU driver supports. The CUDA forward compatibility layer loads partially but fails silently during NCCL initialization for multi-GPU tensor-parallel deployments. No Python exception is raised — the process hangs or is killed by the container runtime.
Example: vllm/vllm-openai:v0.23.0 (CUDA 12.9, requires driver ≥580) on ml.g5.24xlarge (driver ~550).
Fix:
-
Downgrade to a compatible vLLM version:
-
Or use base-image-picker with
instanceTypecontext — it automatically excludes incompatible versions:
Key diagnostic: The CUDA compat log line confirms the mismatch. If you see this followed by silence (no "Loading model..." or error), it's always the driver compatibility issue.
Multi-GPU NCCL fails on plain EKS: "unhandled system error" at ncclCommInitRank¶
Symptoms: On a plain eks deploy of a tensor-parallel model (--tp-size > 1), every TP rank crashes at startup with RuntimeError: NCCL error: unhandled system error (run with NCCL_DEBUG=INFO for details) in ncclCommInitRank. Weights may load first; the failure is at communicator init.
Root cause: Kubernetes defaults a pod's /dev/shm to 64Mi. NCCL uses /dev/shm for intra-node, cross-GPU transport; 64Mi is far too small, so the communicator can't form. (hyperpod-eks doesn't hit this — the operator sizes shared memory.)
Fix: The eks Deployment template mounts a RAM-backed /dev/shm sized HP_GPU_COUNT × 8Gi. If you still see the error, confirm the mount took and raise it:
kubectl exec <pod> -- df -h /dev/shm # should show GiB, not 64M
# raise it in do/config, then redeploy:
echo 'export HP_SHM_SIZE=64Gi' >> do/config
Diagnostic: Set export NCCL_DEBUG=INFO in do/config and read the NCCL INFO/WARN lines just before the failure — they name the subsystem (shm, P2P, NVLS). A shm/shmOpen error is this issue; a P2P/IPC or NVLS error is a different, pod-capability/driver problem.
Server start fails with "Argument list too long" on every command (E2BIG)¶
Symptoms: After an S3 model download completes on plain eks, the serve log shows env, grep, python, and the server binary all failing with Argument list too long, and the engine never starts.
Root cause: aws s3 cp/sync write a progress meter to stdout. The serve script captures resolve_model's stdout as the model path and exports it into the engine's model env var; the leaked progress blob can exceed the kernel's per-string limit (MAX_ARG_STRLEN, 128 KiB), after which every execve fails with E2BIG. It is model-size probabilistic — surfaces on large, many-shard models.
Fix: Fixed in the generated code/serve (aws calls use --no-progress --only-show-errors and the downloader's stdout is isolated from the captured path). If you see this on an older generated project, mcc regenerate to pick up the fix.
SGLang won't start a quantized MoE model with LoRA enabled¶
Symptoms: Weights load, then NotImplementedError: Mxfp4MoEMethod does not expose quant info for 'flashinfer_mxfp4' from init_lora_manager.
Root cause: LoRA was enabled (HP_LORA_ENABLED=true) on an MXFP4-quantized MoE model. SGLang (as of 0.5.x) does not implement LoRA over MXFP4-quantized MoE expert modules.
Fix: If you are serving the base model, disable LoRA — set export HP_LORA_ENABLED=false in do/config and redeploy. (You can hot-patch a live pod by removing SGLANG_ENABLE_LORA/SGLANG_MAX_LORA_RANK/SGLANG_LORA_TARGET_MODULES from its ConfigMap and restarting.) If you genuinely need LoRA on such a model, it is blocked upstream in SGLang — no MLCC setting changes it.
Custom-tokenizer model refuses to start without trust-remote-code¶
Symptoms: A model whose tokenizer ships custom Python (e.g. certain MoE/multimodal models) fails at tokenizer load unless remote code is trusted.
Fix: Set the engine's trust-remote-code flag in do/config — export SGLANG_TRUST_REMOTE_CODE=true (or VLLM_TRUST_REMOTE_CODE=true). The same flag also feeds do/benchmark's tokenizer loading on the HyperPod/SageMaker path.
Adapter IC "InService" but inference returns "Failed to download model data"¶
Symptoms: do/test --adapter <name> returns ValidationError: Failed to download model data (bucket: ..., key: .../adapter-name). The IC shows InService in describe-inference-component.
Root cause: The adapter's ArtifactUrl is missing the trailing slash. SageMaker Inference Components expect S3 directory prefixes to end with / — without it, SageMaker looks for a single object at that exact key (which doesn't exist).
Fix:
-
Delete the IC and re-create with the corrected URL:
-
For adapters from
--from-registry: the template fix (2026-06-29) automatically re-adds the trailing slash for non-tar.gz adapter URIs. Regenerate the project or update thedo/adapterscript.
Additive deploy: "Contract violation: artifact-kind" or "has no load mechanism"¶
Symptoms: do/adapter add or do/add-ic exits with code 3 and one of:
Contract violation: artifact-kind: "This source is a 'full_model', but this verb accepts 'lora_adapter'" (or the reverse).Contract violation: target: "do/add-ic (accepts: full_model) has no load mechanism on hyperpod-eks … not servable on hyperpod-eks yet".
Root cause: Additive deploy follows one contract (ADR-013). The source's artifact kind picks the verb, and the target picks the load mechanism. do/adapter hot-loads LoRA adapters only. do/add-ic deploys full models only, and only on realtime-inference. On hyperpod-eks a full model goes through do/add-pod. Plain eks has no full-model verb yet (planned post-2.0).
Fix:
- A LoRA adapter:
./do/adapter add <name> --from-tune(or--from-weights s3://...,--from-registry <arn>) onrealtime-inference,hyperpod-eks, oreks. - A full model:
./do/add-ic <name> --from-tune(or--from-weights,--from-registry) onrealtime-inference, or./do/add-pod <name> --from-tune(same--from-*flags) onhyperpod-eks. - An "artifact_kind unknown … falling back to a content check" notice is informational. Raw S3 URIs, Hub repos, and older registry versions carry no recorded kind, so the verb proceeds after an advisory
adapter_config.jsoncheck. - "--weights is deprecated" / "--model-data is deprecated" notices: switch to
--from-weights. The old flags still work.
Getting Help¶
Collect diagnostics:
# Container logs
docker logs <container-id>
# SageMaker AI endpoint logs
aws logs tail /aws/sagemaker/Endpoints/<endpoint-name> --follow
# Generator debug output
DEBUG=* ml-container-creator
- GitHub Issues — report bugs
- GitHub Discussions — ask questions
- SageMaker AI Documentation — AWS reference