Troubleshooting¶

Common issues and solutions when using ML Container Creator.

Quick Reference¶

Issue	Fix
`SyntaxError: Unexpected token 'export'`	`nvm use node` (requires Node.js 24.11.1+)
Generator not found	`npm link` in the project directory
Docker build fails: package not found	Pin versions in `requirements.txt`
Container exits immediately	`docker logs <id>` to check for errors
`/ping` returns connection refused	Verify container is running and port 8080 is exposed
ECR authentication failed	`aws ecr get-login-password` (see ECR Auth)
Endpoint stuck in Creating	Check CloudWatch logs for health check or model loading failures
Model file not found in container	Verify `COPY` directive in Dockerfile targets `/opt/ml/model/`
Adapter "Not Found" on first call	Wait 60s after `do/adapter add` (see LoRA Adapter Issues)
Adapter read timeout	`./do/test --cli-read-timeout 120`
Adapter wrong model name	Use base model name, not adapter name (see Wrong Model Name)
HuggingFace API timeout	Use `--offline` flag
HuggingFace access denied	Verify token and model license agreement

Generator Issues¶

Generator Not Found¶

Error: @aws/ml-container-creator generator not found

cd ml-container-creator
npm link
ml-container-creator --help   # Should show available commands

Node.js Version Error¶

SyntaxError: Unexpected token 'export'
SyntaxError: Cannot use import statement outside a module

MCC requires Node.js 24.11.1+ for ES module support:

nvm install node
nvm use node
node --version    # Must be 24.11.1+

CLI Not Found in CI¶

npm install -g @aws/ml-container-creator
ml-container-creator --help

Docker Build Issues¶

Package Not Found¶

ERROR: Could not find a version that satisfies the requirement scikit-learn

Pin versions in requirements.txt:

scikit-learn==1.3.0
numpy==1.24.0

Or rebuild without cache: docker build --no-cache -t my-model .

Permission Denied¶

ERROR: failed to copy files: permission denied

chmod 644 code/*
chmod 755 code/*.sh
docker build -t my-model .

Local Testing Issues¶

Container Exits Immediately¶

docker logs <container-id>           # Check for startup errors
docker run -it my-model /bin/bash    # Debug interactively

Common causes: missing model file at /opt/ml/model/, missing Python dependencies, syntax errors in serve.py.

Health Check Fails¶

curl: (7) Failed to connect to localhost port 8080

docker ps                            # Is the container running?
docker logs <container-id>           # Check for errors

If the container is running but not responding, the server may have failed to bind to port 8080. Check that the Dockerfile exposes port 8080 and the server is configured to listen on 0.0.0.0:8080.

Inference Returns Error¶

docker logs <container-id>           # Check for the full traceback

Common causes:

Wrong input format -- ensure Content-Type: application/json header is set
Feature count mismatch -- input must have the same number of features as training data
Wrong data types -- ensure numeric values are floats, not strings

AWS Deployment Issues¶

ECR Authentication Failed¶

Error: no basic auth credentials

aws ecr get-login-password --region us-east-1 | \
  docker login --username AWS --password-stdin \
  <account-id>.dkr.ecr.us-east-1.amazonaws.com

Verify credentials: aws sts get-caller-identity

IAM Permission Denied¶

Error: User is not authorized to perform: ecr:CreateRepository

Required permissions for deployment: ecr:CreateRepository, ecr:PutImage, ecr:InitiateLayerUpload, ecr:UploadLayerPart, ecr:CompleteLayerUpload, sagemaker:CreateModel, sagemaker:CreateEndpointConfig, sagemaker:CreateEndpoint, iam:PassRole. See the generated IAM_PERMISSIONS.md for the full policy document.

Endpoint Creation Failed¶

aws logs tail /aws/sagemaker/Endpoints/<endpoint-name> --follow

Common causes:

Invalid IAM role -- verify with aws iam get-role --role-name <role>
Image not found in ECR -- verify with aws ecr describe-images --repository-name <repo>
Insufficient instance capacity -- try a different instance type or region

Endpoint Stuck in Creating¶

If the endpoint stays in Creating status for more than 15 minutes, check CloudWatch logs. Common causes:

Container fails to start -- check Dockerfile CMD/ENTRYPOINT and verify port 8080 is exposed
Health check fails -- /ping must return 200 within 2 seconds
Model loading fails -- verify model file exists in the container and format matches the handler code

To recover: delete the endpoint and redeploy.

./do/clean endpoint
./do/deploy

Endpoint Returns 500¶

aws logs tail /aws/sagemaker/Endpoints/<endpoint-name> --follow

Common causes: exception in prediction code, wrong input format, model not loaded. Test locally first:

./do/run
curl -X POST http://localhost:8080/invocations \
  -H "Content-Type: application/json" \
  -d '{"instances": [[1.0, 2.0, 3.0]]}'

Model Loading Issues¶

Server rejects an argument / won't start (junk `--build-*` args)¶

vLLM engine args: [... --build-url https://buildkite.com/... --image-tag vllm/vllm-openai:v0.23.0
                   --build-pipeline ... --build-commit ...]
api_server.py: error: unrecognized arguments: --build-url ...

Cause: vLLM v0.21+ bakes build-provenance env vars (VLLM_BUILD_URL, VLLM_IMAGE_TAG, VLLM_BUILD_PIPELINE, VLLM_BUILD_COMMIT) into the image. A pre-fix serve script forwarded every VLLM_* env var as a CLI flag, so these leaked in as --build-url etc. and the engine rejected them — the IC never reaches InService.

Fix: current serve scripts use a --help introspection whitelist — they forward only args the running engine actually accepts. If you see junk --build-* args, your generated project predates this fix:

Regenerate the project (picks up the fixed code/serve.d/vllm/vllm.ejs), or hand-patch the serve script (it's EJS-free — safe to edit directly).
./do/build && ./do/push — the serve script is baked into the image, so a rebuild is required.
Redeploy. The vLLM engine args: log line should now be clean.

This applies to SGLang too (SGLANG_* → sglang serve). The model-server-version MCP is not at fault — it selects the image correctly; the issue is purely serve-script arg harvesting. Full mechanism + a troubleshooting matrix: Serving Engine Arguments & Version Troubleshooting.

Model File Not Found¶

FileNotFoundError: No such file or directory: '/opt/ml/model/model.pkl'

# Verify model is in the container
docker run my-model ls -la /opt/ml/model/

# Check the COPY directive in Dockerfile
grep "COPY.*model" Dockerfile

For predictive models, the model file must be copied into the container at build time. For transformer models, the serving framework downloads the model at runtime from HuggingFace Hub.

Model Format Mismatch¶

ValueError: Model format not recognized

Verify the model file format matches what you selected during generation:

Framework	Expected formats
sklearn	`.pkl`, `.joblib`
xgboost	`.json`, `.model`, `.ubj`
tensorflow	`SavedModel/` directory, `.keras`, `.h5`

The authoritative per-framework format list lives in each framework's descriptor, templates/code/predictors.d/<framework>/manifest.json (model_formats). The table above mirrors it for convenience.

Pickle Version Mismatch¶

ValueError: unsupported pickle protocol: 5

The Python version used to save the model must match the version in the container. Either re-save the model with a compatible protocol (pickle.dump(model, f, protocol=4)) or update the Python version in the Dockerfile.

HuggingFace Issues¶

API Timeout¶

Warning: HuggingFace API timeout, checking local registry

This is expected behavior -- the generator falls back to local registry data. To skip HuggingFace API calls entirely:

ml-container-creator --offline

Access Denied or Repository Not Found¶

Warning: Model 'my-org/my-model' not found on HuggingFace

For private or gated models:

Verify the model ID at https://huggingface.co/<model-id>
Accept the model's license agreement on HuggingFace (for gated models like Llama)
Provide a valid token: --hf-token='$HF_TOKEN'

See HuggingFace Authentication for details.

Rate Limit Exceeded¶

Warning: HuggingFace API rate limit exceeded, using cached data

Use --offline to skip API calls, or set HF_TOKEN for higher rate limits:

export HF_TOKEN=hf_your_token_here
ml-container-creator

LoRA Adapter Issues¶

"Not Found" error on first adapter invocation¶

Symptoms: After do/adapter add, the first do/test returns {"detail":"Not Found"} but a second attempt succeeds.

Root cause: The adapter inference component reports InService before vLLM finishes loading LoRA weights into GPU memory. SageMaker AI's readiness check passes (base model /ping returns 200) but the adapter isn't actually ready to serve requests yet.

Workaround: Wait 30–60 seconds after do/adapter add reports success before testing. A future release will add a post-attach probe loop to confirm the adapter is serving.

# Wait, then test
./do/adapter add my-sft --from-tune
sleep 60
./do/test

Read timeout on first adapter inference¶

Symptoms: First inference after adapter load returns a "Read timeout" error, but the response body contains valid JSON.

Root cause: The first inference triggers JIT compilation of the adapter code path. Combined with thinking-mode tokens (for reasoning models like DeepSeek R1), this can exceed the default CLI read timeout.

Fix: Increase the read timeout:

./do/test --cli-read-timeout 120

Subsequent calls will be fast — only the first invocation is slow.

Wrong model name in adapter test¶

Symptoms: do/test sends the adapter name (e.g., "val-sft") as the model field in the request, but vLLM returns "Not Found" because it only recognizes the base model name.

Root cause: When adapter config is detected, do/test uses ADAPTER_MODEL_NAME in the request's "model" field. However, SageMaker AI handles adapter routing at the inference component layer — vLLM doesn't need to know the adapter name. The correct value is the base MODEL_NAME (e.g., "Qwen/Qwen3-0.6B").

Workaround: Invoke directly with the base model name:

aws sagemaker-runtime invoke-endpoint \
  --endpoint-name <endpoint> \
  --inference-component-name <adapter-ic-name> \
  --body '{"model": "Qwen/Qwen3-0.6B", "messages": [{"role": "user", "content": "Hello"}]}' \
  --content-type application/json \
  output.json

This will be fixed in a future release so do/test automatically uses the base model name when testing adapters.

Adapter IC stuck in "Creating" state¶

Symptoms: do/adapter add hangs waiting for the inference component to reach InService.

Common causes:

Insufficient GPU memory — The base model + adapter don't fit in the instance's GPU memory. Try a smaller adapter rank or a larger instance.
Invalid S3 path — The adapter weights URI doesn't exist or isn't accessible by the SageMaker AI execution role.
Incompatible adapter — The adapter was trained with a different base model or rank than configured.

Debug:

# Check IC status and failure reason
aws sagemaker describe-inference-component \
  --inference-component-name <adapter-ic-name> \
  --query '[InferenceComponentStatus, FailureReason]'

# Check endpoint logs for vLLM errors
./do/logs

`do/stage` Processing Job fails for large models¶

Symptom: Processing Job exits with code 1. CloudWatch logs may show Killed or out-of-memory errors during model download. No useful error message from do/stage itself.

Cause: The default Processing Job instance (ml.m5.xlarge, 16 GB RAM) doesn't have enough memory to download and checkpoint large models (>15B parameters).

Fix: Use a larger instance:

./do/stage --instance-type ml.m5.4xlarge   # 64 GB RAM — for 15B-70B models
./do/stage --instance-type ml.m5.12xlarge  # 192 GB RAM — for 70B+ models

Model Size	Required Instance
≤15B	`ml.m5.xlarge` (default)
15B–70B	`ml.m5.4xlarge`
70B+	`ml.m5.12xlarge`

Also check: Your account's EBS volume quota for Processing Jobs. New accounts default to 100-500 GB, but do/stage requests 2048 GB. Request an increase to 4096 GB via:

Service Quotas → SageMaker → "Processing job maximum EBS volume size in GB"

The agent (ml-container-creator hey) checks this automatically in its health report.

vLLM container logs go dark after "engine args" on multi-GPU¶

Symptoms: Container starts, logs show CUDA compat: driver X < Y, adding compat libs and vLLM engine args: [...], then no further output. IC reports "InService" but inference returns InternalFailure. Benchmark jobs fail with 400 Bad Request.

Root cause: The vLLM image was compiled against a newer CUDA toolkit than the instance's GPU driver supports. The CUDA forward compatibility layer loads partially but fails silently during NCCL initialization for multi-GPU tensor-parallel deployments. No Python exception is raised — the process hangs or is killed by the container runtime.

Example: vllm/vllm-openai:v0.23.0 (CUDA 12.9, requires driver ≥580) on ml.g5.24xlarge (driver ~550).

Fix:

Downgrade to a compatible vLLM version:

# Check driver compatibility table in MCP Servers docs
# For g5 (driver ~550): use vLLM ≤v0.21.x
sed -i '' 's/vllm-openai:v0.23.0/vllm-openai:v0.20.2/' Dockerfile
./do/build && ./do/push

Or use base-image-picker with instanceType context — it automatically excludes incompatible versions:

ml-container-creator mcp add base-image-picker --bundled
# During generation, base-image-picker filters by driver compatibility

Key diagnostic: The CUDA compat log line confirms the mismatch. If you see this followed by silence (no "Loading model..." or error), it's always the driver compatibility issue.

Multi-GPU NCCL fails on plain EKS: "unhandled system error" at `ncclCommInitRank`¶

Symptoms: On a plain eks deploy of a tensor-parallel model (--tp-size > 1), every TP rank crashes at startup with RuntimeError: NCCL error: unhandled system error (run with NCCL_DEBUG=INFO for details) in ncclCommInitRank. Weights may load first; the failure is at communicator init.

Root cause: Kubernetes defaults a pod's /dev/shm to 64Mi. NCCL uses /dev/shm for intra-node, cross-GPU transport; 64Mi is far too small, so the communicator can't form. (hyperpod-eks doesn't hit this — the operator sizes shared memory.)

Fix: The eks Deployment template mounts a RAM-backed /dev/shm sized HP_GPU_COUNT × 8Gi. If you still see the error, confirm the mount took and raise it:

kubectl exec <pod> -- df -h /dev/shm      # should show GiB, not 64M
# raise it in do/config, then redeploy:
echo 'export HP_SHM_SIZE=64Gi' >> do/config

Diagnostic: Set export NCCL_DEBUG=INFO in do/config and read the NCCL INFO/WARN lines just before the failure — they name the subsystem (shm, P2P, NVLS). A shm/shmOpen error is this issue; a P2P/IPC or NVLS error is a different, pod-capability/driver problem.

Server start fails with "Argument list too long" on every command (E2BIG)¶

Symptoms: After an S3 model download completes on plain eks, the serve log shows env, grep, python, and the server binary all failing with Argument list too long, and the engine never starts.

Root cause: aws s3 cp/sync write a progress meter to stdout. The serve script captures resolve_model's stdout as the model path and exports it into the engine's model env var; the leaked progress blob can exceed the kernel's per-string limit (MAX_ARG_STRLEN, 128 KiB), after which every execve fails with E2BIG. It is model-size probabilistic — surfaces on large, many-shard models.

Fix: Fixed in the generated code/serve (aws calls use --no-progress --only-show-errors and the downloader's stdout is isolated from the captured path). If you see this on an older generated project, mcc regenerate to pick up the fix.

SGLang won't start a quantized MoE model with LoRA enabled¶

Symptoms: Weights load, then NotImplementedError: Mxfp4MoEMethod does not expose quant info for 'flashinfer_mxfp4' from init_lora_manager.

Root cause: LoRA was enabled (HP_LORA_ENABLED=true) on an MXFP4-quantized MoE model. SGLang (as of 0.5.x) does not implement LoRA over MXFP4-quantized MoE expert modules.

Fix: If you are serving the base model, disable LoRA — set export HP_LORA_ENABLED=false in do/config and redeploy. (You can hot-patch a live pod by removing SGLANG_ENABLE_LORA/SGLANG_MAX_LORA_RANK/SGLANG_LORA_TARGET_MODULES from its ConfigMap and restarting.) If you genuinely need LoRA on such a model, it is blocked upstream in SGLang — no MLCC setting changes it.

Custom-tokenizer model refuses to start without trust-remote-code¶

Symptoms: A model whose tokenizer ships custom Python (e.g. certain MoE/multimodal models) fails at tokenizer load unless remote code is trusted.

Fix: Set the engine's trust-remote-code flag in do/config — export SGLANG_TRUST_REMOTE_CODE=true (or VLLM_TRUST_REMOTE_CODE=true). The same flag also feeds do/benchmark's tokenizer loading on the HyperPod/SageMaker path.

Adapter IC "InService" but inference returns "Failed to download model data"¶

Symptoms: do/test --adapter <name> returns ValidationError: Failed to download model data (bucket: ..., key: .../adapter-name). The IC shows InService in describe-inference-component.

Root cause: The adapter's ArtifactUrl is missing the trailing slash. SageMaker Inference Components expect S3 directory prefixes to end with / — without it, SageMaker looks for a single object at that exact key (which doesn't exist).