MCP Servers¶

ML Container Creator supports Model Context Protocol (MCP) as a configuration source. MCP servers provide configuration values -- like recommended instance types or AWS regions -- that the generator merges into its configuration chain during project generation.

How It Works¶

MCP is an open protocol that standardizes how applications communicate with external tool servers. In ML Container Creator, MCP serves as a configuration provider protocol: the generator spawns MCP servers as child processes, queries them for parameter values over stdio, and merges the results into the configuration. No LLM is in the loop -- the generator programmatically queries servers and merges results. The servers themselves are fully MCP-compliant, so any MCP client (Claude, Kiro, or your own) can also connect to them.

sequenceDiagram
    participant User
    participant Generator as ML Container Creator
    participant MCP as MCP Server (child process)

    User->>Generator: ml-container-creator
    Generator->>Generator: Load config files
    Generator->>MCP: Spawn process, handshake
    Generator->>MCP: Call get_ml_config tool
    MCP-->>Generator: Return values + choices
    Generator->>Generator: Merge MCP values
    Generator->>Generator: Load env vars, CLI args
    Generator->>User: Present prompts with MCP choices

MCP is not a numbered level in the configuration precedence chain. The generator queries MCP servers after it has loaded every other source, and uses the results as suggested values and choice lists in the prompts. A value you set explicitly (CLI option, environment variable, config file) is not overwritten by an MCP result.

MCP is entirely optional. If a server is not configured, unreachable, times out (default 10s), or returns errors, the generator logs a warning and continues without MCP values. Prompts fall back to their default choices.

Eligible Parameters¶

Only parameters with unbounded value spaces are eligible for MCP:

Parameter	MCP Eligible	Reason
`instanceType`	yes	Open-ended set of SageMaker AI instance types
`awsRegion`	yes	AWS adds new regions over time
`awsRoleArn`	yes	Arbitrary IAM role ARNs
`framework`	no	Fixed set: sklearn, xgboost, tensorflow (from `predictors.d/`), plus transformers
`modelServer`	no	Fixed set: flask, fastapi, vllm, sglang, etc.
All others	no	Bounded value spaces

MCP servers can return values for any parameter, but the generator silently discards values for ineligible parameters.

Managing MCP Servers¶

Initialize All Bundled Servers¶

ml-container-creator mcp init

Creates config/mcp.json with every bundled server pre-configured. Existing servers are preserved.

Add a Server¶

ml-container-creator mcp add team-config -- node path/to/server.js

With environment variables and options:

ml-container-creator mcp add team-config -- npx -y @corp/mcp-config \
  -e TEAM_ID=ml-platform \
  --tool-name get_approved_config \
  --limit 5

The mcp add command registers a server in your config file. The server is spawned and queried later, when you run the generator.

Add a Bundled Server¶

The generator ships with first-party MCP servers in the servers/ directory. Put --bundled (and any -e KEY=VALUE) before the action, because the mcp command passes everything after the action through as arguments:

ml-container-creator mcp --bundled add instance-sizer

Dependencies are installed automatically on first use.

List, Inspect, Remove¶

ml-container-creator mcp list              # List configured servers
ml-container-creator mcp --bundled list     # List available bundled servers
ml-container-creator mcp get team-config    # Inspect a server
ml-container-creator mcp remove team-config # Remove a server

Config File Format¶

MCP servers are configured under the mcpServers key in config/mcp.json:

{
  "framework": "sklearn",
  "modelServer": "flask",
  "mcpServers": {
    "team-config": {
      "command": "node",
      "args": ["servers/instance-sizer/index.js"],
      "env": { "TEAM_ID": "ml-platform" },
      "toolName": "get_ml_config",
      "limit": 5
    }
  }
}

Field	Type	Required	Default	Description
`command`	string	yes	--	Executable to spawn
`args`	string[]	--	`[]`	Command-line arguments
`env`	object	--	`{}`	Additional environment variables
`toolName`	string	--	`get_ml_config`	MCP tool to call
`limit`	integer	--	`10`	Max choices per parameter

When multiple servers are configured, they are queried in order. Later servers take precedence for conflicting values.

Bundled Servers¶

instance-sizer¶

The single authority for instance recommendations. It estimates the GPU memory a model needs and returns filtered, ranked SageMaker AI instance recommendations. It supports VRAM-driven sizing (when a model name is provided) and tag-based search (when an instanceSearch query is provided).

ml-container-creator mcp --bundled add instance-sizer

How it sizes a model:

Catalogued and uncatalogued models. For a Hugging Face model that is not in the bundled catalog, it reads the parameter count and weight dtype from the Hub's safetensors metadata and the architecture from config.json, including the nested text_config of multimodal models. It refuses to guess when neither is available.
KV cache for the served context. The KV-cache budget uses the context the server will actually serve (the smaller of the model's max_position_embeddings and the engine's default max_model_len), not the model's native maximum.
Deploy time. The ./do/deploy instance prompt calls the instance-sizer for its recommendation and falls back to a built-in Python port of the same formula (do/lib/python/instance_sizer.py) when no MCP server is reachable. The deploy-time right-sizing preflight (./do/deploy --dry-run, Right-Sizing) then checks the chosen instance.
Availability ranking. In discover mode it also ranks candidates by service-quota headroom, capacity reservations and Flexible Training Plans in the account/region. Capacity reservation itself is a deploy-time choice.

It also accepts optional context: CUDA version constraints (from the base image), serving-profile env overrides, and the deployment target.

region-picker¶

Suggests AWS regions based on a search term. Set REGION_SEARCH to filter by region code or location name (e.g., "europe", "tokyo", "us-west"). Without a search term, returns popular SageMaker AI regions.

ml-container-creator mcp --bundled -e REGION_SEARCH=europe add region-picker

model-picker¶

Discovers and resolves model metadata from multiple sources (HuggingFace Hub, S3, SageMaker AI Model Registry). The server still has a JumpStart resolver, but the generator refuses jumpstart:// model IDs; use the Hugging Face ID instead. Returns model configuration including architecture, parameter count, and framework compatibility.

ml-container-creator mcp --bundled add model-picker

base-image-picker¶

Recommends base Docker images for serving frameworks (vLLM, SGLang, TensorRT-LLM, DJL) based on three compatibility dimensions:

Endpoint architecture — Instance family → GPU driver version → CUDA compatibility
Inference AMI version — More precise driver mapping than instance family alone
Model architecture — Model type → minimum transformers version → minimum framework version

Returns the 3 most recent compatible versions after filtering. Incompatible images (e.g., vLLM v0.23.0 on g5 instances with driver 550) are excluded.

ml-container-creator mcp --bundled add base-image-picker

Driver-Aware Selection¶

When instanceType is provided in context, the server resolves the fleet GPU driver version from a static catalog (fleet-drivers.json) and excludes images compiled against a newer CUDA toolkit than the driver supports. For multi-GPU deployments (TP > 1), images requiring the CUDA forward compatibility layer are excluded entirely — compat mode fails silently on NCCL multi-GPU communication.

Instance Family	Fleet Driver	Max CUDA	Compatible vLLM
g4dn	~535	12.2	≤v0.18.x
g5, p4d	~550	12.4	≤v0.21.x
g6, g6e	~560	12.6	≤v0.22.x
p5, p5e	~580	12.9	All

Model Architecture Support¶

When modelId is provided in context, the server resolves the model's architecture class (e.g., Qwen3ForCausalLM, DeepseekV3ForCausalLM) and excludes framework versions that predate support for that architecture.

Architecture	vLLM Since	SGLang Since
LlamaForCausalLM	v0.4.0	v0.3.0
Qwen2ForCausalLM	v0.6.0	—
Qwen3ForCausalLM	v0.20.0	v0.6.0
DeepseekV3ForCausalLM	v0.19.0	v0.5.0
Gemma4ForCausalLM	v0.23.0	—
GptOssForCausalLM	v0.22.0	—

Context Parameters¶

Parameter	Type	Description
`instanceType`	string	Triggers driver filtering (e.g., `ml.g5.24xlarge`)
`inferenceAmiVersion`	string	More precise driver lookup (overrides instance family)
`driverVersion`	string	Explicit override (skips all lookups)
`tensorParallelSize`	number	Affects compat eligibility — TP > 1 excludes compat-only images
`modelId`	string	Triggers model architecture filtering
`modelArchitecture`	string	Direct architecture class (skips HF lookup)

CUDA Forward Compatibility Failure Mode

Images requiring a newer driver than the fleet provides will log CUDA compat: driver X < Y, adding compat libs at container startup, then silently hang on multi-GPU tensor-parallel deployments (NCCL init failure with no log output). This is why TP > 1 incompatible images are excluded rather than warned.

workload-picker¶

Provides named benchmark workload profiles for do/benchmark. Instead of manually configuring concurrency, token distributions, and request counts, the workload-picker server defines reusable traffic patterns that model real-world inference scenarios.

ml-container-creator mcp --bundled add workload-picker

Unlike other bundled servers, workload-picker is not queried during project generation. It is queried at runtime by do/benchmark --workload <name> to resolve all benchmark parameters. No manual benchmark configuration is needed in do/config.

Tools¶

Tool	Description
`list_workloads`	Returns all available workload names with descriptions and use cases
`get_workload_profile`	Returns full benchmark parameters for a named workload

Available Workloads¶

Workload	Use Case	Input Tokens	Output Tokens	Streaming	Concurrency Levels
`sample`	POC sample workload	100	100	yes	2
`multi-turn-chat`	Interactive chat, low latency	550	150	yes	1, 4, 8, 16
`rag-document-qa`	RAG, document Q&A	2000	500	yes	1, 4, 8
`agent-tool-calling`	Agents, structured output	200	100	no	1, 4, 8, 16, 32
`long-context-scaling`	Long context, summarization	8000	1000	yes	1, 2, 4
`production-traffic-mix`	Production fleet sizing	1000	300	yes	4, 8, 16, 32
`shared-system-prompt`	Prefix caching validation	1000	200	yes	4, 8, 16, 32

Usage¶

# Run benchmarks with a named workload
do/benchmark --workload multi-turn-chat

# The workload resolves all parameters at runtime:
#   concurrency, input/output tokens, streaming, dataset type
# S3 paths come from your bootstrap profile.
# No env vars to set.

The workload catalog lives at servers/workload-picker/catalogs/workload-profiles.json. Add custom workloads by appending entries to that file.

draft-model-picker¶

Provides a catalog of known speculative-decoding draft models compatible with MLCC's HyperPod EKS deployment target. Instead of hunting HuggingFace for a draft head that matches your target model, the draft-model-picker server lets you (or an MCP-connected agent) browse, look up, and get a recommendation for a draft model by target model ID and algorithm.

ml-container-creator mcp --bundled add draft-model-picker

Like workload-picker, draft-model-picker is not queried during project generation. It is queried at runtime — do/draft list reads the same catalog to browse draft models, and any MCP client (Kiro, Claude, your own) can call its tools directly. See Speculative Decoding on HyperPod EKS for the do/draft workflow.

Tools¶

Tool	Description
`list_draft_models`	List known draft models. Optional filters: `target_model` (partial match) and `algorithm` (`eagle3`, `eagle2`, `eagle`, `draft-model`, `ngram`, `mtp`).
`get_draft_model`	Return full metadata for one draft model by its HuggingFace model ID (`hf_id`).
`recommend_draft`	Recommend the best draft model for a given target model ID, ranked by algorithm quality (eagle3 first), plus all alternatives.

Example: asking the agent for a recommendation¶

Because the server is MCP-compliant, you can ask a connected agent a natural-language question and it will call recommend_draft under the hood:

You: What draft model should I use for Llama-3.1-8B?

The agent calls recommend_draft with target_model: "Llama-3.1-8B" and gets back the top-ranked match (e.g. thoughtworks/Llama-3.1-8B-Instruct-Eagle3, algorithm eagle3, supported by both vLLM and SGLang) along with every alternative. You can then wire it in:

./do/draft set thoughtworks/Llama-3.1-8B-Instruct-Eagle3 --deploy

The catalog lives at servers/lib/catalogs/draft-models.json. Add draft models by appending entries there (see Registries and Catalogs — draft-models.json).

endpoint-picker¶

Discovers InService SageMaker AI real-time endpoints with available capacity for inference-component attachment. Tool: get_inference_endpoints (discover mode).

ml-container-creator mcp --bundled add endpoint-picker

hyperpod-cluster-picker¶

Discovers available SageMaker HyperPod EKS clusters for the hyperpod-eks deployment target. Tool: get_hyperpod_clusters (discover mode). ./do/deploy --target hyperpod-eks uses it to list clusters with their GPU capacity.

ml-container-creator mcp --bundled add hyperpod-cluster-picker

adapter-picker¶

Discovery-only server that searches the Hugging Face Hub for PEFT/LoRA adapters compatible with a base model. Tools: search_hf_adapters, get_adapter_metadata, recommend_adapter. Pair it with ./do/adapter add <name> --from-hub <hf-repo-id>.

ml-container-creator mcp --bundled add adapter-picker

model-registry¶

Queries registered models, adapters, datasets and evaluators from the SageMaker Model Registry. Tools: get_model_registry, get_model_version, list_datasets, get_dataset, list_dataset_versions, list_evaluators, get_evaluator.

ml-container-creator mcp --bundled add model-registry

e2e-status¶

Optional server that surfaces end-to-end validation status from the CI DynamoDB table (see CI Integration). Tools: get_e2e_status, list_e2e_runs.

ml-container-creator mcp --bundled add e2e-status

agent-knowledge¶

Backs the advisory agent (ml-container-creator hey) with the script reference, configuration docs, troubleshooting patterns and the capability matrix. Tool: query_knowledge.

ml-container-creator mcp --bundled add agent-knowledge

reasoning¶

A stateless reasoning/interpretation tool powered by Amazon Bedrock. Pass context, optional structured data and an objective (explain, recommend, diagnose, plan). Tool: interpret.

ml-container-creator mcp --bundled add reasoning

Smart Mode (Amazon Bedrock)¶

Servers built on the shared picker factory (servers/lib/create-picker-server.js), and the instance-sizer and region-picker, support an optional smart mode that queries Amazon Bedrock for context-aware recommendations instead of returning static lists. Set BEDROCK_SMART=true in the server's environment to enable it. If the Bedrock call fails, the server falls back to static recommendations.

ml-container-creator mcp --bundled -e BEDROCK_SMART=true add instance-sizer

Configuration¶

Environment Variable	Default	Description
`BEDROCK_SMART`	`false`	Enable Bedrock-powered recommendations
`BEDROCK_MODEL`	`global.anthropic.claude-sonnet-4-20250514-v1:0`	Bedrock model ID
`BEDROCK_REGION`	`us-east-1`	AWS region for Bedrock API calls

The default model uses the global cross-region inference profile, which routes requests to the nearest available region. You can override this with any Bedrock model ID that supports the Messages API.

Prerequisites¶

AWS credentials configured (via environment, profile, or IAM role)
Access to the specified Bedrock model enabled in your account

IAM Permissions¶

The calling identity needs bedrock:InvokeModel on the inference profile:

{
    "Effect": "Allow",
    "Action": "bedrock:InvokeModel",
    "Resource": "arn:aws:bedrock:*:*:inference-profile/global.anthropic.claude-sonnet-4-20250514-v1:0"
}

Writing a Custom MCP Server¶

Any process that speaks the MCP protocol over stdio can serve as a configuration provider. Your server needs to:

Handle the MCP initialize handshake
Register a tool (default name: get_ml_config)
Accept { parameters, limit, context } as tool input
Return { values, choices } as a JSON text response

Tool Input¶

{
  "parameters": ["instanceType", "awsRoleArn", "awsRegion"],
  "limit": 10,
  "context": {
    "framework": "transformers",
    "modelServer": "vllm"
  }
}

parameters -- which unbounded parameter names the generator is requesting
limit -- maximum number of choices to return per parameter
context -- current configuration state for informed recommendations

Tool Response¶

{
  "values": {
    "instanceType": "ml.g5.xlarge"
  },
  "choices": {
    "instanceType": [
      "ml.g5.xlarge",
      "ml.g5.2xlarge",
      "ml.g4dn.xlarge"
    ]
  }
}

values -- recommended default value per parameter (merged into config)
choices -- list of options per parameter (shown during prompting)

Both fields are optional. A server may return only values, only choices, or both.

Example Server¶

import { McpServer } from '@modelcontextprotocol/sdk/server/mcp.js'
import { StdioServerTransport } from '@modelcontextprotocol/sdk/server/stdio.js'
import { z } from 'zod'

const server = new McpServer({ name: 'my-config-server', version: '1.0.0' })

server.tool(
    'get_ml_config',
    'Returns ML configuration values',
    {
        parameters: z.array(z.string()),
        limit: z.number().int().positive().default(10),
        context: z.record(z.string(), z.any()).optional()
    },
    async ({ parameters, limit, context }) => {
        const values = {}
        const choices = {}

        // Your logic here — query a database, call an API, read a file, etc.

        return {
            content: [{
                type: 'text',
                text: JSON.stringify({ values, choices })
            }]
        }
    }
)

const transport = new StdioServerTransport()
await server.connect(transport)

Register it:

ml-container-creator mcp add my-server -- node path/to/my-server.js

See servers/README.md for the full directory structure and license requirements for bundled servers.