Skip to content

Install without internet access (resource mirroring)

When an EDH node starts, it installs third-party software such as Amazon DCV, the EFA driver, OpenMPI, Slurm, Python, or GPU drivers. By default, each node downloads these files from public websites (for example github.com, python.org or efa-installer.amazonaws.com).

With resource mirroring, EDH copies all of these files (about 5 GB) once into an Amazon S3 bucket in your own account, at install time. Your nodes then download everything from S3 instead of the internet.

Why use resource mirroring

Benefit
✅ No internet access needed for your nodes: run EDH in private subnets with no outbound internet, and keep downloads inside AWS
✅ More reliable startups: nodes no longer fail when a public website is slow, rate-limited, or unavailable
✅ Faster provisioning: nodes download from S3 in your Region
✅ Files are verified: EDH checks each file against its expected checksum before nodes use it
✅ Automatic fallback: if a file could not be copied, nodes still download it from its original website

Choose a mirroring method

Resource mirroring is enabled by default. When you run the installer, it asks how you want to handle external resources:

Option What happens When to choose it
Mirror resources to S3 automatically via Lambda (running in EDH VPC) at install time (recommended) AWS Lambda functions download the files from the EDH VPC and copy them to S3. Most installations.
Mirror resources to S3 automatically via Lambda (not running in VPC) at install time Same as above, but the Lambda functions run outside your VPC. Your VPC does not allow outbound internet access, even through a NAT gateway.
Mirror resources to S3 via this host (download+upload) before deploying EDH The machine where you run the installer downloads every file and uploads it to S3. Only if Lambda functions cannot reach the internet in your account. This is slower, and uses your own network connection for about 5 GB.
No S3 Mirroring Nodes download files from the internet when they start. You do not need mirroring.

With the two Lambda options, no file goes through the machine running the installer: the copy happens entirely inside AWS.

Skip the question

To choose the method in advance, edit Config.services.resources_mirroring in default_config.yml before you run the installer:

Config:
  services:
    resources_mirroring:
      enabled: true          # false = no mirroring
      method: "ask"          # ask | cloud-in-vpc-nat | cloud-no-vpc | install-host
      failure_mode: "soft"   # soft | hard
      mirror_gpu_nvidia: true
      mirror_gpu_amd: false

The full list of mirrored files is in the Parameters section of default_config.yml.

Other settings

Setting Default What it does
failure_mode soft soft: if a file cannot be copied, the installation continues and nodes download that file from the internet. hard: the installation stops and rolls back if a required file cannot be copied. Use hard when your nodes have no internet access at all.
mirror_gpu_nvidia true Copy the NVIDIA GPU drivers used by GPU virtual desktops and HPC nodes.
mirror_gpu_amd false Copy the AMD GPU drivers. Turn it on if you use AMD GPU instances.
mirror_s3_sources true Also copy files that are hosted in other S3 buckets.

Check and repair the mirror after installation

Run the edhctl mirror commands as root on the EDH controller to see what was copied and to fix any file that failed.

source /etc/environment
EDHCTL="/opt/edh/$EDH_CLUSTER_ID/cluster_manager/edhctl"

# What the last mirror run did
$EDHCTL mirror status

# For each file: mirrored, upstream-only (still uses the internet), or absent
$EDHCTL mirror keys

# Only the files that are not served from S3
$EDHCTL mirror keys --degraded-only

# Copy every missing file again (files already in S3 are skipped)
$EDHCTL mirror recover

# Copy a single file again
$EDHCTL mirror recover -k /system/openmpi/url

edhctl mirror recover is safe to run as often as you need: files already copied and verified are not downloaded again. Add --dry-run to see what would be copied without starting anything.

When to run recover

The mirror runs automatically only once, when you install EDH. Run edhctl mirror recover if edhctl mirror keys shows files that are not served from S3, for example after a temporary network issue during installation, or after you enable AMD GPU drivers.

See edhctl mirror for all options.