Install without internet access (resource mirroring)
When an EDH node starts, it installs third-party software such as Amazon DCV, the EFA driver, OpenMPI, Slurm, Python, or GPU drivers. By default, each node downloads these files from public websites (for example github.com, python.org or efa-installer.amazonaws.com).
With resource mirroring, EDH copies all of these files (about 5 GB) once into an Amazon S3 bucket in your own account, at install time. Your nodes then download everything from S3 instead of the internet.
Why use resource mirroring¶
| Benefit | |
|---|---|
| ✅ | No internet access needed for your nodes: run EDH in private subnets with no outbound internet, and keep downloads inside AWS |
| ✅ | More reliable startups: nodes no longer fail when a public website is slow, rate-limited, or unavailable |
| ✅ | Faster provisioning: nodes download from S3 in your Region |
| ✅ | Files are verified: EDH checks each file against its expected checksum before nodes use it |
| ✅ | Automatic fallback: if a file could not be copied, nodes still download it from its original website |
Choose a mirroring method¶
Resource mirroring is enabled by default. When you run the installer, it asks how you want to handle external resources:
| Option | What happens | When to choose it |
|---|---|---|
| Mirror resources to S3 automatically via Lambda (running in EDH VPC) at install time (recommended) | AWS Lambda functions download the files from the EDH VPC and copy them to S3. | Most installations. |
| Mirror resources to S3 automatically via Lambda (not running in VPC) at install time | Same as above, but the Lambda functions run outside your VPC. | Your VPC does not allow outbound internet access, even through a NAT gateway. |
| Mirror resources to S3 via this host (download+upload) before deploying EDH | The machine where you run the installer downloads every file and uploads it to S3. | Only if Lambda functions cannot reach the internet in your account. This is slower, and uses your own network connection for about 5 GB. |
| No S3 Mirroring | Nodes download files from the internet when they start. | You do not need mirroring. |
With the two Lambda options, no file goes through the machine running the installer: the copy happens entirely inside AWS.
Skip the question
To choose the method in advance, edit Config.services.resources_mirroring in default_config.yml before you run the installer:
Config:
services:
resources_mirroring:
enabled: true # false = no mirroring
method: "ask" # ask | cloud-in-vpc-nat | cloud-no-vpc | install-host
failure_mode: "soft" # soft | hard
mirror_gpu_nvidia: true
mirror_gpu_amd: false
The full list of mirrored files is in the Parameters section of default_config.yml.
Other settings¶
| Setting | Default | What it does |
|---|---|---|
failure_mode |
soft |
soft: if a file cannot be copied, the installation continues and nodes download that file from the internet. hard: the installation stops and rolls back if a required file cannot be copied. Use hard when your nodes have no internet access at all. |
mirror_gpu_nvidia |
true |
Copy the NVIDIA GPU drivers used by GPU virtual desktops and HPC nodes. |
mirror_gpu_amd |
false |
Copy the AMD GPU drivers. Turn it on if you use AMD GPU instances. |
mirror_s3_sources |
true |
Also copy files that are hosted in other S3 buckets. |
Check and repair the mirror after installation¶
Run the edhctl mirror commands as root on the EDH controller to see what was copied and to fix any file that failed.
source /etc/environment
EDHCTL="/opt/edh/$EDH_CLUSTER_ID/cluster_manager/edhctl"
# What the last mirror run did
$EDHCTL mirror status
# For each file: mirrored, upstream-only (still uses the internet), or absent
$EDHCTL mirror keys
# Only the files that are not served from S3
$EDHCTL mirror keys --degraded-only
# Copy every missing file again (files already in S3 are skipped)
$EDHCTL mirror recover
# Copy a single file again
$EDHCTL mirror recover -k /system/openmpi/url
edhctl mirror recover is safe to run as often as you need: files already copied and verified are not downloaded again. Add --dry-run to see what would be copied without starting anything.
When to run recover
The mirror runs automatically only once, when you install EDH. Run edhctl mirror recover if edhctl mirror keys shows files that are not served from S3, for example after a temporary network issue during installation, or after you enable AMD GPU drivers.
See edhctl mirror for all options.