Skip to content

Compute or HPC node fails to join or a job never starts

Compute nodes write logs in two places, depending on how far bootstrap got before it stopped.

Early failures (before the shared filesystem is mounted)

If the node failed early, the logs are only on the node itself:

cd /root/edh_bootstrap_<INSTANCE_ID>/logs
grep -riE "error|fatal|critical" .

Note

<INSTANCE_ID> is the EC2 instance ID of the compute node. The folder name uses an underscore, edh_bootstrap_, not a dash.

Later failures (after /apps is mounted)

Once the shared filesystem is mounted, logs are copied there. You can read them from the controller or a login node without connecting to the compute host:

cd /apps/edh/<EDH_CLUSTER_ID>/shared/logs/bootstrap/compute_node/<JOB_ID>/<HOST>/
grep -riE "error|fatal|critical" .

Useful files

File What it tells you
01_user_data.sh.log First-boot user-data execution
filesystems_mount.log Shared filesystem mount failures (a very common cause)
install_required_packages.sh.log Package installation failures
03_setup_post_reboot.log / soca_final_reboot_setup.log Post-reboot configuration
messages, cloud-init-output-*.log System and cloud-init output copied for convenience

To open a single file, list the folder first, then print or scroll through the file you want:

# List everything in the shared logs folder first
ls -l /apps/edh/<EDH_CLUSTER_ID>/shared/logs/bootstrap/compute_node/<JOB_ID>/<HOST>/

# Print the first-boot user-data log
cat /apps/edh/<EDH_CLUSTER_ID>/shared/logs/bootstrap/compute_node/<JOB_ID>/<HOST>/01_user_data.sh.log

# Check the filesystem mount log (a common cause of failures)
cat /apps/edh/<EDH_CLUSTER_ID>/shared/logs/bootstrap/compute_node/<JOB_ID>/<HOST>/filesystems_mount.log

# Scroll through a long log page by page (press q to quit)
less /apps/edh/<EDH_CLUSTER_ID>/shared/logs/bootstrap/compute_node/<JOB_ID>/<HOST>/01_user_data.sh.log

# Show only the last 100 lines of a log
tail -n 100 /apps/edh/<EDH_CLUSTER_ID>/shared/logs/bootstrap/compute_node/<JOB_ID>/<HOST>/03_setup_post_reboot.log

Placeholders

  • <EDH_CLUSTER_ID> is your EDH cluster ID (for example, edh-abc123).
  • <JOB_ID> is the HPC job ID that requested the node.
  • <HOST> is the node's short hostname.