Skip to content

Run containers in HPC jobs with Apptainer

Apptainer (formerly Singularity) lets your users package an application and everything it needs into a single image file, then run it inside any HPC job. EDH can install Apptainer automatically on every compute node, so users do not have to install anything themselves.

Containers are useful when you want to:

  • run the same software stack on every operating system and instance type,
  • share a ready-to-use environment with your team as a single file,
  • use software that is hard to install on the compute node operating system.

Apptainer runs without root privileges and works with your existing schedulers (OpenPBS, LSF, Slurm). Jobs run as the user who submitted them, with access to their usual home directory and shared file systems.

Key capabilities

Capability
✅ Installed automatically on every HPC compute node when you turn it on
✅ Fast startup: images are mounted directly, not unpacked, each time a job starts
✅ Shared image library on your EDH shared storage, organized by CPU architecture (x86 and Arm/Graviton)
✅ High-speed networking (EFA) inside containers, with no extra settings, for multi-node MPI jobs
✅ Built-in sample workflows to check that containers and EFA work on your cluster

Turn on Apptainer

Apptainer is turned off by default. Turning it on adds time to the startup of each compute node, because EDH builds Apptainer from source when the node starts. To avoid this delay, launch your compute nodes from a custom AMI that already includes it: EDH then skips the installation.

Before you install EDH: set enabled to true in the containers section of default_config.yml.

Config:
  containers:
    apptainer:
      enabled: true

On an existing EDH environment: open the Configuration Editor, find /configuration/HPC/containers/apptainer, and set enabled to true. No restart is needed: the change applies to compute nodes launched after you save it. Nodes already running are not changed.

Other settings

Setting Default What it does
enabled false Turns Apptainer on or off.
images_path /apps/containers Shared folder where your container images are stored.
install_prefix /opt/edh/<EDH cluster name>/apptainer Where Apptainer is installed on each node.
unsupported_platforms amazonlinux2023:aarch64,amazonlinux2,centos7,rhel7 Operating systems (and optionally architectures) where EDH does not install Apptainer. Nodes on these platforms start normally, without Apptainer.

Supported operating systems

Apptainer is installed on Amazon Linux 2023 (x86), RHEL 8 and 9, Rocky Linux 8 and 9, and Ubuntu 22.04 and 24.04. It is not installed on Amazon Linux 2023 on Arm (Graviton), or on Amazon Linux 2, CentOS 7 and RHEL 7.

Where to store your images

EDH creates a shared image library under /apps/containers, with one folder per CPU architecture. An image built for x86 does not run on Arm (Graviton), and the other way around.

/apps/containers/x86_64/     (also available as /apps/containers/amd64)
/apps/containers/aarch64/    (also available as /apps/containers/arm64)

On every compute node, the EDH_CONTAINER_IMAGES variable points to the folder that matches the node's architecture. Use it in your job scripts so the same script works on any instance type:

apptainer exec "$EDH_CONTAINER_IMAGES/my-app.sif" my_solver --input input.dat

Build your first image

An image is built from a short definition file. This example creates an image based on Amazon Linux 2023:

Bootstrap: docker
From: public.ecr.aws/amazonlinux/amazonlinux:2023

%post
    dnf -y install hostname procps-ng
    dnf -y clean all

%environment
    export MY_APP_HOME=/opt/my-app

Save it as my-app.def, then build the image from a job (or from an interactive session) on a compute node of the architecture you plan to use:

apptainer build "$EDH_CONTAINER_IMAGES/my-app.sif" my-app.def

Tip

Pull base images from Amazon ECR Public when you can. It does not limit anonymous downloads, unlike Docker Hub, so many jobs building at the same time do not fail.

Run a container in a job

Call apptainer exec from your job script like any other command. For example, with OpenPBS:

#!/bin/bash
#PBS -N my-container-job
#PBS -q normal
#PBS -l instance_type=c6i.xlarge

cd $PBS_O_WORKDIR
apptainer exec "$EDH_CONTAINER_IMAGES/my-app.sif" my_solver --input input.dat

You can also use Apptainer in your web-based workflows: add the apptainer exec command to the job script of your Workflow Profile.

Temporary files stay on the node

EDH stores Apptainer's download cache and temporary files on the node's local /scratch disk when there is one (or /var/tmp otherwise), so they do not fill up your shared home directory.

Use EFA and MPI inside containers

Elastic Fabric Adapter (EFA) is the low-latency network used by multi-node MPI jobs. Containers can use EFA with no extra settings.

For MPI to work across nodes, the MPI and EFA libraries inside your image must match the ones on the compute nodes. The easiest way is to build your image with the same EFA installer as your nodes. When a job requests EFA (efa_support=true), EDH copies the node's EFA installer to /apps/containers/efa-installer, and the EDH_EFA_INSTALLER_DIR variable points to it.

Use the Container EFA Benchmark (Apptainer) sample workflow as a starting point: its definition file and job script show how to build an EFA-ready image and run MPI across nodes.

Check that containers work

If Apptainer was turned on when EDH was installed, two sample workflows are available under Launch Workflow:

  • Container Test (Apptainer): runs a small container on one compute node and checks that Apptainer works as expected.
  • Container EFA Benchmark (Apptainer): runs MPI across several nodes from inside a container, and compares EFA with standard networking.

Both jobs end with RESULT: PASS (or RESULT: FAIL and the list of failed checks) in their output file.

Info

Make sure each sample workflow is assigned to an EDH Project, otherwise your users do not see it.