GitHub

Installation

Palace can be built and installed using the Spack HPC package manager, following the instructions in the Build using Spack section. Containerized builds are possible with Singularity/Apptainer, described in Build using Singularity/Apptainer. Alternatively, compiling from source using CMake is described in Build from source.

Build using Spack

Palace is a registered package in the built-in Spack package repository. To install the solver, follow the instructions for setting up Spack on your system and run:

spack install palace

More information about the available configuration options and dependencies can be found using spack info palace.

Build using Singularity/Apptainer

Palace can be built in a Singularity/Apptainer container for HPC environments supporting the Singularity/Apptainer container system. To build the container using the provided definition file in the singularity/ directory, first set up Singularity/Apptainer on your system and subsequently run:

singularity build palace.sif <SOURCE_DIR>/singularity/singularity.def

where the repository source code has been cloned to <SOURCE_DIR>. For more information about Singularity/Apptainer, see the Quick Start guide in the Singularity/Apptainer documentation.

Build from source

A build from source requires the following prerequisites installed on your system:

  • CMake version 3.21 or later
  • C++17 compatible C++ compiler
  • C and Fortran (optional) compilers for dependency builds
  • MPI distribution
  • BLAS, LAPACK libraries (described below in Math libraries)
  • CUDA Toolkit or ROCm installation (optional, for GPU support only)

In addition, builds from source require the following system packages which are typically already installed and are available from most package managers (apt, dnf, brew, etc.):

Quick start

To start, clone the code using

git clone https://github.com/awslabs/palace.git

Then, a build using the default options can be performed by running the following from within the directory where the repository was cloned:

mkdir build && cd build
cmake ..
make -j

This installs the binary executable in build/bin/.

Configuration options

To configure a Palace build in <BUILD_DIR> using the source code in <SOURCE_DIR>, run:

mkdir <BUILD_DIR> && cd <BUILD_DIR>
cmake [OPTIONS] <SOURCE_DIR>

Here, [OPTIONS] is a list of options passed to cmake of the form -D<VARIABLE>=<VALUE>. The Palace build respects standard CMake variables, including:

  • CMAKE_CXX_COMPILER, CMAKE_C_COMPILER, and CMAKE_Fortran_COMPILER which define the desired compilers.
  • CMAKE_CXX_FLAGS, CMAKE_C_FLAGS, and CMAKE_Fortran_FLAGS which define the corresponding compiler flags.
  • CMAKE_CUDA_COMPILER, CMAKE_CUDA_FLAGS, CMAKE_CUDA_ARCHITECTURES, and the corresponding CMAKE_HIP_COMPILER, CMAKE_HIP_FLAGS, and CMAKE_HIP_ARCHITECTURES for GPU-accelerated builds with CUDA or HIP.
  • CMAKE_INSTALL_PREFIX which specifies the path for installation (if none is provided, defaults to <BUILD_DIR>).
  • CMAKE_BUILD_TYPE which defines the build type such as Release, Debug, RelWithDebInfo, and MinSizeRel (Release if not otherwise specified).
  • BUILD_SHARED_LIBS which is a flag to create shared libraries for dependency library builds instead of static libraries (OFF by default).
  • CMAKE_PREFIX_PATH which lists directories specifying installation prefixes to be searched for dependencies.
  • CMAKE_INSTALL_RPATH and CMAKE_INSTALL_RPATH_USE_LINK_PATH which configure the rpath for installed library and executable targets.

Additional build options are (with default values in brackets):

  • PALACE_WITH_64BIT_INT [OFF] : Build with 64-bit integer support
  • PALACE_WITH_OPENMP [OFF] : Use OpenMP for shared-memory parallelism
  • PALACE_WITH_CUDA [OFF] : Use CUDA for NVIDIA GPU support
  • PALACE_WITH_HIP [OFF] : Use HIP for AMD or NVIDIA GPU support
  • PALACE_WITH_GPU_AWARE_MPI [OFF] : Option to set if MPI distribution is GPU aware
  • PALACE_WITH_SUPERLU [ON] : Build with SuperLU_DIST sparse direct solver
  • PALACE_WITH_STRUMPACK [OFF] : Build with STRUMPACK sparse direct solver
  • PALACE_WITH_MUMPS [OFF] : Build with MUMPS sparse direct solver
  • PALACE_WITH_SLEPC [ON] : Build with SLEPc eigenvalue solver
  • PALACE_WITH_ARPACK [OFF] : Build with ARPACK eigenvalue solver
  • PALACE_WITH_LIBXSMM [ON] : Build with LIBXSMM backend for libCEED
  • PALACE_WITH_MAGMA [ON] : Build with MAGMA backend for libCEED
  • PALACE_WITH_GSLIB [ON] : Build with GSLIB library for high-order field interpolation
  • PALACE_WITH_SUNDIALS [ON] : Build with SUNDIALS ODE solver library

The build step is invoked by running (for example with 4 make threads)

make -j 4

or

cmake --build . -- -j 4

which installs the binary executable in ${CMAKE_INSTALL_PREFIX}/bin/.

Math libraries

During the configure step, the build system will try to detect system installations of BLAS and LAPACK libraries depending on the system architecture according to the following procedure:

  • For x86_64 systems:

    • If the MKLROOT environment variable is set, looks for an Intel MKL installation.
    • If the AOCL_DIR or AOCLROOT environment variables are set, looks for an AMD Optimizing CPU Libraries (AOCL) installation of BLIS and libFLAME.
    • Otherwise, tries to locate an installation of OpenBLAS which is permissively licensed and available from most package managers.

  • For aarch64/arm64 systems:

If the installation path of OpenBLAS is non-standard or is not found by default, it can be set using the OPENBLAS_DIR or OPENBLASROOT environment variables, or added to CMAKE_PREFIX_PATH when calling CMake.

It is recommended in most cases to use serial BLAS and LAPACK builds (not multithreaded), as the standard parallelization approach in Palace is to use pure MPI parallelism.

Dependencies

Palace leverages the MFEM finite element discretization library. It always configures and builds its own installation of MFEM internally in order to support the most up to date features and patches. Likewise, Palace will always build its own installation of libCEED, and GSLIB, when PALACE_WITH_GSLIB=ON.

As part of the Build from source, the CMake build will automatically build and install a small number of third-party dependencies before building Palace. The source code for these dependencies is downloaded during the build process:

For solving eigenvalue problems, at least one of SLEPc or ARPACK-NG must be specified. Typically only one of the SuperLU_DIST, STRUMPACK, and MUMPS dependencies is required but all can be built so the user can decide at runtime which solver to use.

For unit testing, Palace relies on the Catch2 library, which is automatically downloaded and built when building the unit-tests target. See the Developer Notes for more information.