Spack¶

Testing locally with Spack¶

The Spack package manager is written in Python, therefore it can be used interactively from a Python shell:

> spack python

Concretize and inspect a spec¶

A Spack spec can be concretized and inspected from the Python shell:

from spack.spec import Spec
from spack.concretize import concretize_one

s = Spec("spfft ^[virtuals=fftw-api] nvpl-fft") # (1)!
sc = concretize_one(s) # (2)!

Define the Spack spec to concretize.
Concretize the spec.

One can then inspect the concretized spec, for example:

sc["fftw-api"].libs.ld_flags
sc["fftw-api"].headers

Conflicts¶

The following conflicts are equivalent:

conflicts("+rocm +cuda")
conflicts("+rocm", when="+cuda")
conflicts("+cuda", when="+rocm")

The first option is cleaner, shorter, and less asymmetric.

Ccache¶

To use Ccache (compiler cache) with Spack, one can set the following in ~/.spack/config.yaml (or Spack's default etc/spack/config.yaml):

config:
    ccache: true

! warning

Some packages with compiler wrappers (e.g., Kokkos) may not work correctly with Cache enabled.

Upstream Spack instances and repositories precedence in environments¶

When using an upstream Spack instance with

spack -C /path/to/upstream/config spack <command>

repositories defined in the enviornment have lower precedence than those defined in the upstream Spack instance.

To give precedence to the environment repositories, one can set:

spack:
  include: ["/path/to/upstream/config"]

to include the upstream configuration in the environment.

Upstream Spack instance with custom repositories

spack:
  specs:
  - py-metatrain
  repos:
    metatensor: ~/git/work/my-spack/spack_repo/metatensor/
  include: [ /user-environment/config ]
  view: flase
  concretizer:
    unify: true

$ spack -C /user-environment/config/ -e . repo list
[+] alps          v2.0    /user-environment/repos/spack_repo/alps
[+] builtin       v2.2    /user-environment/repos/spack_repo/builtin
[+] metatensor    v2.2    /users/rmeli/git/work/my-spack/spack_repo/metatensor

$ spack -e . repo list
[+] metatensor    v2.2    /users/rmeli/git/work/my-spack/spack_repo/metatensor
[+] alps          v2.0    /user-environment/repos/spack_repo/alps
[+] builtin       v2.2    /user-environment/repos/spack_repo/builtin

Patches¶

To generate a patch for a package, one can use the diff command.

By default, Spack applies patches with patch -p1, therefore patches should be modified so that file paths start with a/ and b/, followed by the source code root directory.

Upstream Spack instances¶

It is possible to point a Spack installation to another installation to use the already installed packages. See chaining Spack installations for more details.

Permanent upstream Spack instance¶

The upstream Spack instance can be added permanently in ~/.spack/config.yaml:

upstreams:
  spack-instance-1:
    install_tree: /path/to/other/spack/opt/spack
  spack-instance-2:
    install_tree: /path/to/another/spack/opt/spack

Temporary upstream Spack instance¶

The upstream Spack instance can be added temporarily by overriding the local configuration via enviornment variables

export SPACK_SYSTEM_CONFIG_PATH=/user_environment/config/

or using the -C option:

spack -C /user_environment/config/ <command>

Isolation¶

If we install a package in our main Spack install tree, and it depends on something from an upstream Spack instance, we are implicitly creating a link between our Spack install tree and a the upstream Spack instance.

Isolation can be obtained by temporarily changing the install tree:

spack -c 'config:install_tree:root:/temporary/install/tree' <command>

Using custom Spack packages with upstream Spack instances¶

We cannot add our personal Spack builtin repo as first repository otherwise we override all the packages from the upstream Spack instance.

We cannot have it as last otherwise it gets completely overrideen by upstream builtin (unless the target package does not exists at all).

Therefore, we need to add a temporary Spack repository within a Spack environment with the packages we want to override. In this case, the upstream Spack instance builtin repo needs to be added within the environment as follows:

spack -e . config add 'include:[/path/to/config/]'

Using spack -C /path/to/config/ will not work because it will have precedence over the environment configuration.

Spack views for development tools¶

Spack environment views are a way to create a single directory that contains all the dependencies of a package. This is useful for easily installing and accessing development tools (such as editors, languages, etc.).

spack:
  specs:
    - <SPEC1>
    - <SPEC2>
  concretizer:
    unify: true
  view:
    default:
      root: <VIEW_PATH>

Development tools

spack:
  specs:
    - libtree
    - tmux
    - direnv
    - htop +hwloc
    - neovim
    - llvm
    - ripgrep
    - fzf
    - bat
    - ruby
    - go
    - rust +dev ~docs
    - node-js
    - npm
    - ccache
    - python
    - py-uv
  concretizer:
    unify: true
  packages:
    python:
      require:
        - "@3.11"
    llvm:
      require: 
        - "targets=x86" # Select one target to speedup compilation, doen't matter which one
        - "@18"
        - "~gold"
        - "~libomptarget"
  view:
    default:
  root: ~/aarch64-dev.view
      link: roots

Views and the number of symlinks

Views can create a large number of symlinks, which can cause issues with some file systems.

Count the number of symlinks

find <PATH> -type l | wc -l

To avoid creating too many symlinks, one can use link: roots:

view:
  default:
    root: ~/aarch64-dev.view
    link: roots

Use Spack to install dependencies and run CMake¶

The following script can be used to install dependencies and run CMake for a given develop spec:

#!/bin/bash

# The script needs to be sourced to work correctly
# Use return instead of exit to return to the shell

help() {
    echo "Usage: source spackdev.sh <spack-env> <spack-spec>"
    echo "  spack-env: Spack environment (path)"
    echo "  spack-spec: Spack spec"
}

error_dev(){
    echo "ERROR┌ ${1} is not a 'develop' spec."
    echo "ERROR└  Make sure you are using the correct Spack environment and spec."
    return 1
}

# Warn if build_stage is not set in the Spack environment
warn_bs() {
    echo "WARN┌ You are using a standard 'build_stage' directory."
    echo "WARN| Consider adding the following to your Spack environment:"
    echo "WARN|"
    echo "WARN|     config:"
    echo "WARN└   build_stage: <path-to-git-repo>/spack-build-stage/"
}

if [[ $# -ne 2 ]]; then
    help
    return
fi

SPACK_ENV=$1
SPACK_SPEC=$2

if command -v ccache 2>&1 > /dev/null; then
   export CMAKE_CXX_COMPILER_LAUNCHER=ccache
   export CMAKE_C_COMPILER_LAUNCHER=ccache
   export CMAKE_Fortran_COMPILER_LAUNCHER=ccache
   export CMAKE_CUDA_COMPILER_LAUNCHER=ccache
   export CMAKE_HIP_COMPILER_LAUNCHER=ccache
fi

# WARN: This only work if $SPACK_SPEC is just after the 'develop' key
grep -A 2 "develop:" "${SPACK_ENV}/spack.yaml" | grep -q "${SPACK_SPEC}:" || error_dev "${SPACK_SPEC}" || return
grep -q "build_stage:" "${SPACK_ENV}/spack.yaml" || warn_bs

spack -e "${SPACK_ENV}" clean || return # (1)!
spack -e "${SPACK_ENV}" concretize -f || return # (2)!
spack -e "${SPACK_ENV}" install --until=cmake --test=root --keep-stage || return # (3)!

SPACK_SOURCE_DIR=$(spack -e "${SPACK_ENV}" location --source-dir "${SPACK_SPEC}")
SPACK_BUILD_DIR=$(spack -e "${SPACK_ENV}" location --build-dir "${SPACK_SPEC}")

SPACK_ENV_NAME=$(basename "${SPACK_ENV}")
ENVRC_TMP="/tmp/.envrc-${SPACK_ENV_NAME}-${SPACK_SPEC}"
spack -e "${SPACK_ENV}" build-env --dump "${ENVRC_TMP}" "${SPACK_SPEC}" || return # (4)!
echo "SPACK_BUILD_DIR=\"${SPACK_BUILD_DIR}\"; export SPACK_BUILD_DIR" >> "${ENVRC_TMP}"
echo "SPACK_SOURCE_DIR=\"${SPACK_SOURCE_DIR}\"; export SPACK_SOURCE_DIR" >> "${ENVRC_TMP}"

cp "${ENVRC_TMP}" "${SPACK_BUILD_DIR}/.envrc"
direnv allow "${SPACK_BUILD_DIR}"

# Add .envrc in SPACK_SOURCE_DIR so that nvim integrated terminal is correctly set up
 mv "${ENVRC_TMP}" "${SPACK_SOURCE_DIR}/.envrc"
 direnv allow "${SPACK_SOURCE_DIR}"

# Be friendly to LSPs
mkdir -p "${SPACK_SOURCE_DIR}/build"
ln -sf "${SPACK_BUILD_DIR}/compile_commands.json" "${SPACK_SOURCE_DIR}/build/compile_commands.json" # (5)!

# Remove broken symlinks (from previous builds)
find "${SPACK_SOURCE_DIR}" -xtype l -delete

pushd "${SPACK_SOURCE_DIR}" || return

Clean the Spack environment.
Concretize the environment.
Install the dependencies and run CMake.
Dump the build environment to an .envrc file. This is used by direnv to set up the environment.
Create a symlink to the compile_commands.json file in the source directory. This is useful for LSPs.

Note

The script is meant to be sourced (source ...), not executed.