sylabs/singularity: SingularityCE 3.11.0 Release Candidate 2

This is the second release candidate for the upcoming Singularity 3.11.0 release. Users are encouraged to test and report any issues, but should use the stable 3.10 release for production deployments.

3.11.0 Release Candidate 2 [2023-02-02] Bug Fixes

• Avoid UID / GID readonly var warnings with --env-file.
• Ensure proot flow does not override --remote build.

3.11.0 Release Candidate 1 [2023-01-11] Changed defaults / behaviours

• Image driver plugins, implementing the RegisterImageDriver callback, are deprecated and will be removed in 4.0. Support for the example plugin, permitting Ubuntu unprivileged overlay functionality, has been replaced with direct support for kernel unprivileged overlay.
• When the kernel supports unprivileged overlay mounts in a user namespace, the container will be constructed using an overlay instead of underlay layout.
• crun will be used as the low-level OCI runtime, when available, rather than runc. If crun is not available, runc will be used.
• sessiondir maxsize in singularity.conf now defaults to 64 MiB for new installations. This is an increase from 16 MiB in prior versions.
• Instances are started in a cgroup, by default, when run as root or when unified cgroups v2 with systemd as manager is configured. This allows singularity instance stats to be supported by default when possible.

New features / functionalities Image Building

• Support for a custom hashbang in the %test section of a Singularity recipe (akin to the runscript and start sections).
• Non-root users can now build from a definition file, on systems that do not support --fakeroot. This requires the statically built proot command (https://proot-me.github.io/) to be available on the user PATH. These builds:
  • Do not support arch / debootstrap / yum / zypper bootstraps. Use localimage, library, oras, or one of the docker/oci sources.
  • Do not support %pre and %setup sections.
  • Run the %post sections of a build in the container as an emulated root user.
  • Run the %test section of a build as the non-root user, like singularity test.
  • Are subject to any restrictions imposed in singularity.conf. Incur a performance penalty due to proot's ptrace based interception of syscalls.
  • May fail if the %post script requires privileged operations that proot cannot emulate.

Instances

• Instances started by a non-root user can use --apply-cgroups to apply resource limits. Requires cgroups v2, and delegation configured via systemd.
• A new instance stats command displays basic resource usage statistics for a specified instance, running within a cgroup.
• Instance name is available inside an instance via the new SINGULARITY_INSTANCE environment variable.

Mounts & Overlays

• --writable-tmpfs is now available when running unprivileged, or explicitly requesting a user namespace, on systems with a kernel that supports unprivileged overlay mounts in a user namespace.
• The --no-mount flag now accepts the value bind-paths to disable mounting of all bind path entries in singularity.conf.
• Persistent overlays (--overlay) from a directory are now available when running unprivileged, or explicitly requesting a user namespace, on systems with a kernel that supports unprivileged overlay mounts in a user namespace.
• Add --sparse flag to overlay create command to allow generation of a sparse ext3 overlay image.

OCI / Docker Compatibility

• Support for DOCKER_HOST parsing when using docker-daemon://
• DOCKER_USERNAME and DOCKER_PASSWORD supported without SINGULARITY_ prefix.
• A new --oci flag for run/exec/shell enables the experimental OCI runtime mode. This mode:
  • Runs OCI container images from an OCI bundle, using runc or crun.
  • Supports docker://, docker-archive:, docker-daemon:, oci:, oci-archive: image sources.
  • Does not support running Singularity SIF, SquashFS, or EXT3 images.
  • Provides an environment similar to Singularity's native runtime, running with --compat.
  • Supports the following options / flags. Other options are not yet supported:
    • --fakeroot for effective root in the container. Requires subuid/subgid mappings.
    • Bind mounts via --bind or --mount. No image mounts.
    • Additional namespaces requests with --net, --uts, --user.
    • Container environment variables via --env, --env-file, and SINGULARITYENV_ host env vars.
    • --rocm to bind ROCm GPU libraries and devices into the container.
    • --nv to bind Nvidia driver / basic CUDA libraries and devices into the container.
    • --apply-cgroups, and the --cpu*, --blkio*, --memory*, --pids-limit flags to apply resource limits.

Signing & Verification

• The sign command now supports signing with non-PGP key material by specifying the path to a private key via the --key flag.
• The verify command now supports verification with non-PGP key material by specifying the path to a public key via the --key flag.
• The verify command now supports verification with X.509 certificates by specifying the path to a certificate via the --certificate flag. By default, the system root certificate pool is used as trust anchors unless overridden via the --certificate-roots flag. A pool of intermediate certificates that are not trust anchors, but can be used to form a certificate chain can also be specified via the --certificate-intermediates flag.
• Support for online verification checks of x509 certificates using OCSP protocol. (introduced flag: verify --ocsp-verify)

Other

• Add new Linux capabilities: CAP_PERFMON, CAP_BPF, CAP_CHECKPOINT_RESTORE.
• A new --reproducible flag for ./mconfig will configure Singularity so that its binaries do not contain non-reproducible paths. This disables plugin functionality.

Bug Fixes

• In --rocm mode, the whole of /dev/dri is now bound into the container when --contain is in use. This makes /dev/dri/render devices available, required for later ROCm versions.
• Overlay is blocked on the panfs filesystem, allowing sandbox directories to be run from panfs without error.

Development / Testing

• Significant reduction in the use of network image sources in the e2e tests.
• Improved parallelization and use of image caches in the e2e tests.
• The e2e-test makefile target now accepts an argument E2E_GROUPS to only run specified groups of end to end tests. E.g. make -C builddir e2e-test E2E_GROUPS=VERSION,HELP will run end to end tests in the VERSION and HELP groups only.
• The e2e-test makefile target now accepts an argument E2E_TESTS which is a regular expression specifying the names of (top level) end to end tests that should be run. E.g. make -C builddir e2e-test E2E_TESTS=^semantic will only run end to end tests with a name that begins with semantic. These E2E_ variables offer an alternative to the -run flag, which may be easier to use given the structure of e2e tests.

Thanks / Reporting Bugs

Thanks to our contributors for code, feedback and, testing efforts!

As always, please report any bugs to: https://github.com/sylabs/singularity/issues/new

If you think that you've discovered a security vulnerability please report it to: security@sylabs.io

Have fun!

Downloads

Source Code

Please use the singularity-ce-3.11.0-rc.2.tar.gz download below to obtain and install SingularityCE 3.11.0-rc.2. The GitHub auto-generated 'Source Code' downloads do not include required dependencies etc.

Packages

RPM / DEB packages are provided for:

• Ubuntu 18.04 (bionic)
• Ubuntu 20.04 (focal)
• Ubuntu 22.04 (jammy)
• RHEL/CentOS 7 (el7)
• RHEL/CentOS/AlmaLinux/Rocky 8 (el8)
• RHEL/CentOS/AlmaLinux/Rocky 9 (el9)

These packages were built with Go 1.19.3