v0.11.4

Build-environment fix release. Every GPU job that used a JAX engine — the multinomial, bivariate/continuous, and CNN modeling runners — was failing on the cluster, and the failure was not in the modeling code at all: torch's PyPI wheels had silently migrated to CUDA 13, which collides with the CUDA-12 wheels jax[cuda12] requires. This release pins the whole GPU stack to CUDA 12, fixes a second (independent) bug where SLURM never actually bound a GPU to array-job tasks, and floors three Dependabot advisories. No behavioural changes to processing or analysis; no data-integrity fixes. v0.11.3 results are unaffected.

⚠️ Who is affected

• Anyone running GPU modeling jobs (main_univariate_dispatcher / main_model_selection_dispatcher with the JAX engines). These were failing outright.
• Anyone who re-syncs their environment. torch now resolves from PyTorch's CUDA-12.8 index rather than PyPI (see Upgrading).
• Anyone with a deployed cluster checkout. A git pull alone is not sufficient — see Upgrading.

Fixed

• The CUDA-13 / CUDA-12 cuDNN collision that broke every JAX GPU job. torch's PyPI wheels now default to a CUDA 13 build, dragging in nvidia-cudnn-cu13, nvidia-cublas 13, nvidia-cusparselt-cu13, nvidia-nccl-cu13 and cuda-toolkit 13. jax[cuda12] needs the CUDA-12 wheels. Both sets install libcudnn.so.9 into the same nvidia/cudnn/ namespace directory, so the cu13-built cuDNN wins the file and JAX's cu12 process cannot create a cuDNN handle. The job died at the very first GPU op (jnp.array) with:

  Could not create cudnn handle: CUDNN_STATUS_NOT_INITIALIZED
  RET_CHECK failure (gpu_compiler.cc:2641) dnn_support != nullptr
  

  torch and torchvision are now pinned to PyTorch's cu128 index (torch 2.11.0+cu128, torchvision 0.26.0+cu128), so the entire GPU stack stays on CUDA 12 and no *-cu13 wheel is installed at all.

  The pin is scoped to Linux (marker = "sys_platform == 'linux'"). There is no CUDA on macOS and the cu128 index ships no macOS wheel, so an unscoped pin makes the package uninstallable there; macOS and Windows fall back to PyPI, where no cu12/cu13 collision can occur. The CUDA-12 pin is only meaningful on Linux anyway — that is where jax[cuda12] and the GPU cluster live.

  Two notes on the diagnosis, because both cost real time:

  • XLA's accompanying Possibly insufficient driver version: 575.57.8 is a red herring — driver 575 is perfectly adequate for CUDA 12. The message is XLA's generic fallback whenever cuDNN init fails for any reason.
  • CUDA 13 is not a viable target here. The cluster is heterogeneous: some GPU nodes run driver 575 (CUDA 12.9), others 580 (CUDA 13.0). CUDA 12 runs on both; a CUDA-13 stack would fail whenever SLURM landed the job on a 575 node. CUDA 12 is the only safe common target.

• torchvision promoted to a direct dependency. It was transitive-only, so the [tool.uv.sources] cu128 pin did not apply to it and it kept resolving from PyPI — a build whose compiled ops do not match a cu128 torch. Anything importing it (sam2 box-prompt masks, vocalocator, QLVM training) failed with:

  RuntimeError: operator torchvision::nms does not exist
  

  This was invisible to the multinomial job (it never imports torchvision) and was only caught by running the full suite against the actual locked environment.

• SLURM never bound a GPU to array-job tasks (--gpus vs --gres). The modeling scripts requested #SBATCH --gpus=1. That accounts a GPU — it duly appears in AllocTRES as gres/gpu=1 — but it does not reliably bind the device into an array task's cgroup. CUDA_VISIBLE_DEVICES was left unset, and the JAX engines died with CUDA_ERROR_NO_DEVICE on a node that visibly had an idle GPU. Both scripts now use #SBATCH --gres=gpu:1, which actually binds it. The gotcha is documented inline so it does not get "fixed" back.

Security

Three Dependabot advisories closed by flooring two transitive dependencies ([tool.uv] constraint-dependencies):

| alert | package | was | now | issue | |---|---|---|---|---| | #54 | mistune | 3.2.1 | 3.3.3 | quadratic-time DoS in parse_link_text (CWE-400) | | #53 | soupsieve | 2.8 | 2.8.4 | unbounded memory on comma-separated selector lists | | #52 | soupsieve | 2.8 | 2.8.4 | ReDoS via catastrophic backtracking in the attribute-selector VALUE regex |

Both are docs-only transitive dependencies (mistune via nbsphinx/jupyterlab, soupsieve via furo/nbsphinx/jupyterlab), and usv-playpen never parses untrusted Markdown or user-supplied CSS selectors — so practical exposure is nil. They are floored anyway because uv's lock is sticky: regenerating uv.lock for an unrelated change (the torch pin, in this very release) leaves transitive deps untouched, so a patched version is only ever picked up if it is explicitly pinned.

Cluster scripts (housekeeping — no behavioural change)

• model_selection_behavior.sh and univariate_modeling_behavior.sh moved from other/cluster/modeling/ to other/cluster/usv_playpen/, alongside every other cluster job script. The now-empty modeling/ directory is gone.
• XLA log spam silenced. TF_CPP_MIN_LOG_LEVEL=3 + GLOG_minloglevel=2 suppress XLA's ERROR-level device_type: DEVICE_TYPE_INVALID Triton-GEMM messages, which otherwise bury the real output in .err. Verified on the cluster: cosmetic only — no change to the computation, the results, or the runtime.
• GPU provenance line. Every job now logs its node, CUDA_VISIBLE_DEVICES, and nvidia-smi -L. An empty CUDA_VISIBLE_DEVICES is the immediate tell for the binding bug above, and recording which GPU ran the job matters on a heterogeneous cluster (A100-40G vs L40S-46G). Informational only — the pyGAM analysis types (onset/params/category) are CPU-only and are never failed by it.

Upgrading

uv sync --extra gpu --group docs

On Linux, torch and torchvision now come from https://download.pytorch.org/whl/cu128 instead of PyPI: expect torch 2.11.0+cu128 and torchvision 0.26.0+cu128, and every nvidia-*-cu13 wheel to disappear. On macOS and Windows nothing changes — they continue to resolve from PyPI (CPU/MPS builds), since the CUDA pin is Linux-scoped.

On a deployed cluster checkout, git pull alone is not enough — the environment still holds the mismatched PyPI torchvision until you re-sync:

git pull
uv sync --extra gpu     # <- required; installs torchvision 0.26.0+cu128

Without it, any job importing torchvision (sam2, vocalocator, QLVM training) still hits torchvision::nms does not exist.

If you copy the modeling job scripts by path, note they now live under other/cluster/usv_playpen/, and adopt --gres=gpu:1 in any local copy you maintain.

Full suite green (2184 passed).

Full changelog: v0.11.3...v0.11.4