Evaluating the Effect of Numerical Solvers on Learning in Neural ODEs

# Evaluating the Effect of Numerical Solvers on Learning in Neural ODEs

**Supplementary code and reproduction guide**  

**Paper:** *Transactions on Machine Learning Research* (TMLR), October 2026  

**Authors:** Gavin Lee Goodship, Stephen O'Sullivan, and Luis Miralles-Pechuán  

**Paper and review record:** https://openreview.net/forum?id=Mopa70gMnp

## Overview

This archive accompanies the TMLR paper *Evaluating the Effect of Numerical Solvers on Learning in Neural ODEs*. It provides PyTorch experiment scripts and supporting documentation for investigating how fixed-step Runge–Kutta methods influence the coarse numerical maps learned by Neural ODE classifiers.

The paper asks whether a Runge–Kutta method's **external scalar stability polynomial** is sufficient to characterize solver-dependent learning. Two controlled comparisons address this question: varying the Gegenbauer parameter while holding formal order and computational conditions fixed, and comparing ESRKG-15 with LSRKG-15, which have numerically matched external scalar stability polynomials but different internal tableaux and higher-order nonlinear B-series coefficients. The results support the conclusion that external stability structure matters but **does not determine the complete nonlinear Runge–Kutta update**.

Further experiments use frozen-weight solver replacement and common-reference numerical refinement. A checkpoint may classify well under its training-time coarse discretization yet perform very differently when its learned vector field is numerically resolved. For example, on the fixed 1,000-image CIFAR-10 refinement subset, the paper reports an ESRKG-15 checkpoint changing from **85.4% native accuracy to 12.1% resolved accuracy**. This is a representative checkpoint-level result, not an assertion that all trained models show the same change.

The third-order augmented Neural ODE exposes velocity, acceleration and third-order forcing for descriptive numerical diagnostics. Those diagnostics are **not** proposed as intrinsic geometric invariants, exact spectral quantities or stability certificates. First-order Neural ODE controls are also included.

## Contents

The original supplementary ZIP has the following layout. Folder and filenames below follow the archive exactly.

| Location | Contents | Paper connection |

| --- | --- | --- |

| `CIFAR 10/` | RK4, ESRKG-15, ESRKG-21 and LSRKG-15 training/evaluation scripts; experiment PDF | Section 4.3, CIFAR-10 |

| `CIFAR100/` | Corresponding CIFAR-100 solver scripts and PDF | Section 4.3, CIFAR-100 |

| `Tinyimagnet/` | Tiny-ImageNet experiments and dataset-path instructions | Section 4.3, Tiny-ImageNet |

| `Gegenbauer/` | ESRKG-15 parameter variants and `visual.py` stability-region plot | Section 4.4.1 |

| `Realization/` | `real.py` for external stability-polynomial and internal-stage amplification comparisons | Section 4.4.2 |

| `Swap/` | Frozen-weight solver replacement at integration horizons 10 and 20 | Section 4.5 |

| `Refinement/` | Common-reference solver-refinement experiments for CIFAR-10 and CIFAR-100 | Section 4.6 |

| `Appendices/` | First-order solver-swap and numerical-metric controls | Appendix experiments |

| `Documentation_for_code.pdf` | Overall supplementary documentation | Archive guide |

Each experiment directory also contains a PDF with more detailed instructions. **Datasets, pretrained weights/checkpoints and generated experiment results are not bundled in this ZIP**; the supplied material consists of code and documentation. Training or access to separately generated checkpoints is therefore necessary for checkpoint-based evaluation.

## Software and data

- **Language/framework:** Python 3, PyTorch and torchvision. A Python version supporting `argparse.BooleanOptionalAction` (Python 3.9+) is required for the newer scripts.

- **Other packages used across the archive:** NumPy, matplotlib, pandas, `ptflops` and `torchattacks`.

- **Optional adversarial evaluation:** Some scripts also use the official AutoAttack implementation (`autoattack`). This dependency is needed for those evaluations but can be avoided in the newer training/refinement scripts with `--skip_adversarial`.

- **Compute:** A CUDA-capable GPU is recommended for the full image-classification experiments. Runtime and floating-point results depend on the machine and software stack.

- **Data:** CIFAR-10 and CIFAR-100 are loaded via torchvision and can be downloaded by the scripts. Tiny-ImageNet must be supplied separately; pass a directory containing `train/` and `val/` organized into class subdirectories (including the validation classes) as required by `torchvision.datasets.ImageFolder`.

An example environment setup is:

```bash

python -m venv .venv

# Activate .venv using your operating system's normal command.

python -m pip install --upgrade pip

python -m pip install torch torchvision numpy matplotlib pandas ptflops torchattacks

```

Install the official AutoAttack package separately if running the official AutoAttack evaluation. Exact package versions were not pinned in the supplied archive; record the versions used when reproducing the experiments.

All commands below assume the working directory is the extracted **`Supplementary material-revised/`** directory. Paths containing spaces should be quoted.

## Reproducing the principal experiments

### 1. CIFAR-10 and CIFAR-100: solver comparisons (Section 4.3)

The four principal configurations integrate the same total horizon, using different step decompositions:

| Solver | Stages per step | Steps | Step size at horizon `Delta T = 10` | Function evaluations |

| --- | ---: | ---: | ---: | ---: |

| RK4 | 4 | 4 | 2.5 | 16 |

| ESRKG-15 | 15 | 1 | 10 | 15 |

| ESRKG-21 | 21 | 1 | 10 | 21 |

| LSRKG-15 | 15 | 1 | 10 | 15 |

Example CIFAR-10 runs:

```bash

python "CIFAR 10/rkcifar10.py" --dataset cifar10 --h 2.5 --steps 4 --skip_adversarial

python "CIFAR 10/esrk15cifar10.py.txt" --dataset cifar10 --h 10 --steps 1 --skip_adversarial

python "CIFAR 10/esrk21cifar10.py" --dataset cifar10 --h 10 --steps 1 --skip_adversarial

python "CIFAR 10/lsrkcifar10.py" --dataset cifar10 --h 10 --steps 1 --skip_adversarial

```

The ESRKG-15 CIFAR-10 script has the literal filename `esrk15cifar10.py.txt`; Python can execute it directly as shown, or it can be renamed locally to `.py`.

For CIFAR-100, use the corresponding filenames under `CIFAR100/` **and explicitly pass `--dataset cifar100`**. For example:

```bash

python CIFAR100/esrk15cifar100.py --dataset cifar100 --h 10 --steps 1 --skip_adversarial

```

At `Delta T = 20`, set `--h 20 --steps 1` for ESRKG/LSRKG methods and `--h 5 --steps 4` for RK4. At `Delta T = 30`, use `--h 30 --steps 1` or `--h 7.5 --steps 4`, respectively. The paper reports three independent seeds per configuration; rerun with appropriately recorded seeds, software versions and hardware details when making a quantitative comparison. In the newer training scripts, `--attack_seed` also initializes the training RNG.

Use `--save_checkpoint PATH` in the newer training scripts if a weight file is needed for subsequent analysis.

### 2. Tiny-ImageNet (Section 4.3)

Tiny-ImageNet experiments use `Delta T = 20`. Replace `/path/to/tiny-imagenet` with your local dataset root:

```bash

python Tinyimagnet/esrk15tiny.py --dataset tinyimagenet --data /path/to/tiny-imagenet --h 20 --steps 1 --skip_adversarial

python Tinyimagnet/rktiny.py --dataset tinyimagenet --data /path/to/tiny-imagenet --h 5 --steps 4 --skip_adversarial

```

The other solver scripts are `Tinyimagnet/esrk21tiny.py` and `Tinyimagnet/lsrktiny.py`. See `Tinyimagnet/TINYImagenet (1).pdf` for details.

### 3. Gegenbauer-parameter sweep (Section 4.4.1)

The folder contains four ESRKG-15 configurations with different Gegenbauer parameters: `esrk15base.py` (nu = 1/4), `esrkhalf.py` (1/2), `esrktenth.py` (1/10), and `esrkv2.py` (2). The paper compares their sampled numerical regimes with order, stages, function evaluations and training protocol controlled.

```bash

python Gegenbauer/esrk15base.py --dataset cifar100 --data ./data

python Gegenbauer/esrkhalf.py --dataset cifar100 --data ./data

python Gegenbauer/esrktenth.py --dataset cifar100 --data ./data

python Gegenbauer/esrkv2.py --dataset cifar100 --data ./data

python Gegenbauer/visual.py

```

The default run configuration in these scripts uses horizon 20 (`--h 20 --steps 1`). The visualization and numerical comparison are described in `Gegenbauer/CIFAR100gan (1).pdf`.

### 4. Matched external polynomials, distinct tableaux (Section 4.4.2)

```bash

python Realization/real.py

```

This script examines ESRKG-15 and LSRKG-15 external stability-function agreement and tableau-dependent internal-stage amplification. They are **distinct Runge–Kutta methods**, not mathematically identical implementations of the same nonlinear update. See `Realization/REAL.pdf`.

### 5. Frozen-weight solver swaps (Section 4.5)

The `Swap/` scripts train using their designated solver, freeze the trained weights and evaluate with other solver choices. Examples for CIFAR-100:

```bash

python Swap/thirdrkswap.py --dataset cifar100 --data ./data

python Swap/thirdesrk15swap.py --dataset cifar100 --data ./data

python Swap/thirdesrk21twenty.py --dataset cifar100 --data ./data

```

Files ending in `swap.py` cover horizon 10; files ending in `twenty.py` cover horizon 20. Each performs a solver-swap comparison for its training solver. **Run these in separate working/output directories if retaining checkpoints**, because the scripts write a generically named `model_finals.pth` file. See `Swap/CIFAR100swap (1).pdf`.

### 6. Common-reference numerical refinement (Section 4.6)

The `Refinement/` scripts run frozen-checkpoint evaluations with RK4, ESRKG-15, ESRKG-21 and LSRKG-15 at progressively finer resolutions. The reported paper study uses steps `1,2,4,8,16,32,64,128`, with an RK4 256-step reference checked against RK4 128 and a fixed 1,000-example subset.

A script can train a model and then run refinement, or load a compatible saved checkpoint with `--checkpoint` and skip training with `--epochs 0`. For example:

```bash

python Refinement/esrk15refinecifar10.py \

  --dataset cifar10 --h 10 --steps 1 \

  --checkpoint /path/to/compatible_checkpoint.pt --epochs 0 \

  --skip_adversarial --refinement_output_dir ./results/refinement_esrk15

```

Omit `--checkpoint` and `--epochs 0` to train a fresh checkpoint before refinement. The script outputs refinement tables and summaries; see `Refinement/REFINEMENT_Study (1).pdf`. Other CIFAR-10 and CIFAR-100 refinement scripts are provided for the remaining training solvers.

### 7. First-order Neural ODE controls (Appendix)

```bash

python Appendices/first_order_solver_swap_rooted_trees.py \

  --dataset cifar10 --train_solvers rk4,esrk,lsrk \

  --eval_solvers rk4,esrk,lsrk --epochs 50 \

  --max_tree_order 4 --tree_batches 3 --tree_samples 4

python Appendices/solver_metric_benchmark..py \

  --epochs 2 --train_subset 3000 --test_subset 1000 --attack_batches 1

```

The first script records first-order solver swaps and rooted-tree/B-series quantities. The second provides a shorter first-order metric benchmark. Refer to `Appendices/Appendices.pdf`.

## Interpretation and reproducibility notes

- **Native vs. resolved accuracy:** Report both the original coarse-solver classification accuracy and the accuracy reached under numerical refinement. Convergence of different evaluators for one frozen checkpoint does not imply that independently trained checkpoints define the same continuous flow.

- **Scope:** The paper studies four fixed-step explicit Runge–Kutta configurations on the stated image benchmarks; it does not establish the same outcomes for every adaptive or implicit solver.

- **Diagnostics:** Third-order forcing, velocity-normalized response and finite-iteration JVP amplification are descriptive measurements, not general stability or robustness guarantees.

- **Reproduction environment:** Different GPU hardware, CUDA/PyTorch versions, augmentation randomness, and seeds can lead to different individual results. Preserve run logs, configuration, checkpoints and package versions for reproducibility.

- **Archive naming:** The actual filenames are `CIFAR 10/esrk15cifar10.py.txt`, `Gegenbauer/esrk15base.py`, `Refinement/lsrkgrefinecifar100.py.py`, and `Appendices/solver_metric_benchmark..py`. Some accompanying PDFs use different spellings. Use the filenames listed in the ZIP rather than those outdated references.

- **Data and weights:** The deposited code ZIP does not contain raw datasets or pretrained checkpoints. See each experiment PDF and inspect the script's `--help` output for additional options. Not every legacy experiment script implements every flag of the newer scripts.

## Citation

If using this supplementary material, please cite the associated paper:

```bibtex

@article{goodship2026evaluating,

  title   = {Evaluating the Effect of Numerical Solvers on Learning in Neural ODEs},

  author  = {Goodship, Gavin Lee and O'Sullivan, Stephen and Miralles-Pechuán, Luis},

  journal = {Transactions on Machine Learning Research},

  year    = {2026},

  url     = {https://openreview.net/forum?id=Mopa70gMnp}

}

```