ExplainTS: A Benchmark Suite for Reproducible Time-Series XAI Research

Description

Technical info (English)

Model Training

The time series classification model was trained using a deep neural architecture based on stacked ConvLSTM1D layers. Input data consisted of time series samples of shape ($T$, $F$), where $T$ denotes the number of timesteps and $F$ the number of features. The same model architecture was used for both univariate and multivariate inputs; in the univariate case, the number of features is $F = 1$. Prior to training, the data was normalized to zero mean and unit variance using StandardScaler.

To facilitate temporal feature learning, the input sequences were divided into smaller temporal blocks. Specifically, each sequence of length $T$ was segmented into n_steps parts, where n_steps corresponds to the third smallest integer divisor of $T$ above 2. The segment length was then computed as n_length = $T$ / n_steps, resulting in a reshaped input tensor of shape (n_steps, n_length, $F$). This restructuring enables the model to capture both local patterns within segments and long-range dependencies across segments.

The model architecture includes two ConvLSTM1D layers with 64 and 32 filters, respectively, each using a kernel size of 9 and ReLU activation. A dropout layer with a rate of 0.5 follows for regularization, and the output is flattened to create an intermediate embedding representation of length $N$. This is followed by two fully connected layers: one with 100 ReLU units and another with softmax activation for classification. Class imbalance was addressed using computed class weights during training. The model was trained using the Adam optimizer and categorical cross-entropy loss for 25 epochs with a batch size of 64.

The generic architecture of the models across all datasets is presented below. The exact number of trainable parameters depends on the dataset-specific input shapes ($T$, $F$, and $C$ classes).

Layer (type)                                     Output Shape              
=============================================================
reshape (Reshape)                            (None, n_steps, n_length, F)          
                                                                 
conv_lstm1d (ConvLSTM1D)             (None, n_steps, n_length, 64)          
                                                                 
conv_lstm1d_1 (ConvLSTM1D)          (None, n_steps, n_length, 32)            
                                                                 
dropout (Dropout)                              (None, n_steps, n_length, 32)            
                                                                 
embedding (Flatten)                          (None, N)                              
                                                                 
dense (Dense)                                    (None, 100)                              
                                                                 
dense_1 (Dense)                                 (None, C)                         
=============================================================

The dataset was split into training and testing sets using a predefined stratified partitioning strategy: 75% of the samples were used for model training, while the remaining 25% were held out for evaluation, preserving the class distribution across both sets. Labels were one-hot encoded to support the categorical cross-entropy loss.

Post-hoc Local Explanations

The explanations were computed for both training and test subsets. All indices in the explanation files are aligned with the respective train/test instances.

Explanation coverage:
- Univariate: Anchor (31.33%), LIME (100.00%), SHAP (100.00%), PHAR (100.00%)
- Multivariate: Anchor (20.00%), LIME (95.00%), SHAP (100.00%), PHAR (100.00%)

Rule-based explanations:

Anchor:

Each explanation is a list of rule sets grouped per instance, e.g.:

  [
    [
      {
        'index': 0,
        'success': True,
        'prediction': '1',
        'rule': {
          'feature_1': ['>-0.74'],
          'feature_4': ['<=0.73'],
          'feature_11': ['>-0.74'],
          'feature_42': ['<=0.73'],
          'feature_110': ['>-0.74']
        },
        'confidence': 0.9565,
        'coverage': 0.7197
      }
    ]
  ]

Each entry corresponds to a sample index. The `rule` defines a conjunction of feature constraints satisfied by the sample. `confidence` measures the fraction of samples fulfilling the rule for which the model gives the same prediction, while `coverage` is the fraction of the train/test set satisfying the rule.

PHAR rules follow a similar pattern, here for multivariate series:

[
  [
    {
      'index': 0,
      'success': True,
      'prediction': '2',
      'rule': {
        'var_0_ts_24': ['>-1.05', '<=0.45'],
        'var_0_ts_25': ['>-0.98', '<=0.51'],
        'var_0_ts_26': ['>-1.12', '<=0.33'],
        'var_2_ts_142': ['>0.15', '<=1.89'],
        'var_2_ts_143': ['>0.10', '<=1.95'],
        'var_5_ts_210': ['>-0.55', '<=-0.42'],
        ...
      },
      'confidence': 0.9812,
      'coverage': 0.8450
    }
  ]
]

In addition to the extracted rules, the PHAR archive includes two metadata files generated during its two-stage hyperparameter optimization (HPO) process via the Optuna framework:

- phar_metadata.json: Captures the single optimal configuration applied globally to extract the final rules. It details the winning parameters (such as the base explainer, threshold percentile, and perturb sigma) and provides aggregated statistics for confidence, coverage, and sparsity.
- phar_trials_log.jsonl: A JSON Lines file recording the complete history of all multi-objective Pareto-front optimization trials. Each line details a single trial's tested configuration, execution time, evaluated metrics, and the intermediate rules generated for the evaluation pool.

Attribution-based explanations:

SHAP and LIME:
Unlike the discrete rule records, SHAP and LIME provide continuous feature attributions. The explanations are stored as serialized NumPy arrays (.pickle files) containing float values.

Format: numpy.array of shape (M, C, T, F)
Where:
- M is the number of observations (instances) in the respective train or test split.
- C is the number of target classes.
- T is the length of the time series (number of timesteps).
- F is the dimensionality (number of features or channels; F=1 for univariate series).

Each numerical value in this 4D array represents the local attribution score (importance weight) for a specific class, timestep, and channel of a given instance. A positive weight indicates a feature that contributed towards the model's predicted probability for that specific class, while a negative weight indicates a feature that pushed the prediction away from it.

Accompanying GitHub repository

The complete open-source codebase used to construct the ExplainTS benchmark, train the underlying deep learning models, and compute the post-hoc explanations is publicly hosted on GitHub. 

Repository link: https://github.com/mozo64/papers/tree/main/zenodo-ucr. 

Code used for generation:
The repository is structured to ensure full reproducibility and is organized into the following key directories:

notebooks/
Contains the core Jupyter notebooks executing the entire pipeline:
- UCR-train.ipynb: Handles data loading, preprocessing, and training of the unified ConvLSTM-based classifiers.
- UCR-explainers-lime-shap.ipynb: Executes the extraction of continuous feature attributions using DeepSHAP and LIME.
- UCR-explainers-anchor.ipynb: Manages the Anchor explainer pipeline, including the multi-stage cascade retry mechanism for continuous signals.
- UCR-explainers-phar.ipynb: Runs the PHAR rule extraction, including the Optuna-based multi-objective Pareto-front hyperparameter optimization.
- datasets_summary.ipynb: Evaluates model accuracies and aggregates dataset statistics.
- ExplainTS_CaseStudy.ipynb: An educational template demonstrating how to load the precomputed artifacts, visualize them, and calculate XAI metrics (like the Jaccard Index).

services/
Contains helper Python utilities (model_manager.py, model_server.py) used to parallelize and distribute computationally heavy extraction jobs (specifically for Anchor) across GPUs.

scripts/
Includes bash shell utilities used to validate completeness and assemble the final Zenodo archives (e.g., compress_models.sh, compress_explainers.sh, filter_move.sh, and compress_all.sh).

List of all Time series

The following tables summarize the properties of the 20 multivariate and 83 univariate time-series datasets included in the ExplainTS benchmark. For each dataset, the tables report the number of instances in the standardized 75/25 train/test splits, sequence length, dimensionality, and the number of target classes.

Furthermore, they detail the test-set accuracy of the provided baseline ConvLSTM classifiers, along with the availability of precomputed local explanations across the four evaluated methods (LIME, DeepSHAP, Anchor, and PHAR). A "Yes" indicates that the respective explanations were successfully generated and are available in the repository archives for both the training and test sets.

Multivariate:

Univariate:

Notes (English)