MLNX Job Placement Failure Dataset for Simulated Datacenter Clusters with Reconfigurable Optical Networks

📌 Overview

This dataset contains cluster-level snapshots and job placement outcomes generated using a simulated large-scale datacenter environment.
The data is intended for training and evaluating machine learning models that predict whether a job submission will succeed or fail given the current cluster state and job resource request.

The dataset was produced as part of the MLSysOps project (EU Horizon Europe) and supports research on:

• job admission control,
• failure prediction,
• resource fragmentation,
• and network feasibility in modern datacenter architectures.

Each data sample represents a single scheduling decision and includes both:

• detailed cluster state features, and
• the observed outcome of the placement attempt.

🏢 System Context

Simulated Datacenter Architecture

The dataset is generated using a proprietary datacenter simulator modeling a hierarchical cluster composed of Scalable Units (SUs).

Cluster configuration:

• 32 Scalable Units (SUs)
• 32 servers per SU (1024 servers total)
• 8 leaf switches per SU
• 8 GPUs per server
• Leaf switches interconnected via a reconfigurable optical circuit switch (OCS)

Failure Modes Captured

Each job placement attempt can result in:

1. Successful placement
2. Failure due to insufficient servers
3. Failure due to insufficient or infeasible uplink connectivity

While server insufficiency can be determined via simple capacity checks,
uplink infeasibility is more complex, as it depends on:

• current optical circuit configurations,
• contention between jobs,
• and connectivity constraints of the OCS fabric.

The dataset explicitly captures these outcomes to support learning-based approaches for failure prediction.

📂 Dataset Structure

• Format: Apache Parquet
• Granularity: One row per scheduling decision
• Each row contains:
  1. Job request features
  2. Cluster state features (scalar + vector)
  3. Ground-truth placement outcome label

Rows are treated as independent samples.

🏷️ Ground-Truth Labels

The dataset includes a label column encoding the observed outcome of the job placement:

Notes:

• Labels 1 and 2 both indicate job failure, but with different root causes.
• This encoding allows:
  • binary failure prediction,
  • failure cause analysis,
  • and future multi-class modeling.

📊 Feature Description

Scalar Cluster Features

These features summarize utilization, imbalance, and fragmentation across the cluster:

Vector Features

Job Request Feature

🧪 Data Collection Methodology

• Environment: Simulated datacenter
• Workloads: Synthetic job traces with varying sizes and arrival patterns
• Placement policy: Simulator-internal scheduling logic
• Labeling: Determined by placement outcome (success or failure cause)

The simulator executes job placement attempts under varying load, fragmentation, and network conditions to generate diverse training examples.

⚠️ The simulator itself is not publicly released. Only the resulting dataset is provided.

📊 Statistical Summary

The dataset contains a total of 1,062,943 rows, each corresponding to a single job placement attempt in the simulated cluster.

The table below summarizes the distribution of all numeric columns, including the ground-truth label.

Column Summary and Data Types

Ground-truth label distribution note:
The l1_failed column encodes job outcomes as:

• 0: success
• 1: failure due to insufficient servers
• 2: failure due to insufficient uplinks / infeasible connectivity

Both 1 and 2 correspond to job failures.

🧰 Working with the Data

Loading the Dataset (Python)

import pandas as pd

df = pd.read_parquet("final_merged.parquet")
print(df.head())

Loading Selected Columns

cols = ["f20_requested_nodes", "f2_mean_util", "l1_failed"]
df = pd.read_parquet("final_merged.parquet", columns=cols)

Tools and Documentation

Apache Parquet specification: https://parquet.apache.org/docs

Pandas Parquet I/O: https://pandas.pydata.org/docs/reference/api/pandas.read_parquet.html

PyArrow Parquet support: https://arrow.apache.org/docs/python/parquet.html

🎯 Intended Use

This dataset is intended for:

• machine learning research on job failure prediction,
• benchmarking admission-control models,
• studying resource fragmentation and network feasibility,
• offline evaluation of scheduling heuristics.

It is not intended to represent any specific production datacenter.

⚠️ Limitations

• Data is generated from a simulator, not a production system.
• The cluster topology is fixed and may not generalize to other architectures.
• Temporal dependencies between jobs are not explicitly modeled.
• Network behavior is abstracted and may differ from real optical fabrics.

📜 Citation

If you use this dataset in your research, please cite it using the citation provided by Zenodo (available in the right sidebar of the dataset record

🤝 Acknowledgements & Funding

This work is part of the MLSysOps project and is funded by the European Union’s Horizon Europe research and innovation programme under grant agreement No. 101092912.

More information: https://mlsysops.eu/