[Data] Prediction of soft proton intensities in the near-Earth space using machine learning

1  Introduction

The dataset consists of four files:

File Name                             Type

RAPID_OMNI_ML_023_raw.h5              dataset
RAPID_OMNI_ML_023_traincut.h5         dataset
RAPID_OMNI_ML_023_testcut.h5          dataset
RAPID_OMNI_ML_023_robusttranscut.pkl  pre-fit scaler

1.1 Variables

The dataset files contain the following variables.

Variable Name  Type      Unit              Description      
                       
p1, p2, p3,    float     1/(s⋅cm²⋅sr⋅keV)  proton intensities  
p4, p5, p6,
p7                  

x, y, z        float     RE                position in GSE coordinates

rdist          float     RE                radial distance from the Earth  
        
AE_index       float     nT                Auroral Electroject (AE) index  
        
SYM-H_index    float     nT                symmetric disturbance field in horiz. direction

F107           float     sfu               the solar radio flux at 10.7 cm
 
BimfxGSE,      float     nT                x, y and z components of the Interplanetary
BimfyGSE,                                  Magnetic Field in GSE coordinates
BimfzGSE

VxSW_GSE,      float     km/s              x, y and z components of the solar wind speed
VySW_GSE,
VzSW_GSE

NpSW           float     n/cc              solar wind density   
                   
Temp           float     K                 solar wind temperature
             
Pdyn           float     nPa               solar wind dynamic pressure

DateTime       datetime                    timestamp                               

1.2 Data Split Ranges

The datasets contain data from the following time ranges.

Dataset                        Start                End                  Count

RAPID_OMNI_ML_023_raw.h5       2001-01-09 15:21:00  2018-02-19 09:57:00  6,051,937
RAPID_OMNI_ML_023_traincut.h5  2001-01-09 15:21:00  2014-07-24 22:44:00  4,524,200
RAPID_OMNI_ML_023_testcut.h5   2014-07-24 22:45:00  2018-02-19 09:57:00  1,173,865

2  Raw Data Preparation

2.1 Data Source

OMNIWeb (NASA)

From NASA/GSFC's OMNI data set through OMNIWeb, we extracted the following variables from 2001 to 2019:

Variable     Original Name in OMNIWeb  Resolution

AE_index     AE Index, nT              1-min     
SYM-H_index  SYM/H, nT                 1-min     
F107         Solar index F10.7         1-hour *  
BimfxGSE     Bx, GSE/GSM, nT           1-min     
BimfyGSE     By, GSE, nT               1-min     
BimfzGSE     Bz, GSE, nT               1-min     
VxSW_GSE     Vx Velocity, GSE, km/s    1-min     
VySW_GSE     Vy Velocity, GSE, km/s    1-min     
VzSW_GSE     Vz Velocity, GSE, km/s    1-min     
NpSW         Proton Density, n/cc      1-min     
Temp         Proton Temperature, K     1-min     

* Solar index F10.7 is not available at higher resolution.

Cluster Science Archive (ESA)

Through the Cluster Archive Inter-Operability Subsystem, we have access to the proton intensities in 7 energy channels measured by RAPID instrument onboard Cluster satellite. We got the following variables between 2001 and 2009 from the CDF files:

Variables          Dataset ID in CAIO   Variable in CDF Files              Resolution

p1, p2, p3, p4,    C4_CP_RAP_HSPCT      Proton_Dif_flux__C4_CP_RAP_HSPCT   4127-ms
p5, p6, p7

x_km, y_km, z_km*  C4_CP_AUX_POSGSE_1M  sc_r_xyz_gse__C4_CP_AUX_POSGSE_1M  1-min

* Positions in km are not included in the final dataset.

2.2 Custom Features

OMNIWeb (NASA)

Variable  Source

Pdyn      NpSW * (VxSW_GSE^2 + VySW_GSE^2 + VzSW_GSE^2) * 1.67e-6

Cluster Science Archive (ESA)

Variable  Source

x         x_km / 6371.1
y         y_km / 6371.1
z         z_km / 6371.1
rdist     sqrt(x^2 + y^2 + z^2)

2.3 Sampling and Interpolation

- We use the value of F107 at a 1-hour resolution to represent all values in each 1-hour bin.

- For successful integration, we sampled the proton intensities to a resolution of 1 minute. More specifically, we calculate the averaged proton intensities for seven channels in each minute and use them to represent the values at first second in each minute, e.g., values at 2001/1/9 15:22:00 are calculated with the data from 15:22:00 - 15:22:59.

2.4 Integration

As now data from different sources can be aligned with Datetime at a resolution of 1 minute, we can merge them.

2.5 Cleaning

At last, we dropped the rows with outliers (fill values) in any OMNI variable. Please refer to the in the description from OMNIWeb for more information about the fill values.

The raw data generated is available in the package with the name RAPID_OMNI_ML_023_raw.h5.

3  Experiment-specific Pre-processing

Besides, we took the steps below to the dataset for our experiments.

3.1 Splitting

The dataset is split into a training set and a test set with a ratio of 8:2.

3.2 Filtering

- We filtered out the rows with rdist less than or equal to 6.

- We also use NaNs to substitute the proton intensities less than or equal to the threshold, which is 5, 1, 0.5, 0.1, 0.05, 0.005 or 0.001 respectively for 7 channels.

3.3 Transform

- We did not perform any transform or scaling directly on the data in pre-processing. Instead, a Robust Scaler fit with the training data was dumped as a file and used in the experiment.

The pre-processed data and scaler are available under names RAPID_OMNI_ML_023_traincut.h5, RAPID_OMNI_ML_023_testcut.h5 and RAPID_OMNI_ML_023_robusttranscut.pkl.