Deep ensemble for ENSO-forecasting

In this tutorial you learn how to use a neural network model called Deep Ensemble (DE) for the ENSO forecasting. This network architecture was initially developed Lakshminarayanan et al. (2017).

DEs are feed foreword neural networks that predict the mean and the standard deviation of a Gaussian. Hence, their predicion comes with an uncertainty estimation which is a valuable feature for ENSO-forecasting.

Create a data pipe line

At first, we define a data pipeline. This is in general quite useful to keep your code clean and also to reuse the pipeline for later purpose.

The data pipeline generates returns:

  1. The feature array

  2. The label array

  3. The time array corresponding to the time of the label

NOTE (again): Lead time is defined as the time that passed between the last observed and the first date of the target season. Hence, negative appear, e.g. if you compare the DJF season with the target season JFM, you have a lead time of -2 month (Last observed date: Feburary 28/29, First date of the target season January 1).

[1]:
import numpy as np
from sklearn.preprocessing import StandardScaler

from ninolearn.utils import include_time_lag
from ninolearn.IO.read_post import data_reader

def pipeline(lead_time):
    """
    Data pipeline for the processing of the data before the Deep Ensemble
    is trained.

    :type lead_time: int
    :param lead_time: The lead time in month.

    :returns: The feature "X" (at observation time), the label "y" (at lead
    time), the target season "timey" (least month)
    """
    reader = data_reader(startdate='1980-01', enddate='2018-12')

    # indeces
    oni = reader.read_csv('oni')
    iod = reader.read_csv('iod')
    wwv = reader.read_csv('wwv')

    # seasonal cycle
    sc = np.cos(np.arange(len(oni))/12*2*np.pi)

    # network metrics
    network_ssh = reader.read_statistic('network_metrics', variable='sst', dataset='ERSSTv5', processed="anom")
    c2 = network_ssh['fraction_clusters_size_2']
    H = network_ssh['corrected_hamming_distance']

    # time lag
    time_lag = 12

    # shift such that lead time corresponds to the definition of lead time
    shift = 3

    # process features
    feature_unscaled = np.stack((oni, sc, wwv, iod,
                                 c2, H), axis=1)

    # scale each feature
    scalerX = StandardScaler()
    Xorg = scalerX.fit_transform(feature_unscaled)

    # set nans to 0.
    Xorg = np.nan_to_num(Xorg)

    # arange the feature array
    X = Xorg[:-lead_time-shift,:]
    X = include_time_lag(X, max_lag=time_lag)

    # arange label
    yorg = oni.values
    y = yorg[lead_time + time_lag + shift:]

    # get the time axis of the label
    timey = oni.index[lead_time + time_lag + shift:]

    return X, y, timey
/home/paul/miniconda2/envs/ninolearn/lib/python3.6/site-packages/distributed/utils.py:139: RuntimeWarning: Couldn't detect a suitable IP address for reaching '8.8.8.8', defaulting to '127.0.0.1': [Errno 101] Network is unreachable
  RuntimeWarning,

Split the data set

For the training and testing of machine learning models it is crucial to split the data set into:

  1. Train data set which is used to train the weights of the neural network

  2. Validation data set which is used to check for overfitting (e.g. when using early stopping) and to optimize the hyperparameters

  3. Test data set which is used to to evaluate the trained model.

NOTE: It is important to understand that hyperparamters must be tuned so that the result is best for the Validation data set and not for the test data set. Otherwise you can not rule out the case that the specific hyperparameter setting just works good for the specific test data set but is not generally a good hyperparameter setting.

In the following cell the train and the validation data set are still one data set, because this array will be later splitted into two arrays when th model is fitted.

[10]:
import keras.backend as K
from ninolearn.learn.models.dem import DEM

# clear memory from previous sessions
K.clear_session()

# define the lead time
lead_time = 3

# get the features (X), the label (y) and
# the time axis of the label (timey)
X, y, timey = pipeline(lead_time)

# split the data set into
test_indeces = (timey>='2001-01-01') & (timey<='2018-12-01')
train_val_indeces = np.invert(test_indeces)

train_val_X, train_val_y, train_val_timey = X[train_val_indeces,:], y[train_val_indeces], timey[train_val_indeces]
testX, testy, testtimey = X[test_indeces,:], y[test_indeces], timey[test_indeces]

Fit the model

Now it is time to train the model! For this a random search is used for all keyword arguments that are passed in a list to the DEM.set_parameters() method.

[25]:
# initiated an instance of the DEM (Deep Ensemble Model) class
model = DEM()

# Set parameters
model.set_parameters(layers=1, dropout=[0.1, 0.5], noise_in=[0.1,0.5], noise_sigma=[0.1,0.5],
                     noise_mu=[0.1,0.5], l1_hidden=[0.0, 0.2], l2_hidden=[0, 0.2],
                     l1_mu=[0.0, 0.2], l2_mu=[0.0, 0.2], l1_sigma=[0.0, 0.2],
                     l2_sigma=[0.0, 0.2], lr=[0.0001,0.01], batch_size=100, epochs=500, n_segments=5,
                     n_members_segment=1, patience=30, verbose=0, std=True)

# Use a random search to find the optimal hyperparamteres
model.fit_RandomizedSearch(train_val_X, train_val_y, n_iter=20)

##################################################################
Search iteration Nr 1/20
##################################################################

Train member Nr 1/5
--------------------------------------
46/46 [==============================] - 0s 45us/step
Train member Nr 2/5
--------------------------------------
46/46 [==============================] - 0s 76us/step
Train member Nr 3/5
--------------------------------------
46/46 [==============================] - 0s 61us/step
Train member Nr 4/5
--------------------------------------
46/46 [==============================] - 0s 79us/step
Train member Nr 5/5
--------------------------------------
46/46 [==============================] - 0s 45us/step
Computation time: 8.4s
New best hyperparameters
--------------------------------------
Mean loss: 0.34392209892849557
{'layers': 1, 'neurons': 16, 'dropout': 0.12378305611318924, 'noise_in': 0.13682762378098945, 'noise_mu': 0.1594007153319167, 'noise_sigma': 0.26941913115524324, 'l1_hidden': 0.15920416814139493, 'l2_hidden': 0.14973220769991027, 'l1_mu': 0.0968840864748789, 'l2_mu': 0.18567445044732792, 'l1_sigma': 0.1606583760628583, 'l2_sigma': 0.02928805643650441, 'lr': 0.0054775495740690665, 'batch_size': 100}

##################################################################
Search iteration Nr 2/20
##################################################################

Train member Nr 1/5
--------------------------------------
46/46 [==============================] - 0s 51us/step
Train member Nr 2/5
--------------------------------------
46/46 [==============================] - 0s 45us/step
Train member Nr 3/5
--------------------------------------
46/46 [==============================] - 0s 46us/step
Train member Nr 4/5
--------------------------------------
46/46 [==============================] - 0s 51us/step
Train member Nr 5/5
--------------------------------------
46/46 [==============================] - 0s 78us/step
Computation time: 9.3s
New best hyperparameters
--------------------------------------
Mean loss: 0.3435987807486368
{'layers': 1, 'neurons': 16, 'dropout': 0.25141333998178195, 'noise_in': 0.2329862534019585, 'noise_mu': 0.2813460046840733, 'noise_sigma': 0.14595068639744221, 'l1_hidden': 0.08553348071011933, 'l2_hidden': 0.11981746410750115, 'l1_mu': 0.10558548934586615, 'l2_mu': 0.05080434557231146, 'l1_sigma': 0.037151218694687606, 'l2_sigma': 0.19769939661821626, 'lr': 0.006687247196565577, 'batch_size': 100}

##################################################################
Search iteration Nr 3/20
##################################################################

Train member Nr 1/5
--------------------------------------
46/46 [==============================] - 0s 81us/step
Train member Nr 2/5
--------------------------------------
46/46 [==============================] - 0s 47us/step
Train member Nr 3/5
--------------------------------------
46/46 [==============================] - 0s 121us/step
Train member Nr 4/5
--------------------------------------
46/46 [==============================] - 0s 49us/step
Train member Nr 5/5
--------------------------------------
46/46 [==============================] - 0s 66us/step
Computation time: 10.3s
New best hyperparameters
--------------------------------------
Mean loss: 0.11411247535244277
{'layers': 1, 'neurons': 16, 'dropout': 0.30452543907304586, 'noise_in': 0.21534727932400557, 'noise_mu': 0.2830000465254644, 'noise_sigma': 0.20073478581880644, 'l1_hidden': 0.08204434904474006, 'l2_hidden': 0.15817156327512816, 'l1_mu': 0.044876511619725834, 'l2_mu': 0.07501615431427888, 'l1_sigma': 0.13807434833687507, 'l2_sigma': 0.003930864203235141, 'lr': 0.005892578009524626, 'batch_size': 100}

##################################################################
Search iteration Nr 4/20
##################################################################

Train member Nr 1/5
--------------------------------------
46/46 [==============================] - 0s 44us/step
Train member Nr 2/5
--------------------------------------
46/46 [==============================] - 0s 84us/step
Train member Nr 3/5
--------------------------------------
46/46 [==============================] - 0s 90us/step
Train member Nr 4/5
--------------------------------------
46/46 [==============================] - 0s 50us/step
Train member Nr 5/5
--------------------------------------
46/46 [==============================] - 0s 64us/step
Computation time: 8.7s

##################################################################
Search iteration Nr 5/20
##################################################################

Train member Nr 1/5
--------------------------------------
46/46 [==============================] - 0s 43us/step
Train member Nr 2/5
--------------------------------------
46/46 [==============================] - 0s 86us/step
Train member Nr 3/5
--------------------------------------
46/46 [==============================] - 0s 73us/step
Train member Nr 4/5
--------------------------------------
46/46 [==============================] - 0s 49us/step
Train member Nr 5/5
--------------------------------------
46/46 [==============================] - 0s 85us/step
Computation time: 9.3s

##################################################################
Search iteration Nr 6/20
##################################################################

Train member Nr 1/5
--------------------------------------
46/46 [==============================] - 0s 43us/step
Train member Nr 2/5
--------------------------------------
46/46 [==============================] - 0s 49us/step
Train member Nr 3/5
--------------------------------------
46/46 [==============================] - 0s 49us/step
Train member Nr 4/5
--------------------------------------
46/46 [==============================] - 0s 45us/step
Train member Nr 5/5
--------------------------------------
46/46 [==============================] - 0s 64us/step
Computation time: 10.0s

##################################################################
Search iteration Nr 7/20
##################################################################

Train member Nr 1/5
--------------------------------------
46/46 [==============================] - 0s 74us/step
Train member Nr 2/5
--------------------------------------
46/46 [==============================] - 0s 47us/step
Train member Nr 3/5
--------------------------------------
46/46 [==============================] - 0s 49us/step
Train member Nr 4/5
--------------------------------------
46/46 [==============================] - 0s 49us/step
Train member Nr 5/5
--------------------------------------
46/46 [==============================] - 0s 55us/step
Computation time: 9.2s

##################################################################
Search iteration Nr 8/20
##################################################################

Train member Nr 1/5
--------------------------------------
46/46 [==============================] - 0s 43us/step
Train member Nr 2/5
--------------------------------------
46/46 [==============================] - 0s 49us/step
Train member Nr 3/5
--------------------------------------
46/46 [==============================] - 0s 49us/step
Train member Nr 4/5
--------------------------------------
46/46 [==============================] - 0s 47us/step
Train member Nr 5/5
--------------------------------------
46/46 [==============================] - 0s 67us/step
Computation time: 8.9s

##################################################################
Search iteration Nr 9/20
##################################################################

Train member Nr 1/5
--------------------------------------
46/46 [==============================] - 0s 46us/step
Train member Nr 2/5
--------------------------------------
46/46 [==============================] - 0s 70us/step
Train member Nr 3/5
--------------------------------------
46/46 [==============================] - 0s 130us/step
Train member Nr 4/5
--------------------------------------
46/46 [==============================] - 0s 44us/step
Train member Nr 5/5
--------------------------------------
46/46 [==============================] - 0s 47us/step
Computation time: 9.8s

##################################################################
Search iteration Nr 10/20
##################################################################

Train member Nr 1/5
--------------------------------------
46/46 [==============================] - 0s 43us/step
Train member Nr 2/5
--------------------------------------
46/46 [==============================] - 0s 109us/step
Train member Nr 3/5
--------------------------------------
46/46 [==============================] - 0s 48us/step
Train member Nr 4/5
--------------------------------------
46/46 [==============================] - 0s 83us/step
Train member Nr 5/5
--------------------------------------
46/46 [==============================] - 0s 50us/step
Computation time: 8.5s

##################################################################
Search iteration Nr 11/20
##################################################################

Train member Nr 1/5
--------------------------------------
46/46 [==============================] - 0s 46us/step
Train member Nr 2/5
--------------------------------------
46/46 [==============================] - 0s 46us/step
Train member Nr 3/5
--------------------------------------
46/46 [==============================] - 0s 67us/step
Train member Nr 4/5
--------------------------------------
46/46 [==============================] - 0s 63us/step
Train member Nr 5/5
--------------------------------------
46/46 [==============================] - 0s 93us/step
Computation time: 8.7s

##################################################################
Search iteration Nr 12/20
##################################################################

Train member Nr 1/5
--------------------------------------
46/46 [==============================] - 0s 57us/step
Train member Nr 2/5
--------------------------------------
46/46 [==============================] - 0s 82us/step
Train member Nr 3/5
--------------------------------------
46/46 [==============================] - 0s 73us/step
Train member Nr 4/5
--------------------------------------
46/46 [==============================] - 0s 64us/step
Train member Nr 5/5
--------------------------------------
46/46 [==============================] - 0s 60us/step
Computation time: 9.3s

##################################################################
Search iteration Nr 13/20
##################################################################

Train member Nr 1/5
--------------------------------------
46/46 [==============================] - 0s 57us/step
Train member Nr 2/5
--------------------------------------
46/46 [==============================] - 0s 47us/step
Train member Nr 3/5
--------------------------------------
46/46 [==============================] - 0s 46us/step
Train member Nr 4/5
--------------------------------------
46/46 [==============================] - 0s 78us/step
Train member Nr 5/5
--------------------------------------
46/46 [==============================] - 0s 54us/step
Computation time: 8.9s

##################################################################
Search iteration Nr 14/20
##################################################################

Train member Nr 1/5
--------------------------------------
46/46 [==============================] - 0s 47us/step
Train member Nr 2/5
--------------------------------------
46/46 [==============================] - 0s 72us/step
Train member Nr 3/5
--------------------------------------
46/46 [==============================] - 0s 54us/step
Train member Nr 4/5
--------------------------------------
46/46 [==============================] - 0s 54us/step
Train member Nr 5/5
--------------------------------------
46/46 [==============================] - 0s 95us/step
Computation time: 8.6s

##################################################################
Search iteration Nr 15/20
##################################################################

Train member Nr 1/5
--------------------------------------
46/46 [==============================] - 0s 48us/step
Train member Nr 2/5
--------------------------------------
46/46 [==============================] - 0s 50us/step
Train member Nr 3/5
--------------------------------------
46/46 [==============================] - 0s 48us/step
Train member Nr 4/5
--------------------------------------
46/46 [==============================] - 0s 57us/step
Train member Nr 5/5
--------------------------------------
46/46 [==============================] - 0s 79us/step
Computation time: 8.6s

##################################################################
Search iteration Nr 16/20
##################################################################

Train member Nr 1/5
--------------------------------------
46/46 [==============================] - 0s 52us/step
Train member Nr 2/5
--------------------------------------
46/46 [==============================] - 0s 60us/step
Train member Nr 3/5
--------------------------------------
46/46 [==============================] - 0s 42us/step
Train member Nr 4/5
--------------------------------------
46/46 [==============================] - 0s 58us/step
Train member Nr 5/5
--------------------------------------
46/46 [==============================] - 0s 83us/step
Computation time: 8.9s

##################################################################
Search iteration Nr 17/20
##################################################################

Train member Nr 1/5
--------------------------------------
46/46 [==============================] - 0s 97us/step
Train member Nr 2/5
--------------------------------------
46/46 [==============================] - 0s 39us/step
Train member Nr 3/5
--------------------------------------
46/46 [==============================] - 0s 84us/step
Train member Nr 4/5
--------------------------------------
46/46 [==============================] - 0s 79us/step
Train member Nr 5/5
--------------------------------------
46/46 [==============================] - 0s 70us/step
Computation time: 13.1s

##################################################################
Search iteration Nr 18/20
##################################################################

Train member Nr 1/5
--------------------------------------
46/46 [==============================] - 0s 54us/step
Train member Nr 2/5
--------------------------------------
46/46 [==============================] - 0s 45us/step
Train member Nr 3/5
--------------------------------------
46/46 [==============================] - 0s 50us/step
Train member Nr 4/5
--------------------------------------
46/46 [==============================] - 0s 109us/step
Train member Nr 5/5
--------------------------------------
46/46 [==============================] - 0s 78us/step
Computation time: 10.5s

##################################################################
Search iteration Nr 19/20
##################################################################

Train member Nr 1/5
--------------------------------------
46/46 [==============================] - 0s 92us/step
Train member Nr 2/5
--------------------------------------
46/46 [==============================] - 0s 59us/step
Train member Nr 3/5
--------------------------------------
46/46 [==============================] - 0s 45us/step
Train member Nr 4/5
--------------------------------------
46/46 [==============================] - 0s 81us/step
Train member Nr 5/5
--------------------------------------
46/46 [==============================] - 0s 75us/step
Computation time: 8.6s

##################################################################
Search iteration Nr 20/20
##################################################################

Train member Nr 1/5
--------------------------------------
46/46 [==============================] - 0s 49us/step
Train member Nr 2/5
--------------------------------------
46/46 [==============================] - 0s 47us/step
Train member Nr 3/5
--------------------------------------
46/46 [==============================] - 0s 54us/step
Train member Nr 4/5
--------------------------------------
46/46 [==============================] - 0s 49us/step
Train member Nr 5/5
--------------------------------------
46/46 [==============================] - 0s 51us/step
Computation time: 9.8s

##################################################################
Refit the model with best hyperparamters
##################################################################

{'layers': 1, 'neurons': 16, 'dropout': 0.30452543907304586, 'noise_in': 0.21534727932400557, 'noise_mu': 0.2830000465254644, 'noise_sigma': 0.20073478581880644, 'l1_hidden': 0.08204434904474006, 'l2_hidden': 0.15817156327512816, 'l1_mu': 0.044876511619725834, 'l2_mu': 0.07501615431427888, 'l1_sigma': 0.13807434833687507, 'l2_sigma': 0.003930864203235141, 'lr': 0.005892578009524626, 'batch_size': 100}
Train member Nr 1/5
--------------------------------------
46/46 [==============================] - 0s 56us/step
Train member Nr 2/5
--------------------------------------
46/46 [==============================] - 0s 47us/step
Train member Nr 3/5
--------------------------------------
46/46 [==============================] - 0s 51us/step
Train member Nr 4/5
--------------------------------------
46/46 [==============================] - 0s 50us/step
Train member Nr 5/5
--------------------------------------
46/46 [==============================] - 0s 58us/step
Computation time: 9.9s
best loss search: 0.11411247535244277
loss refitting : 0.14777255660813787

Make predictions for the test data set

Now we can use the trained models to make predicitons on the test data set to evaluate how good the model perfoms on a data set that it never saw before.

[26]:
from ninolearn.plot.evaluation import plot_correlation

pred_mean, pred_std = model.predict(testX)

Plot the prediction

Let’s see how the predicion is looking like

[27]:
import matplotlib.pyplot as plt
from ninolearn.plot.prediction import plot_prediction
import pandas as pd

plt.subplots(figsize=(15,3.5))
plt.axhspan(-0.5,
            -6,
            facecolor='blue',
            alpha=0.1,zorder=0)

plt.axhspan(0.5,
            6,
            facecolor='red',
            alpha=0.1,zorder=0)

plt.xlim(testtimey[0], testtimey[-1])
plt.ylim(-3,3)

# plot the prediction
plot_prediction(testtimey, pred_mean, std=pred_std, facecolor='royalblue', line_color='navy')

# plot the observation
plt.plot(timey, y, "k")
[27]:
[<matplotlib.lines.Line2D at 0x7fd9fe77cb38>]
../_images/jupyter_notebook_tutorials_neural_networks_10_1.png

Evaluate the model

We can evaluate the model a bit more quantitatively using the loss function that was used to train the model, namely the negative-log-likelihood of the Gaussian and the correlation between the predicted mean and the observed ONI index.

[29]:
loss = model.evaluate(testy, pred_mean, pred_std)
print(f"Loss (Negative-Log-Likelihood): {loss}")

# make a plot of the seasonal correaltion
# note: - pd.tseries.offsets.MonthBegin(1) appears to ensure that the correlations are plotted
# agains the correct season
plot_correlation(testy, pred_mean, testtimey - pd.tseries.offsets.MonthBegin(1), title="")
Loss (Negative-Log-Likelihood): 0.1747595046770062
../_images/jupyter_notebook_tutorials_neural_networks_12_1.png