icecube/dnn_reco: Version 2.0.0

Release Notes

This is a major release that changes the internal structure of dnn-reco from usage of tf.compat.v1 symbols to native tensorflow 2. This required a major restructuring of the code. Any models created prior to this version will be incompatible. Some of the larger changes are listed below. Notes on how to port old configuration files for model training to dnn-reco v2.0.1 are provided below.

Major changes and added functionality:

• Removal of tf.compat.v1 symbols
• Class-based NN architecture via tf.Module
• Added support for automatic data_settings extraction for training data generated with the processing framework ic3-processing
• Updated documentation and tutorial to reflect these changes and shift to ic3-processing
• Save/load of complete state including optimizer is now supported during training. In version 1, the optimizer state nor the global training step counter were saved
• Added version check for restoring models: requires major version number to match, warning if different version but compatible
• Modules defined in configuration file (labels, misc, filter, weighting, loss, model architecture) are now defined via a full import string instead of requiring these to be part of dnn-reco itself. This now allows for more flexible usage of dnn-reco as individual components can now easily be used from external python packages.

Porting v1 configs to v2

As much of the underlying code has changed, some necessary changes to the configuration file format and keys were applied. Some examples for v2 configs are provided in the configs directory. A list of necessary updates are listed below

Change to full class/function import string:

'model_file' : 'general_IC86_models'
'model_name' : 'general_model_IC86_opt4'
--> model_class: 'dnn_reco.modules.models.general_IC86_cnn.GeneralIC86CNN'

'data_handler_label_file': 'default_labels'
'data_handler_label_name': 'simple_label_loader'
--> data_handler_label_class: dnn_reco.modules.data.labels.default_labels.simple_label_loader

'data_handler_misc_file': 'default_misc'
'data_handler_misc_name': 'general_misc_loader'
--> data_handler_misc_class: dnn_reco.modules.data.misc.default_misc.general_misc_loader

'data_handler_filter_file': 'default_filter'
'data_handler_filter_name': 'general_filter'
--> data_handler_filter_class: dnn_reco.modules.data.filter.default_filter.general_filter

'event_weight_file': 'event_weights'
'event_weight_name': 'clipped_astroness_weights
--> event_weight_class: dnn_reco.modules.data.event_weights.event_weights.clipped_astroness_weights

'evaluation_file': default_evaluation
'evaluation_name': eval_direction
--> evaluation_class: dnn_reco.modules.evaluation.default_evaluation.eval_direction

'loss_file': 'default_loss',
'loss_name': 'mse_and_cross_entropy',
--> 'loss_class': 'dnn_reco.modules.loss.default_loss.mse_and_cross_entropy',

'loss_file': 'default_loss',
'loss_name': 'gaussian_likelihood',
--> 'loss_class': 'dnn_reco.modules.loss.default_loss.gaussian_likelihood',

'model_checkpoint_path' : "../checkpoints/nn_model/\
                          {model_file}__\
                          {model_name}/\
                          {unique_name}/model"
 --> 'model_checkpoint_path' : "../checkpoints/nn_model/{unique_name}/model"

 'log_path' : "../logs/\
            {unique_name}/\
            {model_file}__\
            {model_name}"
--> 'log_path' : "../logs/{unique_name}"

Changes for optimizer settings

The keys clip_gradients_value and remove_nan_gradients are now explicitly required to be defined in the model_optimizer_dict. Previously their definition in the configuration file was optional.

Changes to checkpoint creation

A tf.train.CheckpointManager object is now used to write checkpoints. The manager is configured with the defined keyword arguments in the required key model_checkpoint_manager_kwargs. Example to retain the latest 3 checkpoints:

'model_checkpoint_manager_kwargs': {
    'max_to_keep': 3,
}

NN architecture settings

All settings for the NN architecture must now be in a separate dictionary of the name model_kwargs. These were typically all keys defined below the # NN Model Architecture header in the standard configuration files. Some name changes include:

conv_upper_DeepCore_settings -> conv_upper_deepcore_settings
conv_lower_DeepCore_settings -> conv_lower_deepcore_settings
conv_IC78_settings -> conv_ic78_settings

Additional model settings that were previously defined elsewhere:

• tf_random_seed is now defined in model_kwargs under the key random_seed
• Previously there was a general_model_IC86 and general_model_IC86_opt4 model. These are now a single model and one can choose the same functionality by setting the add_prediction_to_unc_input key (True for old general_model_IC86, False for old general_model_IC86_opt4)
• model gets its own dtype key to specify the float precision
• keep_probability_list was changed to individual keys added to the model_kwargs (see below)

The setting for the convolution layers must now define the convolution type via the method_list key. Additionally the keep rates for dropout layers must now be provided explicitly in addition to random_seed, is_training and dtype. An example:

model_class: 'dnn_reco.modules.models.general_IC86_cnn.GeneralIC86CNN'

model_kwargs: {
    is_training : True,
    random_seed: 42,

    dtype: 'float32',
    enforce_direction_norm: False,
    add_prediction_to_unc_input: False,

    keep_prob_dom: 0.95,
    keep_prob_conv: 1.0,
    keep_prob_flat: 1.0,
    keep_prob_fc: 1.0,

    # 2D convolutional layer of upper DeepCore
    conv_upper_deepcore_settings: {
        ...
        'method_list': 'convolution',
    },

    # 2D convolutional layer of lower DeepCore
    conv_lower_deepcore_settings: {
        ...
        'method_list': 'convolution',
    },

    # 3D hexagonal convolution over main IceCube array (IC78)
    conv_ic78_settings : {
        ...
        'method_list': 'hex_convolution',
    },

    # Fully connected layer settings (Combine results from convolutions)
    fc_settings: {...},

    # Fully connected layer settings for uncertainty subnetwork
    fc_unc_settings: {...},
}

Full Changelog: https://github.com/icecube/dnn_reco/compare/v1.0.2...v2.0.1