deephyper/deephyper: Changelog - DeepHyper 0.2.5

General Full API documentation

The DeepHyper API is now fully documented at DeepHyper API

Tensorflow-Probability as a new dependency

TensorFlow Probability is now part of DeepHyper default set of dependencies

Automatique submission with Ray at ALCF

It is now possible to directly submit with deephyper ray-submit ... for DeepHyper at the ALCF. This feature is only available on ThetaGPU for now but can be extended to other systems by following this script.

ThetaGPU at ALCF

• New installation documentation is available at Installation ThetaGPU (ALCF)
• A new user guide is available at Running on ThetaGPU (ALCF) to understand how to run manually and automatically DeepHyper on ThetaGPU.

New documentation for auto-sklearn search with DeepHyper

The access to auto-sklearn features was changed to deephyper.sklearn and a new documentation is available for this feature at User guide: AutoSklearn

New command lines for DeepHyper Analytics

The deephyper-analytics command was modified and enhanced with new features. The see the full updated documentation follow DeepHyper Analytics Tools.

The topk command is now available to have quick feedback from the results of an experiment:

$ deephyper-analytics topk combo_8gpu_8_agebo/infos/results.csv -k 2
'0':
arch_seq: '[229, 0, 22, 1, 1, 53, 29, 1, 119, 1, 0, 116, 123, 1, 273, 0, 1, 388]'
batch_size: 59
elapsed_sec: 10259.2741303444
learning_rate: 0.0001614947
loss: log_cosh
objective: 0.9236862659
optimizer: adam
patience_EarlyStopping: 22
patience_ReduceLROnPlateau: 10
'1':
arch_seq: '[229, 0, 22, 0, 1, 235, 29, 1, 313, 1, 0, 116, 123, 1, 37, 0, 1, 388]'
batch_size: 51
elapsed_sec: 8818.2674164772
learning_rate: 0.0001265946
loss: mae
objective: 0.9231553674
optimizer: nadam
patience_EarlyStopping: 23
patience_ReduceLROnPlateau: 14

Neural architecture search New documentation for the problem definition

A new documentation for the neural architecture search problem setup can be found here.

It is now possible to defined auto-tuned hyperparameters in addition of the architecture in a NAS Problem.

New Algorithms for Joint Hyperparameter and Neural Architecture Search

Three new algorithms are available to run a joint Hyperparameter and neural architecture search. The Hyperparameter optimisation is defined as HPO and neural architecture search as NAS.

• agebo (Aging Evolution for NAS with Bayesian Optimisation for HPO)
• ambsmixed (an extension of Asynchronous Model-Based Search for HPO + NAS)
• regevomixed (an extension of regularised evolution for HPO + NAS)

A run function to use data-parallelism with Tensorflow

A new run function to use data-parallelism during neural architecture search is available (link to code)

To use this function pass it to the run argument of the command line such as:

deephyper nas agebo ... --run deephyper.nas.run.tf_distributed.run ... --num-cpus-per-task 2 --num-gpus-per-task 2 --evaluator ray --address auto ...

This function allows for new hyperparameters in the Problem.hyperparameters(...):

...
Problem.hyperparameters(
    ...
    lsr_batch_size=True,
    lsr_learning_rate=True,
    warmup_lr=True,
    warmup_epochs=5,
    ...
)
...

Optimization of the input pipeline for the training

The data-ingestion pipeline was better optimised to reduce the overheads on GPU instances:

self.dataset_train = (
  self.dataset_train.cache()            
  .shuffle(self.train_size, reshuffle_each_iteration=True)            
  .batch(self.batch_size)
  .prefetch(tf.data.AUTOTUNE)
  .repeat(self.num_epochs)        
)

Easier model generation from Neural Architecture Search results

A new method is now available from the Problem object Problem.get_keras_model(arch_seq) to easily build a Keras model instance from an arch_seq (list encoding a neural network).