Epistimio/orion: Python API and New Algorithms

Important changes

The python API is finally ready for release v0.1.8! :tada:

Python API

An API is now available to run experiments directly from python instead of using the commandline.

from orion.client import create_experiment

experiment = create_experiment(
   name='foo',
   space=dict(x='uniform(-50,50)'))

trial = experiment.suggest()

# Do something using trial.params['x']

results = [dict(
    name='dummy_objective',
    type='objective',
    value=dummy_objective)]

experiment.observe(trial, results)

Current API provides a simple function workon for cheap experiments that can be executed by a single worker, and a generic ExperimentClient (see example above) object for optimization with multiple workers.

See documentation for more details.

New Algorithms Hyperband

Hyperband extends the Successive Halving algorithm by providing a way to exploit a fixed budget with different number of configurations for SuccessiveHalving algorithm to evaluate. It is especially useful when the trials are expensive to run and cheap noisy evaluations are possible. Think of it as using early evaluation during training to filter out bad candidates.

For more information on the algorithm, see original paper.

TPE

Tree-structured Parzen Estimator (TPE) algorithm is one of Sequential Model-Based Global Optimization (SMBO) algorithms, which will build models to propose new points based on the historical observed trials.

Instead of modeling p(y|x) like other SMBO algorithms, TPE models p(x|y) and p(y), and p(x|y) is modeled by transforming that generative process, replacing the distributions of the configuration prior with non-parametric densities.

TPE has the advantage of scaling particularly well compared to most Model-Based algorithm which are typically sequential. It does not model however dependencies between hyper-parameters, they are assumed independent.

For more information on the algorithm, see original papers at:

• Algorithms for Hyper-Parameter Optimization
• Making a Science of Model Search: Hyperparameter Optimizationin Hundreds of Dimensions for Vision Architectures

Storage

To support integration with other tools and services such as MLFlow or Weight & Biases we wrapped our previous database backend with a storage backend. The database backends are now available within the Legagy storage backend. In addition, we now have a backend for Track. The latter is planned to serve as a bridge between Oríon and other experiment management platforms or services. Track package development is on the ice for now, but contributions are very much welcomed. :)

Drop python 3.5, support 3.8

Although Oríon may still be compatible with python 3.5 we do not maintain it's support anymore. Python 3.8 is now officially supported.

Precision of real dimensions

By default Oríon now rounds hyperparameters to 4 decimals (ex 0.00041239123 would become 0.0004124). The rational is that little variations on continuous hyperparameters typically leads to little variations in the in objective. When sharing hyperparameters (ex: in publications), one can now share the rounded values with the exact corresponding objectives instead of rounding the hyperparameters after the execution and risk sharing unreproducible results.

Rework of documentation

The documentation has been through a major rework.

• The introduction has been updated to better reflect current features.
• A section Getting Started and an Overview was added.
• The plugins documentation was updated with a new cookiecutter template to create algorithm plugins.
• A minimalist example of scikit-learn was added to serve as the simplest tutorial for Oríon.
• Extensive documentation on contributions has been added.

Detailed list of changes New features

• Storage Backend with Track and Legacy (#289, #302, #316, #318)
• Python API (#297, #299, #300, #305, #307, #311, #312, #355, #399, #401)
• Add option to ignore some commandline options of user's script (#308)
• Add option to ignore code changes in user's script (#310)
• Add trial hash id that does not include fidelifty (useful to resume across fidelities)(#313, #322)
• Customize precision (#331)
• Support commandline call where first argument is not user's script (#333, #338)
• Add support for non-standard -args (#341)
• Add more helper functions for cmdline client (#343)
• Hyperband implementation (#354, #363, #383)
• TPE implementation (#381, #387, #389)

Breaking changes

• Drop python 3.5 support in favor of 3.8 (#303)
• Remove deprecated score_handle (#315)
• Make global, local and cmdline args coherent (#349, #404)

Bug Fixes

• Fix EphemeralDB index update after document deletion (#306)
• Support small fidelity scales for ASHA (#314)
• Algo redefinition should not drop algo config (#319)
• Random algorithm will hang if search space is smaller than specified trials number (#336)
• Inconsistent output in orion status and orion list (#342)
• Use EVC tree trials in producer (#347)
• Update fidelity sample method to honor required n_samples (#351)
• Add max_trials to algorithm.is_done (#360)
• Handle trials with corrupted status (#372)
• Making upper bound inclusive (#373)
• Fix --debug (#374)
• Round ASHA budget properly (#402)
• Update Plugins documentation section (#406)
• Print help when calling orion alone (#408)

Other improvements

• improve error message for ASHA (#317)
• Print important information at end of worker (#345)
• Add comprehensive error message for branching err (#346)
• Make global, local and cmdline args coherent (#349, #404)
• Improve tox configuration (#350, #398)
• Generic cardinality check in base algorithm (#352)
• Avoid duplicates in RandomSearch (#361)
• Handle timeout in DBs (PickledDB in particular) (#362)
• Add stress test (#364)
• Remove useless deprecation warnings (#376)
• Ignore test files for code coverage (#377)
• Show proper yaml syntax in resolve_config deprecation warnings containing None (#391)
• Raise exception when no prior is provided (#410)

Documentation improvements

• Add citation section (#301, #379)
• Fix doc for bayesopt's alpha (#309)
• Formatted commands and added a command description (#327)
• Clone via HTTPS instead of SSH (#328)
• Minimalist example with scikit-learn (#339)
• Full configuration documentation (#349)
• Contribution guidelines (#356)
• Update developer documentation (#365, #378)
• Release & Packaging (#371, #411)
• Update the documentation structure (#375)
• Refreshed introduction of Oríon's features (#380, #384)
• Overhaul of the documentation for new comers (#385, #403)
• Add Issue Template (#386)
• Add PR Template (#388)
• Add doc about label categories (#397)
• Update Plugins documentation section (#405)
• Document Windows compatibility (#409)
• General documentation updates (#332, #340, #344, #366, #367, #370)