Merge pull request #695 from Epistimio/release-v0.2.0rc1

Release v0.2.0rc1
Epistimio · Nov 24, 2021 · 6ee3d63 · 6ee3d63
2 parents 0ef3eea + 93e49b5
commit 6ee3d63
Show file tree

Hide file tree

Showing 110 changed files with 2,957 additions and 2,304 deletions.
diff --git a/LICENSE b/LICENSE
@@ -1,6 +1,6 @@
 Software License Agreement (BSD License)
 
- Copyright (c) 2017-2020, Epistímio.
+ Copyright (c) 2017-2021, Epistímio.
  All rights reserved.
 
  Redistribution and use in source and binary forms, with or without

diff --git a/README.rst b/README.rst
@@ -118,7 +118,7 @@ If you use Oríon for published work, please cite our work using the following b
 
 .. code-block:: bibtex
 
-   @software{xavier_bouthillier_2021_0_1_15,
+   @software{xavier_bouthillier_2021_0_2_0,
      author       = {Xavier Bouthillier and
                      Christos Tsirigotis and
                      François Corneau-Tremblay and
@@ -142,10 +142,10 @@ If you use Oríon for published work, please cite our work using the following b
                      Pascal Lamblin and
                      Christopher Beckham},
      title        = {{Epistimio/orion: Asynchronous Distributed Hyperparameter Optimization}},
-     month        = may,
+     month        = nov,
      year         = 2021,
      publisher    = {Zenodo},
-     version      = {v0.1.17},
+     version      = {v0.2.0},
      doi          = {10.5281/zenodo.3478592},
      url          = {https://doi.org/10.5281/zenodo.3478592}
    }

diff --git a/ROADMAP.md b/ROADMAP.md
@@ -1,25 +1,35 @@
 # Roadmap
-Last update Sep 14th, 2021
+Last update Nov 23rd, 2021
 
 ## Next releases - Short-Term
 
-### v0.2
+### v0.2.1
 
-#### Generic `Optimizer` interface supporting various types of algorithms
+- New master process to enhance parallelisation efficiency.
+- [PBT](https://arxiv.org/abs/1711.09846)
 
-Change interface to support trial object instead of curated lists. This is necessary to support algorithms such as PBT.
+### v0.2.2
+
+- Use shared algo serialization instead of replications to enhance parallelisation efficiency.
+- [DEBH](https://arxiv.org/abs/2105.09821)
+
+### v0.2.3
+
+- [HEBO](https://github.com/huawei-noah/HEBO/tree/master/HEBO/archived_submissions/hebo)
+
+### v0.2.4
 
-#### More Optimizers
-- [PBT](https://arxiv.org/abs/1711.09846)
 - [BOHB](https://ml.informatik.uni-freiburg.de/papers/18-ICML-BOHB.pdf)
 
+## Next releases - Mid-Term
+
 #### Simple dashboard specific to monitoring and benchmarking of Black-Box optimization
 - Specific to hyper parameter optimizations
 - Provide status of experiments
 
 #### Leveraging previous experiences
 Leveraging the knowledge base contained in the EVC of previous trials to optimize and drive new
- trials.
+trials.
 
 ## Next releases - Long-Term
 

diff --git a/docs/src/code/algo/asha.rst b/docs/src/code/algo/asha.rst
@@ -3,6 +3,6 @@ Asynchronous Successive Halving Algorithm
 
 Can't build documentation because of import order.
 Sphinx is loading ``orion.algo.asha`` before ``orion.algo`` and therefore
-there is a cycle between the definition of ``OptimizationAlgorithm`` and
+there is a cycle between the definition of ``BaseAlgorithm`` and
 ``ASHA`` as the meta-class ``Factory`` is trying to import ``ASHA``.
 `PR #135 <https://github.com/Epistimio/orion/pull/135/files>`_ should get rid of this problem.
diff --git a/docs/src/code/core/io/database.rst b/docs/src/code/core/io/database.rst
@@ -12,5 +12,3 @@ Databases
 .. automodule:: orion.core.io.database
    :members:
    :show-inheritance:
-
-
diff --git a/docs/src/code/core/utils.rst b/docs/src/code/core/utils.rst
@@ -9,7 +9,6 @@ Utilities
    utils/format_trials
    utils/format_terminal
    utils/singleton
-   utils/points
 
 .. automodule:: orion.core.utils
    :members:
diff --git a/docs/src/code/core/utils/points.rst b/docs/src/code/core/utils/points.rst
diff --git a/docs/src/install/gettingstarted.rst b/docs/src/install/gettingstarted.rst
@@ -55,7 +55,7 @@ For the previous example, we would run
 
 .. code-block:: console
 
-   $ orion hunt -n <experiment name> script.py --lr~'loguniform(1e-5, 1.0)'
+   $ orion hunt -n <experiment name> --max-trials 10 python script.py --lr~'loguniform(1e-5, 1.0)'
 
 This is going to start the optimization process using the default optimization algorithm and sample
 the values for the ``lr`` hyper-parameter in a log uniform distribution between 0.00001 et 1.0. Each

diff --git a/docs/src/user/config.rst b/docs/src/user/config.rst
@@ -110,7 +110,7 @@ Full Example of Global Configuration
         heartbeat: 120
         interrupt_signal_code: 130
         max_broken: 10
-        max_idle_time: 60
+        reservation_timeout: 60
         max_trials: 1000000000
         user_script_config: config
 
@@ -365,7 +365,7 @@ Worker
         heartbeat: 120
         interrupt_signal_code: 130
         max_broken: 10
-        max_idle_time: 60
+        reservation_timeout: 60
         max_trials: 1000000000
         user_script_config: config
 
@@ -464,21 +464,37 @@ max_broken
     Maximum number of broken trials before worker stops.
 
 
+.. _config_worker_reservation_timeout:
+
+reservation_timeout
+~~~~~~~~~~~~~~~~~~~
+
+:Type: int
+:Default: 60
+:Env var: ORION_RESERVATION_TIMEOUT
+:Description:
+    Maximum time the experiment can spend trying to reserve a new suggestion. Such timeout are
+    generally caused by slow database, large number of concurrent workers leading to many race
+    conditions or small search spaces with integer/categorical dimensions that may be fully
+    explored.
+
 
 .. _config_worker_max_idle_time:
 
 max_idle_time
 ~~~~~~~~~~~~~
 
+.. warning::
+
+   **DEPRECATED.** This argument will be removed in v0.3.
+   Use :ref:`config_worker_reservation_timeout` instead.
+
 :Type: int
 :Default: 60
 :Env var: ORION_MAX_IDLE_TIME
 :Description:
-    Maximum time the producer can spend trying to generate a new suggestion.Such timeout are
-    generally caused by slow database, large number of concurrent workers leading to many race
-    conditions or small search spaces with integer/categorical dimensions that may be fully
-    explored.
-
+    (DEPRECATED) This argument will be removed in v0.3. Use :ref:`config_worker_reservation_timeout`
+    instead.
 
 
 .. _config_worker_interrupt_signal_code:

diff --git a/docs/src/user/parallel.rst b/docs/src/user/parallel.rst
@@ -42,7 +42,7 @@ Executor backends
 It is also possible to execute multiple workers using the argument ``--n-workers`` in commandline
 or ``experiment.workon(n_workers)`` using the python API. The workers will work together
 using the same mechanisms explained above, but an
-:class:`orion.executor.base.Executor` backend will be used in addition
+:class:`orion.executor.base.BaseExecutor` backend will be used in addition
 to spawn the workers and maintain them alive. The default backend is :ref:`executor-joblib`.
 
 You can configure it

diff --git a/docs/src/user/storage.rst b/docs/src/user/storage.rst
@@ -477,18 +477,22 @@ Here's an example on how you could remove an experiment
 --------------
 
 .. automethod:: orion.core.io.database.Database.read
+   :noindex:
 
 :hidden:`write`
 ---------------
 
 .. automethod:: orion.core.io.database.Database.write
+   :noindex:
 
 :hidden:`remove`
 ----------------
 
 .. automethod:: orion.core.io.database.Database.remove
+   :noindex:
 
 :hidden:`read_and_write`
 ------------------------
 
 .. automethod:: orion.core.io.database.Database.read_and_write
+   :noindex:
diff --git a/setup.py b/setup.py
@@ -46,19 +46,24 @@
         "console_scripts": [
             "orion = orion.core.cli:main",
         ],
-        "OptimizationAlgorithm": [
+        "BaseAlgorithm": [
             "random = orion.algo.random:Random",
             "gridsearch = orion.algo.gridsearch:GridSearch",
             "asha = orion.algo.asha:ASHA",
             "hyperband = orion.algo.hyperband:Hyperband",
             "tpe = orion.algo.tpe:TPE",
             "EvolutionES = orion.algo.evolution_es:EvolutionES",
         ],
-        "Storage": [
+        "Database": [
+            "ephemeraldb = orion.core.io.database.ephemeraldb:EphemeralDB",
+            "pickleddb = orion.core.io.database.pickleddb:PickledDB",
+            "mongodb = orion.core.io.database.mongodb:MongoDB",
+        ],
+        "BaseStorageProtocol": [
             "track = orion.storage.track:Track",
             "legacy = orion.storage.legacy:Legacy",
         ],
-        "Executor": [
+        "BaseExecutor": [
             "singleexecutor = orion.executor.single_backend:SingleExecutor",
             "joblib = orion.executor.joblib_backend:Joblib",
             "dask = orion.executor.dask_backend:Dask",

diff --git a/src/orion/algo/asha.py b/src/orion/algo/asha.py
@@ -155,6 +155,8 @@ def __init__(
 
         self.brackets = self.create_brackets()
 
+        self.seed_rng(seed)
+
     def compute_bracket_idx(self, num):
         def assign_resources(n, remainings, totals):
             if n == 0 or remainings.sum() == 0:
@@ -192,7 +194,7 @@ def sample(self, num):
         return samples
 
     def suggest(self, num):
-        return super(ASHA, self).suggest(1)
+        return super(ASHA, self).suggest(num)
 
     def create_bracket(self, i, budgets, iteration):
         return ASHABracket(self, budgets, iteration)
@@ -217,32 +219,39 @@ def sample(self, num):
         should_have_n_trials = self.rungs[0]["n_trials"]
         return self.hyperband.sample_for_bracket(num, self)
 
-    def get_candidate(self, rung_id):
-        """Get a candidate for promotion"""
+    def get_candidates(self, rung_id):
+        """Get a candidate for promotion
+
+        Raises
+        ------
+        TypeError
+            If get_candidates is called before the entire rung is completed.
+        """
         rung = self.rungs[rung_id]["results"]
         next_rung = self.rungs[rung_id + 1]["results"]
 
         rung = list(
             sorted(
-                (objective, point)
-                for objective, point in rung.values()
+                (objective, trial)
+                for objective, trial in rung.values()
                 if objective is not None
             )
         )
         k = len(rung) // self.hyperband.reduction_factor
         k = min(k, len(rung))
 
+        candidates = []
         for i in range(k):
-            point = rung[i][1]
-            _id = self.hyperband.get_id(point, ignore_fidelity=True)
+            trial = rung[i][1]
+            _id = self.hyperband.get_id(trial, ignore_fidelity=True)
             if _id not in next_rung:
-                return point
+                candidates.append(trial)
 
-        return None
+        return candidates
 
     @property
     def is_filled(self):
-        """ASHA's first rung can always sample new points"""
+        """ASHA's first rung can always sample new trials"""
         return False
 
     def is_ready(self, rung_id=None):
@@ -254,7 +263,7 @@ def promote(self, num):
 
         The rungs are iterated over in reversed order, so that high rungs
         are prioritised for promotions. When a candidate is promoted, the loop is broken and
-        the method returns the promoted point.
+        the method returns the promoted trial.
 
         .. note ::
 
@@ -266,27 +275,34 @@ def promote(self, num):
         if num < 1 or self.is_done:
             return []
 
+        candidates = []
         for rung_id in range(len(self.rungs) - 2, -1, -1):
-            candidate = self.get_candidate(rung_id)
-            if candidate:
-
+            for candidate in self.get_candidates(rung_id):
                 # pylint: disable=logging-format-interpolation
                 logger.debug(
-                    "Promoting {point} from rung {past_rung} with fidelity {past_fidelity} to "
+                    "Promoting {trial} from rung {past_rung} with fidelity {past_fidelity} to "
                     "rung {new_rung} with fidelity {new_fidelity}".format(
-                        point=candidate,
+                        trial=candidate,
                         past_rung=rung_id,
-                        past_fidelity=candidate[self.hyperband.fidelity_index],
+                        past_fidelity=candidate.params[self.hyperband.fidelity_index],
                         new_rung=rung_id + 1,
                         new_fidelity=self.rungs[rung_id + 1]["resources"],
                     )
                 )
 
-                candidate = list(copy.deepcopy(candidate))
-                candidate[self.hyperband.fidelity_index] = self.rungs[rung_id + 1][
-                    "resources"
-                ]
+                candidate = candidate.branch(
+                    status="new",
+                    params={
+                        self.hyperband.fidelity_index: self.rungs[rung_id + 1][
+                            "resources"
+                        ]
+                    },
+                )
+
+                if not self.hyperband.has_suggested(candidate):
+                    candidates.append(candidate)
 
-                return [tuple(candidate)]
+                if len(candidates) >= num:
+                    return candidates
 
-        return []
+        return candidates