lenstronomy.Sampling.Pool package

Submodules

lenstronomy.Sampling.Pool.multiprocessing module

this file is taken from schwimmbad (https://github.com/adrn/schwimmbad) and an explicit fork by Aymeric Galan to replace the multiprocessing with the multiprocess dependence as for multi-threading, multiprocessing is not supporting dill (only pickle) which is required.

The class also extends with a is_master() definition

class lenstronomy.Sampling.Pool.multiprocessing.MultiPool(processes=None, initializer=None, initargs=(), **kwargs)

Bases: multiprocess.pool.Pool

A modified version of multiprocessing.pool.Pool that has better behavior with regard to KeyboardInterrupts in the map() method. (Original author: Peter K. G. Williams)

processes : int, optional

The number of worker processes to use; defaults to the number of CPUs.

initializer : callable, optional

If specified, a callable that will be invoked by each worker process when it starts.

initargs : iterable, optional

Arguments for initializer; it will be called as initializer(*initargs).

kwargs:

Extra arguments passed to the multiprocessing.pool.Pool superclass.

static enabled()

is_master()

is_worker()

map(func, iterable, chunksize=None, callback=None)

Equivalent to the built-in map() function and multiprocessing.pool.Pool.map(), without catching KeyboardInterrupt.

func : callable

A function or callable object that is executed on each element of the specified tasks iterable. This object must be picklable (i.e. it can’t be a function scoped within a function or a lambda function). This should accept a single positional argument and return a single object.

iterable : iterable

A list or iterable of tasks. Each task can be itself an iterable (e.g., tuple) of values or data to pass in to the worker function.

callback : callable, optional

An optional callback function (or callable) that is called with the result from each worker run and is executed on the master process. This is useful for, e.g., saving results to a file, since the callback is only called on the master thread.

results : list

A list of results from the output of each worker() call.

wait_timeout = 3600

lenstronomy.Sampling.Pool.pool module

this file is taken from schwimmbad (https://github.com/adrn/schwimmbad) and an explicit fork by Aymeric Galan to replace the multiprocessing with the multiprocess dependence as for multi-threading, multiprocessing is not supporting dill (only pickle) which is required.

Tests show that the MPI mode works with Python 3.7.2 but not with Python 3.7.0 on a specific system due to mpi4py dependencies and configurations.

Contributions by: - Peter K. G. Williams - Júlio Hoffimann Mendes - Dan Foreman-Mackey - Aymeric Galan - Simon Birrer

Implementations of four different types of processing pools:

• MPIPool: An MPI pool.
• MultiPool: A multiprocessing for local parallelization.
• SerialPool: A serial pool, which uses the built-in map function

lenstronomy.Sampling.Pool.pool.choose_pool(mpi=False, processes=1, **kwargs)

Extends the capabilities of the schwimmbad.choose_pool method.

It handles the use_dill parameters in kwargs, that would otherwise raise an error when processes > 1. Any thread in the returned multiprocessing pool (e.g. processes > 1) also default

The requirement of schwimmbad relies on the master branch (as specified in requirements.txt). The ‘use_dill’ functionality can raise if not following the requirement specified.

Choose between the different pools given options from, e.g., argparse.

mpi : bool, optional

Use the MPI processing pool, MPIPool. By default, False, will use the SerialPool.

processes : int, optional

Use the multiprocessing pool, MultiPool, with this number of processes. By default, processes=1, will use them:class:~schwimmbad.serial.SerialPool.

Any additional kwargs are passed in to the pool class initializer selected by the arguments.

Module contents