lenstronomy.Sampling.Pool package

Submodules

lenstronomy.Sampling.Pool.multiprocessing module

this file is taken from schwimmbad (https://github.com/adrn/schwimmbad) and an explicit fork by Aymeric Galan to replace the multiprocessing with the multiprocess dependence as for multi-threading, multiprocessing is not supporting dill (only pickle) which is required.

The class also extends with a is_master() definition

class MultiPool(processes=None, initializer=None, initargs=(), **kwargs)

Bases: Pool

A modified version of multiprocessing.pool.Pool that has better behavior with regard to KeyboardInterrupts in the map() method.

(Original author: Peter K. G. Williams)

wait_timeout = 3600

__init__(processes=None, initializer=None, initargs=(), **kwargs)

Parameters:

• processes (int, optional) – The number of worker processes to use; defaults to the number of CPUs.

• initializer (callable, optional) – If specified, a callable that will be invoked by each worker process when it starts.

• initargs (iterable, optional) – Arguments for initializer; it will be called as initializer(*initargs).

• kwargs – Extra arguments passed to the multiprocessing.pool.Pool superclass.

is_master()

is_worker()

static enabled()

map(func, iterable, chunksize=None, callback=None)

Equivalent to the built-in map() function and multiprocessing.pool.Pool.map(), without catching KeyboardInterrupt.

Parameters:

• func (callable) – A function or callable object that is executed on each element of the specified tasks iterable. This object must be picklable (i.e. it can’t be a function scoped within a function or a lambda function). This should accept a single positional argument and return a single object.

• iterable (iterable) – A list or iterable of tasks. Each task can be itself an iterable (e.g., tuple) of values or data to pass in to the worker function.

• callback (callable, optional) – An optional callback function (or callable) that is called with the result from each worker run and is executed on the master process. This is useful for, e.g., saving results to a file, since the callback is only called on the master thread.

Returns:

A list of results from the output of each worker() call.

lenstronomy.Sampling.Pool.pool module

this file is taken from schwimmbad (https://github.com/adrn/schwimmbad) and an explicit fork by Aymeric Galan to replace the multiprocessing with the multiprocess dependence as for multi-threading, multiprocessing is not supporting dill (only pickle) which is required.

Tests show that the MPI mode works with Python 3.7.2 but not with Python 3.7.0 on a specific system due to mpi4py dependencies and configurations.

Contributions by: - Peter K. G. Williams - Júlio Hoffimann Mendes - Dan Foreman-Mackey - Aymeric Galan - Simon Birrer

Implementations of four different types of processing pools:

• MPIPool: An MPI pool.

• MultiPool: A multiprocessing for local parallelization.

• SerialPool: A serial pool, which uses the built-in map function

choose_pool(mpi=False, processes=1, **kwargs)

Extends the capabilities of the schwimmbad.choose_pool method.

It handles the use_dill parameters in kwargs, that would otherwise raise an error when processes > 1. Any thread in the returned multiprocessing pool (e.g. processes > 1) also default

The requirement of schwimmbad relies on the master branch (as specified in requirements.txt). The ‘use_dill’ functionality can raise if not following the requirement specified.

Choose between the different pools given options from, e.g., argparse.

Parameters:

• mpi (bool, optional) – Use the MPI processing pool, MPIPool. By default, False, will use the SerialPool.

• processes (int, optional) – Use the multiprocessing pool, MultiPool, with this number of processes. By default, processes=1, will use them:class:~schwimmbad.serial.SerialPool.

• kwargs (keyword arguments) – Any additional kwargs are passed in to the pool class initializer selected by the arguments.

Module contents