lenstronomy.Sampling.Pool package¶
Submodules¶
lenstronomy.Sampling.Pool.multiprocessing module¶
this file is taken from schwimmbad (https://github.com/adrn/schwimmbad) and an explicit fork by Aymeric Galan to replace the multiprocessing with the multiprocess dependence as for multi-threading, multiprocessing is not supporting dill (only pickle) which is required.
The class also extends with a is_master()
definition
-
class
MultiPool
(processes=None, initializer=None, initargs=(), **kwargs)[source]¶ Bases:
multiprocess.pool.Pool
A modified version of
multiprocessing.pool.Pool
that has better behavior with regard toKeyboardInterrupts
in themap()
method.(Original author: Peter K. G. Williams)
-
__init__
(processes=None, initializer=None, initargs=(), **kwargs)[source]¶ Parameters: - processes (int, optional) – The number of worker processes to use; defaults to the number of CPUs.
- initializer (callable, optional) – If specified, a callable that will be invoked by each worker process when it starts.
- initargs (iterable, optional) – Arguments for
initializer
; it will be called asinitializer(*initargs)
. - kwargs – Extra arguments passed to the
multiprocessing.pool.Pool
superclass.
-
map
(func, iterable, chunksize=None, callback=None)[source]¶ Equivalent to the built-in
map()
function andmultiprocessing.pool.Pool.map()
, without catchingKeyboardInterrupt
.Parameters: - func (callable) – A function or callable object that is executed on each element of
the specified
tasks
iterable. This object must be picklable (i.e. it can’t be a function scoped within a function or alambda
function). This should accept a single positional argument and return a single object. - iterable (iterable) – A list or iterable of tasks. Each task can be itself an iterable (e.g., tuple) of values or data to pass in to the worker function.
- callback (callable, optional) – An optional callback function (or callable) that is called with the result from each worker run and is executed on the master process. This is useful for, e.g., saving results to a file, since the callback is only called on the master thread.
Returns: A list of results from the output of each
worker()
call.- func (callable) – A function or callable object that is executed on each element of
the specified
-
wait_timeout
= 3600¶
-
lenstronomy.Sampling.Pool.pool module¶
this file is taken from schwimmbad (https://github.com/adrn/schwimmbad) and an explicit fork by Aymeric Galan to replace the multiprocessing with the multiprocess dependence as for multi-threading, multiprocessing is not supporting dill (only pickle) which is required.
Tests show that the MPI mode works with Python 3.7.2 but not with Python 3.7.0 on a specific system due to mpi4py dependencies and configurations.
Contributions by: - Peter K. G. Williams - Júlio Hoffimann Mendes - Dan Foreman-Mackey - Aymeric Galan - Simon Birrer
Implementations of four different types of processing pools:
- MPIPool: An MPI pool.
- MultiPool: A multiprocessing for local parallelization.
- SerialPool: A serial pool, which uses the built-in map function
-
choose_pool
(mpi=False, processes=1, **kwargs)[source]¶ Extends the capabilities of the schwimmbad.choose_pool method.
It handles the use_dill parameters in kwargs, that would otherwise raise an error when processes > 1. Any thread in the returned multiprocessing pool (e.g. processes > 1) also default
The requirement of schwimmbad relies on the master branch (as specified in requirements.txt). The ‘use_dill’ functionality can raise if not following the requirement specified.
Choose between the different pools given options from, e.g., argparse.
Parameters: - mpi (bool, optional) – Use the MPI processing pool,
MPIPool
. By default,False
, will use theSerialPool
. - processes (int, optional) – Use the multiprocessing pool,
MultiPool
, with this number of processes. By default,processes=1
, will use them:class:~schwimmbad.serial.SerialPool. - kwargs (keyword arguments) – Any additional kwargs are passed in to the pool class initializer selected by the arguments.
- mpi (bool, optional) – Use the MPI processing pool,