Lib Multicore

Multi-core related objects.

idpconfgen.libs.libmulticore.consume_iterable_in_list(func, *args, **kwargs)

Consumes a generator to a list.

Used as a wrapper to send generators to Python multiprocessing lib.

Returns:

list of length N – Where N is the number of times func yields.

idpconfgen.libs.libmulticore.flat_results_from_chunk(execute, func, *args, **kwargs)

Flattens a result coming from consume_iterable_in_list execution.

Parameters:

• execute (callable) – The prepared multiprocessing function. Each item of its iteration should be iterable of iterable.

• func (callable) – The function to process each yielded result from the results in the multiprocessing fragment.

idpconfgen.libs.libmulticore.pool_function(func, items, method='imap_unordered', ncores=1)

Multiprocess Pools a function.

Parameters:

• func (callable) – The function to execute along the iterable items. If this function expects additional args and `kwargs, prepare it previously using functools.partial.

• items (interable) – Elements to pass to the func.

• method (str) – The Pool method to execute. Defaults to imap_unordered.

• ncores (int) – The number of cores to use. Defaults to 1.

idpconfgen.libs.libmulticore.pool_function_in_chunks(func, items, chunks=5000, **kwargs)

Execute func in chunks of items using Pool.

Yields the results after each fragment.

Parameters:

• func (callable) – The function to execute over items. If this function expects additional args and `kwargs, prepare it previously using functools.partial.

• items (iterable) – The items to process by the funct.

• *args (any) – Additional positional arguments to send to func.

• chunks (int) – The size of each fragment processed multiprocessing before yielding.

• **kwargs (any) – Additional keyword arguments to send to func.

Yields:

list – Containing the results after each fragment.

idpconfgen.libs.libmulticore.starunpack(func, *args, **kwargs)

Unpack first argument of args.