ReL
@ s d Z d d g Z d d l m Z d d l m Z d d l m Z d d l m
Z
d d l m Z d d l
m Z d d l m Z e r d d
l m Z m Z m Z m Z m Z d d l m Z e e j e j f Z e d Z e d
Z y d d l Z Wn e k
rd Z Yn Xd Z d Z e d d Z d d d Z d d d Z d d d Z! e spe r}e Z" Z# n e Z" e! Z# d S)ab Convenient parallelization of higher order functions.
This module provides two helper functions, with appropriate fallbacks on
Python 2 and on systems lacking support for synchronization mechanisms:
- map_multiprocess
- map_multithread
These helpers work like Python 3's map, with two differences:
- They don't guarantee the order of processing of
the elements of the iterable.
- The underlying process/thread pools chop the iterable into
a number of chunks, so that for very long iterables using
a large value for chunksize can make the job complete much faster
than using the default value of 1.
map_multiprocessmap_multithread )contextmanager)Pool)DEFAULT_POOLSIZE)PY2)map)MYPY_CHECK_RUNNING)CallableIterableIteratorUnionTypeVar)poolSTNTFi c
c s/ z | VWd | j | j | j Xd S)z>Return a context manager making sure the pool closes properly.N)closejoin terminate)r r /builddir/build/BUILDROOT/alt-python35-pip-20.2.4-5.el8.x86_64/opt/alt/python35/lib/python3.5/site-packages/pip/_internal/utils/parallel.pyclosing4 s
r c C s
t | | S)zMake an iterator applying func to each element in iterable.
This function is the sequential fallback either on Python 2
where Pool.imap* doesn't react to KeyboardInterrupt
or when sem_open is unavailable.
)r )funciterable chunksizer r r
_map_fallbackB s r c C s0 t t } | j | | | SWd QRXd S)zChop iterable into chunks and submit them to a process pool.
For very long iterables using a large value for chunksize can make
the job complete much faster than using the default value of 1.
Return an unordered iterator of the results.
N)r ProcessPoolimap_unordered)r r r r r r r _map_multiprocessM s r c C s3 t t t } | j | | | SWd QRXd S)zChop iterable into chunks and submit them to a thread pool.
For very long iterables using a large value for chunksize can make
the job complete much faster than using the default value of 1.
Return an unordered iterator of the results.
N)r
ThreadPoolr r )r r r r r r r _map_multithreadZ s r! )$__doc____all__
contextlibr multiprocessingr r Zmultiprocessing.dummyr Z pip._vendor.requests.adaptersr Zpip._vendor.sixr pip._vendor.six.movesr pip._internal.utils.typingr typingr
r r r
r r r r Zmultiprocessing.synchronize ImportErrorZ
LACK_SEM_OPENTIMEOUTr r r r! r r r r r r s8 (