ž
¨ÿ fu[ c @ sf d g Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l m Z m Z m Z d d l
m Z m Z d Z
d Z d Z e j ƒ Z d d „ Z d d
„ Z Gd d „ d e ƒ Z e f e d
d „ Z Gd d „ d e ƒ Z Gd d „ d e ƒ Z e Z Gd d „ d e ƒ Z Gd d „ d e ƒ Z Gd d „ d e ƒ Z Gd d „ d e ƒ Z d S( u Pooli N( u Processu cpu_countu TimeoutError( u Finalizeu debugi i c C s t t | Œ ƒ S( N( u listu map( u args( ( u9 /opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu mapstar' s u mapstarc C s t t j | d | d ƒ ƒ S( Ni i ( u listu itertoolsu starmap( u args( ( u9 /opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu starmapstar* s u starmapstarc sD | Ee Z d Z d Z ‡ f d d † Z d d „ Z d d „ Z ‡ S( u MaybeEncodingErroruV Wraps possible unpickleable errors, so they can be
safely sent through the socket.c sA t | ƒ | _ t | ƒ | _ t t | ƒ j | j | j ƒ d S( N( u repru excu valueu superu MaybeEncodingErroru __init__( u selfu excu value( u __class__( u9 /opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu __init__5 s u MaybeEncodingError.__init__c C s d | j | j f S( Nu( Error sending result: '%s'. Reason: '%s'( u valueu exc( u self( ( u9 /opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu __str__: s u MaybeEncodingError.__str__c C s d t | ƒ S( Nu ( u str( u self( ( u9 /opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu __repr__> s u MaybeEncodingError.__repr__( u __name__u
__module__u __qualname__u __doc__u __init__u __str__u __repr__( u
__locals__( ( u __class__u9 /opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu MaybeEncodingError1 s u MaybeEncodingErrorc ) C sÆ | j } | j } t | d ƒ r> | j j ƒ | j j ƒ n | d k rW | | Œ n d } xT| d k s~ | r³| | k r³y
| ƒ } Wn# t t f k
r° t d ƒ PYn X| d k rË t d ƒ Pn | \ } }
} } }
y d | | |
Ž f } Wn1 t k
r,} z d | f } WYd d } ~ Xn Xy | | |
| f ƒ Wn_ t k
r¥} z? t
| | d ƒ } t d | ƒ | | |
d | f f ƒ WYd d } ~ Xn X| d 7} q` Wt d | ƒ d S(
Nu _writeri u) worker got EOFError or IOError -- exitingu worker got sentinel -- exitingi u0 Possible encoding error while sending result: %su worker exiting after %d tasksTF( u putu getu hasattru _writeru closeu _readeru Noneu EOFErroru IOErroru debugu Trueu Exceptionu Falseu MaybeEncodingError( u inqueueu outqueueu initializeru initargsu maxtasksu putu getu completedu tasku jobu iu funcu argsu kwdsu resultu eu wrapped( ( u9 /opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu workerB s>
!
,u workerc B sÁ | Ee Z d Z d Z e Z d7 d7 f d7 d d „ Z d d „ Z d d „ Z d d „ Z d
d „ Z
f i d d
„ Z d7 d d „ Z d7 d d „ Z
d7 d7 d7 d d „ Z d d d „ Z d d d „ Z f i d7 d7 d d „ Z d7 d7 d7 d d „ Z d7 d7 d7 d d „ Z e d d „ ƒ Z e d! d" „ ƒ Z e d# d$ „ ƒ Z e d% d& „ ƒ Z d' d( „ Z d) d* „ Z d+ d, „ Z d- d. „ Z e d/ d0 „ ƒ Z e d1 d2 „ ƒ Z d3 d4 „ Z d5 d6 „ Z! d7 S(8 u PooluS
Class which supports an async version of applying functions to arguments.
c C s- | j ƒ t j ƒ | _ i | _ t | _ | | _ | | _ | | _ | d k r} y
t ƒ } Wq} t k
ry d } Yq} Xn | d k r˜ t
d ƒ ‚ n | d k rÀ t | ƒ rÀ t d ƒ ‚ n | | _ g | _ | j ƒ t j d t j d | f ƒ | _ d | j _ t | j _ | j j ƒ t j d t j d | j | j | j | j | j f ƒ | _ d | j _ t | j _ | j j ƒ t j d t j d | j | j | j f ƒ | _! d | j! _ t | j! _ | j! j ƒ t" | | j# d | j | j$ | j | j | j | j | j! | j f d d ƒ| _% d S( Ni u& Number of processes must be at least 1u initializer must be a callableu targetu argsu exitpriorityi T(& u
_setup_queuesu queueu Queueu
_taskqueueu _cacheu RUNu _stateu _maxtasksperchildu _initializeru _initargsu Noneu cpu_countu NotImplementedErroru
ValueErroru callableu TypeErroru
_processesu _poolu _repopulate_poolu threadingu Threadu Poolu _handle_workersu _worker_handleru Trueu daemonu startu
_handle_tasksu
_quick_putu _outqueueu
_task_handleru _handle_resultsu
_quick_getu _result_handleru Finalizeu _terminate_poolu _inqueueu
_terminate( u selfu processesu initializeru initargsu maxtasksperchild( ( u9 /opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu __init__r sX
u
Pool.__init__c C sw d } xj t t t | j ƒ ƒ ƒ D]M } | j | } | j d k r" t d | ƒ | j ƒ d } | j | =q" q" W| S( u Cleanup after any worker processes which have exited due to reaching
their specified lifetime. Returns True if any workers were cleaned up.
u cleaning up worker %dFNT(
u Falseu reversedu rangeu lenu _poolu exitcodeu Noneu debugu joinu True( u selfu cleanedu iu worker( ( u9 /opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu _join_exited_workers s "
u Pool._join_exited_workersc
C s¦ xŸ t | j t | j ƒ ƒ D] } | j d t d | j | j | j | j | j
f ƒ } | j j | ƒ | j j
d d ƒ | _ d | _ | j ƒ t d ƒ q Wd S( u€ Bring the number of pool processes up to the specified number,
for use after reaping workers which have exited.
u targetu argsu Processu
PoolWorkeru added workerNT( u rangeu
_processesu lenu _poolu Processu workeru _inqueueu _outqueueu _initializeru _initargsu _maxtasksperchildu appendu nameu replaceu Trueu daemonu startu debug( u selfu iu w( ( u9 /opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu _repopulate_pool¼ s #
u Pool._repopulate_poolc C s | j ƒ r | j ƒ n d S( uE Clean up any exited workers and start replacements for them.
N( u _join_exited_workersu _repopulate_pool( u self( ( u9 /opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu _maintain_poolÌ s u Pool._maintain_poolc C sP d d l m } | ƒ | _ | ƒ | _ | j j j | _ | j j j | _ d S( Ni ( u SimpleQueue(
u queuesu SimpleQueueu _inqueueu _outqueueu _writeru sendu
_quick_putu _readeru recvu
_quick_get( u selfu SimpleQueue( ( u9 /opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu
_setup_queuesÒ s
u Pool._setup_queuesc C s | j | | | ƒ j ƒ S( u6
Equivalent of `func(*args, **kwds)`.
( u apply_asyncu get( u selfu funcu argsu kwds( ( u9 /opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu applyÙ s u
Pool.applyc C s | j | | t | ƒ j ƒ S( ux
Apply `func` to each element in `iterable`, collecting the results
in a list that is returned.
( u
_map_asyncu mapstaru get( u selfu funcu iterableu chunksize( ( u9 /opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu mapà s u Pool.mapc C s | j | | t | ƒ j ƒ S( uÌ
Like `map()` method but the elements of the `iterable` are expected to
be iterables as well and will be unpacked as arguments. Hence
`func` and (a, b) becomes func(a, b).
( u
_map_asyncu starmapstaru get( u selfu funcu iterableu chunksize( ( u9 /opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu starmapç s u Pool.starmapc C s | j | | t | | | ƒ S( u=
Asynchronous version of `starmap()` method.
( u
_map_asyncu starmapstar( u selfu funcu iterableu chunksizeu callbacku error_callback( ( u9 /opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu
starmap_asyncï s u Pool.starmap_asynci c sÝ | j t k r t d ƒ ‚ n | d k rr t | j ƒ ‰ | j j ‡ ‡ f d d † t | ƒ Dƒ ˆ j f ƒ ˆ St j
ˆ | | ƒ } t | j ƒ ‰ | j j ‡ f d d † t | ƒ Dƒ ˆ j f ƒ d d „ ˆ Dƒ Sd S( uP
Equivalent of `map()` -- can be MUCH slower than `Pool.map()`.
u Pool not runningi c 3 s0 | ]& \ } } ˆ j | ˆ | f i f Vq d S( N( u _job( u .0u iu x( u funcu result( u9 /opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu ÿ s u Pool.imap..c 3 s0 | ]& \ } } ˆ j | t | f i f Vq d S( N( u _jobu mapstar( u .0u iu x( u result( u9 /opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu s c s s" | ] } | D] } | Vq
q d S( N( ( u .0u chunku item( ( u9 /opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu s N( u _stateu RUNu
ValueErroru IMapIteratoru _cacheu
_taskqueueu putu enumerateu _set_lengthu Poolu
_get_tasks( u selfu funcu iterableu chunksizeu task_batches( ( u funcu resultu9 /opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu imap÷ s u Pool.imapc sÝ | j t k r t d ƒ ‚ n | d k rr t | j ƒ ‰ | j j ‡ ‡ f d d † t | ƒ Dƒ ˆ j f ƒ ˆ St j
ˆ | | ƒ } t | j ƒ ‰ | j j ‡ f d d † t | ƒ Dƒ ˆ j f ƒ d d „ ˆ Dƒ Sd S( uL
Like `imap()` method but ordering of results is arbitrary.
u Pool not runningi c 3 s0 | ]& \ } } ˆ j | ˆ | f i f Vq d S( N( u _job( u .0u iu x( u funcu result( u9 /opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu s u& Pool.imap_unordered..c 3 s0 | ]& \ } } ˆ j | t | f i f Vq d S( N( u _jobu mapstar( u .0u iu x( u result( u9 /opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu s c s s" | ] } | D] } | Vq
q d S( N( ( u .0u chunku item( ( u9 /opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu s N( u _stateu RUNu
ValueErroru IMapUnorderedIteratoru _cacheu
_taskqueueu putu enumerateu _set_lengthu Poolu
_get_tasks( u selfu funcu iterableu chunksizeu task_batches( ( u funcu resultu9 /opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu imap_unordered
s u Pool.imap_unorderedc C sb | j t k r t d ƒ ‚ n t | j | | ƒ } | j j | j d | | | f g d f ƒ | S( u;
Asynchronous version of `apply()` method.
u Pool not runningN( u _stateu RUNu
ValueErroru ApplyResultu _cacheu
_taskqueueu putu _jobu None( u selfu funcu argsu kwdsu callbacku error_callbacku result( ( u9 /opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu apply_async s
+u Pool.apply_asyncc C s | j | | t | | | ƒ S( u9
Asynchronous version of `map()` method.
( u
_map_asyncu mapstar( u selfu funcu iterableu chunksizeu callbacku error_callback( ( u9 /opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu map_async( s u Pool.map_asyncc s | j t k r t d ƒ ‚ n t | d ƒ s< t | ƒ } n | d k r† t t | ƒ t | j ƒ d ƒ \ } } | r† | d 7} q† n t | ƒ d k r¡ d } n t j
| | | ƒ } t | j | t | ƒ | d | ƒ‰ | j
j ‡ ‡ f d d † t | ƒ Dƒ d f ƒ ˆ S(
uY
Helper function to implement map, starmap and their async counterparts.
u Pool not runningu __len__i i i u error_callbackc 3 s0 | ]& \ } } ˆ j | ˆ | f i f Vq d S( N( u _job( u .0u iu x( u mapperu result( u9 /opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu D s u" Pool._map_async..N( u _stateu RUNu
ValueErroru hasattru listu Noneu divmodu lenu _poolu Poolu
_get_tasksu MapResultu _cacheu
_taskqueueu putu enumerate( u selfu funcu iterableu mapperu chunksizeu callbacku error_callbacku extrau task_batches( ( u mapperu resultu9 /opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu
_map_async0 s ( u Pool._map_asyncc C so t j ƒ } xB | j t k s6 | j rP | j t k rP | j ƒ t j d ƒ q W| j j
d ƒ t d ƒ d S( Ngš™™™™™¹?u worker handler exiting(
u threadingu current_threadu _stateu RUNu _cacheu TERMINATEu _maintain_poolu timeu sleepu
_taskqueueu putu Noneu debug( u poolu thread( ( u9 /opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu _handle_workersH s *
u Pool._handle_workersc C s„ t j ƒ } xt | j d ƒ D]ê \ } } d
} xÔ t | ƒ D]¢ \ } } | j ra t d ƒ Pn y | | ƒ Wq> t k
rß }
zN | d d … \ } } y | | j | d |
f ƒ Wn t k
rÌ Yn XWYd d }
~
Xq> Xq> W| r t d ƒ | | d ƒ q q Pq Wt d ƒ y@ t d ƒ | j d ƒ t d ƒ x | D] }
| d ƒ qBWWn t
k
rut d ƒ Yn Xt d ƒ d S( Ni u' task handler found thread._state != RUNi u doing set_length()u task handler got sentinelu/ task handler sending sentinel to result handleru( task handler sending sentinel to workersu/ task handler got IOError when sending sentinelsu task handler exitingiÿÿÿÿF( u threadingu current_threadu iteru getu Noneu enumerateu _stateu debugu Exceptionu _setu Falseu KeyErroru putu IOError( u taskqueueu putu outqueueu poolu cacheu threadu tasksequ
set_lengthu iu tasku eu jobu indu p( ( u9 /opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu
_handle_tasksU s<
u Pool._handle_tasksc C sè t j ƒ } x¦ y
| ƒ } Wn& t t f k
rD t d ƒ d SYn X| j r\ t d ƒ Pn | d k rv t d ƒ Pn | \ } } } y | | j | | ƒ Wq t k
r± Yq Xq x§ | r^| j t k r^y
| ƒ } Wn& t t f k
rt d ƒ d SYn X| d k r t d ƒ q¸ n | \ } } } y | | j | | ƒ Wq¸ t k
rZYq¸ Xq¸ Wt
| d ƒ rËt d ƒ y5 x. t d ƒ D] } | j j
ƒ s¡Pn | ƒ qˆWWqËt t f k
rÇYqËXn t d t | ƒ | j ƒ d S( Nu. result handler got EOFError/IOError -- exitingu, result handler found thread._state=TERMINATEu result handler got sentinelu& result handler ignoring extra sentinelu _readeru" ensuring that outqueue is not fulli
u7 result handler exiting: len(cache)=%s, thread._state=%s( u threadingu current_threadu IOErroru EOFErroru debugu _stateu Noneu _setu KeyErroru TERMINATEu hasattru rangeu _readeru pollu len( u outqueueu getu cacheu threadu tasku jobu iu obj( ( u9 /opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu _handle_results sV
u Pool._handle_resultsc c sC t | ƒ } x0 t t j | | ƒ ƒ } | s1 d S| | f Vq d S( N( u iteru tupleu itertoolsu islice( u funcu itu sizeu x( ( u9 /opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu
_get_tasks¹ s u Pool._get_tasksc C s t d ƒ ‚ d S( Nu: pool objects cannot be passed between processes or pickled( u NotImplementedError( u self( ( u9 /opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu
__reduce__Â s u Pool.__reduce__c C s5 t d ƒ | j t k r1 t | _ t | j _ n d S( Nu closing pool( u debugu _stateu RUNu CLOSEu _worker_handler( u self( ( u9 /opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu closeÇ s
u
Pool.closec C s- t d ƒ t | _ t | j _ | j ƒ d S( Nu terminating pool( u debugu TERMINATEu _stateu _worker_handleru
_terminate( u self( ( u9 /opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu terminateÍ s
u Pool.terminatec C sS t d ƒ | j j ƒ | j j ƒ | j j ƒ x | j D] } | j ƒ q; Wd S( Nu joining pool( u debugu _worker_handleru joinu
_task_handleru _result_handleru _pool( u selfu p( ( u9 /opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu joinÓ s
u Pool.joinc C sW t d ƒ | j j ƒ x9 | j ƒ rR | j j ƒ rR | j j ƒ t j d ƒ q Wd S( Nu7 removing tasks from inqueue until task handler finishedi ( u debugu _rlocku acquireu is_aliveu _readeru pollu recvu timeu sleep( u inqueueu task_handleru size( ( u9 /opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu _help_stuff_finishÜ s
u Pool._help_stuff_finishc
C sˆ t d ƒ t | _ t | _ t d ƒ | j | | t | ƒ ƒ t | _ | j d ƒ t d ƒ t j ƒ | k r~ | j ƒ n | rÑ t
| d d ƒ rÑ t d ƒ x- | D]" } | j d k r¨ | j ƒ q¨ q¨ Wn t d ƒ t j ƒ | k rú | j ƒ n t d ƒ t j ƒ | k r#| j ƒ n | r„t
| d d ƒ r„t d ƒ x; | D]0 } | j
ƒ rMt d
| j ƒ | j ƒ qMqMWn d S( Nu finalizing poolu& helping task handler/workers to finishu joining worker handleri u terminateu terminating workersu joining task handleru joining result handleru joining pool workersu cleaning up worker %d( u debugu TERMINATEu _stateu _help_stuff_finishu lenu putu Noneu threadingu current_threadu joinu hasattru exitcodeu terminateu is_aliveu pid(
u clsu taskqueueu inqueueu outqueueu poolu worker_handleru task_handleru result_handleru cacheu p( ( u9 /opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu _terminate_poolå s6
u Pool._terminate_poolc C s | S( N( ( u self( ( u9 /opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu __enter__ s u Pool.__enter__c C s | j ƒ d S( N( u terminate( u selfu exc_typeu exc_valu exc_tb( ( u9 /opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu __exit__ s u
Pool.__exit__N(" u __name__u
__module__u __qualname__u __doc__u Processu Noneu __init__u _join_exited_workersu _repopulate_poolu _maintain_poolu
_setup_queuesu applyu mapu starmapu
starmap_asyncu imapu imap_unorderedu apply_asyncu map_asyncu
_map_asyncu staticmethodu _handle_workersu
_handle_tasksu _handle_resultsu
_get_tasksu
__reduce__u closeu terminateu joinu _help_stuff_finishu classmethodu _terminate_poolu __enter__u __exit__( u
__locals__( ( u9 /opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu Pooll sB :
*: . c B sb | Ee Z d Z d d „ Z d d „ Z d d „ Z d
d d „ Z d
d d
„ Z d d „ Z d
S( u ApplyResultc C sJ t j ƒ | _ t t ƒ | _ | | _ | | _ | | _ | | | j
s. *ÿ ²&%I