Yf{ @ s d d d d g Z d d l Z d d l Z d d l Z d d l Z d d l m Z d d l m Z d d l m
Z
d d
l m Z d d l m Z d d l m
Z
d d
l m Z d d l m Z d d l m Z d d Z e j e j e d d d d d f D Z e d e k rJd d Z x e D] Z e j e e q0WGd d d e Z f i d d Z d d Z Gd d d e Z d d! Z d" d# Z Gd$ d% d% e Z Gd& d' d' e Z d( e
j! e
j" f d) e
j# e
j$ f i Z% Gd* d d e Z& Gd+ d, d, e' Z( Gd- d d e Z) d. d/ Z* i d0 d1 Z+ d d d d2 d3 d4 Z, Gd5 d6 d6 e Z- Gd7 d8 d8 e Z. d2 d9 d: Z/ Gd; d< d< e) Z0 Gd= d> d> e) Z1 Gd? d@ d@ e1 Z2 GdA dB dB e) Z3 GdC dD dD e) Z4 GdE dF dF e) Z5 GdG dH dH e) Z6 e+ dI dJ dK dL dM dN dO dP dQ dR dS dT dU dV dW dX dY dZ d[ d\ f Z7 Gd] d^ d^ e7 Z8 e+ d_ dK dL dM dN dR d` da db dc d d dX dd de df d f Z9 e+ dg dN dM dR f Z: e+ dh di dj dk dl dm dn do dp dq dr ds f Z; dj dt dp dt dr dt dl du dm du i e; _< Gdv dh dh e; Z= Gdw d d e& Z> e> j dx e j? e> j dy e j? e> j dz e j@ e3 e> j d{ e jA e1 e> j d| e jB e1 e> j d} e jC e1 e> j d~ e jD e1 e> j d e jE e2 e> j d e jF e4 e> j d e jG e= e> j d e e8 e> j d eH e9 e> j d8 e. e6 e> j d: e/ e: e> j d6 e- e5 e> j du d e0 d d e> j dt d d d S) BaseManager SyncManager BaseProxyToken N)time)
format_exc )
connection)context)pool)process) reduction)util) get_contextc C s t j | j | j f f S)N)arraytypecodetobytes)a r =/opt/alt/python35/lib64/python3.5/multiprocessing/managers.py reduce_array% s r c C s( g | ] } t t i | q Sr )typegetattr).0namer r r
) s r itemskeysvaluesc C s t t | f f S)N)list)objr r r rebuild_as_list+ s r! c @ sL e Z d Z d Z d Z d d Z d d Z d d
Z d d Z d
S)r z4
Type to uniquely indentify a shared object
typeidaddressidc C s! | | | | _ | _ | _ d S)N)r" r# r$ )selfr" r# r$ r r r __init__: s zToken.__init__c C s | j | j | j f S)N)r" r# r$ )r% r r r __getstate__= s zToken.__getstate__c C s | \ | _ | _ | _ d S)N)r" r# r$ )r% stater r r __setstate__@ s zToken.__setstate__c C s# d | j j | j | j | j f S)Nz %s(typeid=%r, address=%r, id=%r)) __class____name__r" r# r$ )r% r r r __repr__C s zToken.__repr__N)r" r# r$ ) r+
__module__ __qualname____doc__ __slots__r& r' r) r, r r r r r 4 s c C sN | j | | | | f | j \ } } | d k r; | St | | d S)zL
Send a message to manager using connection `c` and return response
z#RETURNN)sendrecvconvert_to_error)cr$
methodnameargskwdskindresultr r r dispatchK s
r: c C s~ | d k r | S| d k r> t | t k s4 t t | S| d k rp t | t k sb t t d | St d Sd S)Nz#ERRORz
#TRACEBACKz#UNSERIALIZABLEzUnserializable message: %s
zUnrecognized message type)r strAssertionError RemoteError
ValueError)r8 r9 r r r r3 U s
r3 c @ s e Z d Z d d Z d S)r= c C s) d d d d t | j d d d S)N
-K r )r; r6 )r% r r r __str__b s zRemoteError.__str__N)r+ r- r. rB r r r r r= a s r= c C sI g } x< t | D]. } t | | } t | r | j | q W| S)z4
Return a list of names of methods of `obj`
)dirr callableappend)r tempr funcr r r all_methodsi s rH c C s d d t | D S)zP
Return a list of names of methods of `obj` which do not start with '_'
c S s& g | ] } | d d k r | q S)r _r )r r r r r r x s z"public_methods..)rH )r r r r public_methodst s rJ c @ s e Z d Z d Z d d d d d d d d d
g Z d d Z d
d Z d d Z d d Z d d Z d d Z
d d Z d d Z d e d e d e
i Z
d d Z d d! Z d" d# Z d$ d% Z d& d' Z d( d) Z d* d+ Z d, d- Z d. d/ Z d0 S)1ServerzM
Server class which runs in a process controlled by a manager object
shutdowncreateaccept_connection get_methods
debug_infonumber_of_objectsdummyincrefdecrefc C s t | t s t | | _ t j | | _ t | \ } } | d | d d | _ | j j | _ d d f f i | _
i | _ t j
| _ d S)Nr# Zbacklog 0)
isinstancebytesr< registryr AuthenticationStringauthkeylistener_clientlistenerr# id_to_objid_to_refcount threadingRLockmutex)r% rY r# r[
serializerListenerClientr r r r& s zServer.__init__c C s t j | _ | t j _ zq t j d | j } d | _ | j y* x# | j j
sq | j j d qO WWn t t
f k
r Yn XWd t j t j k r t j d t j t _ t j t _ t j d Xd S)z(
Run the server forever
targetTr Nzresetting stdout, stderrr )r` Event
stop_eventr current_process_manager_serverThreadaccepterdaemonstartis_setwaitKeyboardInterrupt
SystemExitsysstdout
__stdout__r debug
__stderr__stderrexit)r% rl r r r
serve_forever s
zServer.serve_foreverc
C sg x` y | j j } Wn t k
r- w Yn Xt j d | j d | f } d | _ | j q Wd S)Nrf r6 T)r] ZacceptOSErrorr` rk handle_requestrm rn )r% r4 tr r r rl s
zServer.accepterc ' C s d } } } yv t j | | j t j | | j | j } | \ } } } } | | j k st t d | t | | } Wn! t k
r d t f } YnG Xy | | | | } Wn! t k
r d t f } Yn
Xd | f } y | j
| Wn t k
r}
zd y | j
d t f Wn t k
rBYn Xt j d | t j d | t j d |
WYd d }
~
Xn X| j
d S)z)
Handle a new connection
Nz%r unrecognizedz
#TRACEBACKz#RETURNzFailure to send message: %rz ... request was %rz ... exception was %r)r Zdeliver_challenger[ Zanswer_challenger2 publicr< r Exceptionr r1 r infoclose) r% r4 funcnamer9 requestignorer6 r7 rG msger r r r| s4
#zServer.handle_requestc * C s% t j d t j j | j } | j } | j } x| j j s y,d } } | } | \ } } } }
| | \ } } } | | k r t
d | t | | f t | | }
y |
| |
} Wn1 t
k
r} z d | f } WYd d } ~ Xnp X| o| j | d } | re| j | | | \ } } t | | j | } d | | f f } n d | f } Wn t
k
r | d k rd t f } n\ y8 | j | } | | | | | | |
} d | f } Wn! t
k
rd t f } Yn XYnX t k
r7t j d t j j t j d Yn! t
k
rWd t f } Yn XyM y | | Wn8 t
k
r} z | d
t f WYd d } ~ Xn XWq7 t
k
r } zU t j d t j j t j d | t j d
| | j t j d WYd d } ~ Xq7 Xq7 Wd S)zQ
Handle requests from the proxies in a particular process/thread
z$starting server thread to service %rNz+method %r of %r object is not in exposed=%rz#ERRORz#PROXYz#RETURNz
#TRACEBACKz$got EOF -- exiting thread serving %rr z#UNSERIALIZABLEz exception in thread serving %rz ... message was %rz ... exception was %rr )r rv r` current_threadr r2 r1 r^ rh ro AttributeErrorr r r getrM r r# r fallback_mappingEOFErrorrs ry r r )r% connr2 r1 r^ r5 r r identr6 r7 exposedZ gettypeidZfunctionresr r r" ZridentZrexposedtokenZ
fallback_funcr9 r r r serve_client sl
*
zServer.serve_clientc C s | S)Nr )r% r r r r r r fallback_getvalue s zServer.fallback_getvaluec C s
t | S)N)r; )r% r r r r r r fallback_str" s zServer.fallback_strc C s
t | S)N)repr)r% r r r r r r
fallback_repr% s zServer.fallback_reprrB r, z #GETVALUEc C s d S)Nr )r% r4 r r r rR . s z Server.dummyc C s | j g } t | j j } | j xV | D]N } | d k r6 | j d | | j | t | j | d d d f q6 Wd j | SWd QRXd S)zO
Return some info --- useful to spot problems with refcounting
rV z %s: refcount=%s
%sr NrA r? ) rb r r^ r sortrE r_ r; join)r% r4 r9 r r r r r rP 1 s
*zServer.debug_infoc C s t | j d S)z*
Number of shared objects
r )lenr^ )r% r4 r r r rQ @ s zServer.number_of_objectsc C sX zC y t j d | j d Wn d d l } | j Yn XWd | j j Xd S)z'
Shutdown this process
z!manager received shutdown message#RETURNNr )r N)r rv r1 traceback print_excrh set)r% r4 r r r r rL F s
zServer.shutdownc O sB | j 2| j | \ } } } } | d k r[ t | d k rH | sN t | d } n | | | } | d k r t | } | d k r t | t k s t t | t | } d t | }
t j
d | |
| t | | f | j |
<|
| j
k rd | j
|
<| j | |
|
t | f SWd QRXd S)z>
Create a new shared object and return its id
Nr r z%xz&%r callable returned object with id %r)rb rY r r< rJ r dictr r$ r rv r r^ r_ rS tuple) r% r4 r" r6 r7 rD r method_to_typeid proxytyper r r r r rM S s$
z
Server.createc C s t | j | j d S)zL
Return the methods of the shared object indicated by token
r )r r^ r$ )r% r4 r r r r rO v s zServer.get_methodsc C s- | t j _ | j d | j | d S)z=
Spawn a new thread to serve this connection
#RETURNN)r N)r` r r r1 r )r% r4 r r r r rN | s
zServer.accept_connectionc C s& | j | j | d 7
Connect manager object to the server process
r[ NrR ) r\ r r r r: r r r r )r% rd re r r r r connect s zBaseManager.connectc C sj | j j t j k s t | d k r@ t | r@ t d t j d d \ } } | j j
d t | j d | j
| j | j | j | | | f | _ d j d d | j j D } t | j d
| | j _ | j j | j | j | _ | j t j | j _ t j | t | j d | j | j | j | j | j f d d | _ d S)
z@
Spawn a server process for this manager object
Nz initializer must be a callableZduplexFrf r6 :c s s | ] } t | Vq d S)N)r; )r ir r r s z$BaseManager.start..r@ exitpriorityr ) r r r r r< rD TypeErrorr ZPiper ZProcessr _run_serverr r r r _processr Z _identityr+ r rn r r2 r r Finalize_finalize_managerr rL )r% initializerinitargsreaderwriterr r r r rn s( "
zBaseManager.startc C si | d k r | | | j | | | | } | j | j | j t j d | j | j d S)z@
Create a server, report its address and run it
Nzmanager serving at %r)_Serverr1 r# r r r rz ) clsrY r# r[ rc r r r serverr r r r s
zBaseManager._run_serverc O s | j j t j k s! t d | j | j d | j } z) t | d d | f | | \ } } Wd | j Xt
| | j | | f S)zP
Create a new shared object; return the token and exposed tuple
zserver not yet startedr[ NrM ) r r r r r< r r r r: r r )r% r" r6 r7 r r$ r r r r _create s !) zBaseManager._createc C s; | j d k r7 | j j | | j j s7 d | _ d S)zC
Join the manager process (if it has been spawned)
N)r r is_alive)r% timeoutr r r r s zBaseManager.joinc
C sA | j | j d | j } z t | d d SWd | j Xd S)zS
Return some info about the servers shared objects and connections
r[ NrP )r r r r: r )r% r r r r _debug_info s zBaseManager._debug_infoc
C sA | j | j d | j } z t | d d SWd | j Xd S)z5
Return the number of shared objects
r[ NrQ )r r r r: r )r% r r r r _number_of_objects" s z BaseManager._number_of_objectsc C s> | j j t j k r | j | j j t j k s: t | S)N)r r r r rn r r< )r% r r r __enter__, s
zBaseManager.__enter__c C s | j d S)N)rL )r% exc_typeexc_valexc_tbr r r __exit__2 s zBaseManager.__exit__c C s | j r t j d y8 | | d | } z t | d d Wd | j XWn t k
re Yn X| j d d | j r t j d t | d r t j d | j | j d d
| j r t j d t j
| _ y t j
| =Wn t k
r Yn Xd S) zQ
Shutdown the manager process; will be registered as a finalizer
z#sending shutdown message to managerr[ NrL r g ?zmanager still alive terminatez'trying to `terminate()` manager processg?z#manager still alive after terminate)r r r r: r r r hasattrr r r r r _address_to_localKeyError)r r# r[ r( r r r r r r 5 s.
z BaseManager._finalize_managerc C s | j S)N)r )r% r r r U s zBaseManager.Tc
s! d | j k r! | j j | _ d k r3 t | pH t d d } | p` t d d } | r xd t | j D]P \ } } t | t k s t d | t | t k s| t d | q| W| | | f | j <| r f d d } | _
t | | d S)z9
Register a typeid with the manager type
r N _exposed__method_to_typeid_z%r is not a stringc s t j d | j | | \ } } | | j d | d | j d | } | j | j d | j } t | d d | j f | S)Nz)requesting creation of a shared %r objectmanagerr[ r rT ) r rv r r r r r# r: r$ )r% r6 r7 r Zexpproxyr )r r" r r rF r s z"BaseManager.register..temp) __dict__r copy AutoProxyr r r r r; r< r+ setattr)
r r" rD r r r
create_methodkeyr rF r )r r" r registerW s "&
zBaseManager.register)r+ r- r. r/ r rK r r& r r rn classmethodr r r r r r r staticmethodr propertyr# r r r r r r s* $
c @ s( e Z d Z d d Z d d Z d S)ProcessLocalSetc C s t j | d d d S)Nc S s
| j S)N)clear)r r r r r s z*ProcessLocalSet.__init__..)r register_after_fork)r% r r r r& s zProcessLocalSet.__init__c C s t | f f S)N)r )r% r r r
__reduce__ s zProcessLocalSet.__reduce__N)r+ r- r. r& r r r r r r s r c @ s e Z d Z d Z i Z e j Z d d d d d d Z d d Z f i d d Z
d
d Z d d
Z e
d d Z d d Z d d Z d d Z d d Z d d Z d S)r z.
A base for proxies of shared objects
NTc
C s* t j N t j j | j d } | d k rS t j t f } | t j | j )r r+ r r" r$ )r% r r r r, % s zBaseProxy.__repr__c C sB y | j d SWn* t k
r= t | d d d SYn Xd S)zV
Return representation of the referent (or a fall-back if that fails)
r, Nr z; '__str__()' failed>r )r r r )r% r r r rB ) s
zBaseProxy.__str__)r+ r- r. r/ r r ZForkAwareThreadLockr r& r r r r r r r r r r, rB r r r r r s "
c C s t t j d d } | rB | j | j k rB | j | j d S| j d d oj t t j d d } | | | d | | Sd S)z
Function used for unpickling proxy objects.
If possible the shared object is returned, or otherwise a proxy for it.
rj Nr rS TZ _inheritingF)r r ri r# r^ r$ pop)rG r rc r7 r rS r r r r 6 s r c
C s t | } y | | | f SWn t k
r2 Yn Xi } x% | D] } t d | | f | q@ Wt | t f | } | | _ | | | | f <| S)zB
Return a proxy type whose methods are given by `exposed`
zLdef %s(self, *args, **kwds):
return self._callmethod(%r, args, kwds))r r execr r r )r r _cacheZdicmeth ProxyTyper r r
MakeProxyTypeK s
r Tc
C s t | d } | d k rY | | j d | } z t | d d | f } Wd | j X| d k rz | d k rz | j } | d k r t j j } t d | j | } | | | d | d | d | } d | _
| S) z*
Return an auto-proxy for `token`
r Nr[ rO z
AutoProxy[%s]r rS T) r\ r# r: r r r ri r[ r r" r )
r rc r r[ r rS r r r r r r r r a s r c @ s( e Z d Z d d Z d d Z d S) Namespacec K s | j j | d S)N)r update)r% r7 r r r r& s zNamespace.__init__c C s t | j j } g } x: | D]2 \ } } | j d s" | j d | | f q" W| j d | j j d j | f S)NrI z%s=%rz%s(%s)z, ) r r r
startswithrE r r* r+ r )r% r rF r r r r r r, s
zNamespace.__repr__N)r+ r- r. r& r, r r r r r ~ s r c @ sR e Z d Z d d d Z d d Z d d Z d d Z e e e Z d
S) ValueTc C s | | _ | | _ d S)N) _typecode_value)r% r r lockr r r r& s zValue.__init__c C s | j S)N)r )r% r r r r s z Value.getc C s
| | _ d S)N)r )r% r r r r r s z Value.setc C s d t | j | j | j f S)Nz
%s(%r, %r))r r+ r r )r% r r r r, s zValue.__repr__N) r+ r- r. r& r r r, r r r r r r r s
r c C s t j | | S)N)r )r Zsequencer r r r Array s r c @ sR e Z d Z d Z d d Z d d Z d d
Z d d Z d
d Z d S)
IteratorProxy__next__r1 throwr c C s | S)Nr )r% r r r __iter__ s zIteratorProxy.__iter__c G s | j d | S)Nr )r )r% r6 r r r r s zIteratorProxy.__next__c G s | j d | S)Nr1 )r )r% r6 r r r r1 s zIteratorProxy.sendc G s | j d | S)Nr )r )r% r6 r r r r s zIteratorProxy.throwc G s | j d | S)Nr )r )r% r6 r r r r s zIteratorProxy.closeN)r r1 r r ) r+ r- r. r r r r1 r r r r r r r s r c @ sL e Z d Z d
Z d d d d Z d d Z d d
Z d d Z d S)
AcquirerProxyacquirereleaseTNc C s1 | d k r | f n | | f } | j d | S)Nr )r )r% Zblockingr r6 r r r r s !zAcquirerProxy.acquirec C s
| j d S)Nr )r )r% r r r r s zAcquirerProxy.releasec C s
| j d S)Nr )r )r% r r r r s zAcquirerProxy.__enter__c C s
| j d S)Nr )r )r% r r r r r r r s zAcquirerProxy.__exit__)r r )r+ r- r. r r r r r r r r r r
s
r
c @ sL e Z d Z d Z d d d Z d d
Z d d Z d d
d Z d S)ConditionProxyr r rp notify
notify_allNc C s | j d | f S)Nrp )r )r% r r r r rp s zConditionProxy.waitc C s
| j d S)Nr )r )r% r r r r s zConditionProxy.notifyc C s
| j d S)Nr )r )r% r r r r s zConditionProxy.notify_allc C s | } | r | S| d k r/ t | } n d } d } xF | s | d k rj | t } | d k rj P| j | | } q> W| S)Nr )_timerp )r% Z predicater r9 ZendtimeZwaittimer r r wait_for s
zConditionProxy.wait_for)r r rp r r )r+ r- r. r rp r r r r r r r r
s
r
c @ sI e Z d Z d Z d d Z d d Z d d
Z d d d
Z d S)
EventProxyro r r rp c C s
| j d S)Nro )r )r% r r r ro s zEventProxy.is_setc C s
| j d S)Nr )r )r% r r r r s zEventProxy.setc C s
| j d S)Nr )r )r% r r r r s zEventProxy.clearNc C s | j d | f S)Nrp )r )r% r r r r rp s zEventProxy.wait)ro r r rp )r+ r- r. r ro r r rp r r r r r s
r c @ ss e Z d Z d Z d d d Z d d Z d
d Z e d d
Z e d d Z e d d Z
d S) BarrierProxy__getattribute__rp abortresetNc C s | j d | f S)Nrp )r )r% r r r r rp s zBarrierProxy.waitc C s
| j d S)Nr )r )r% r r r r s zBarrierProxy.abortc C s
| j d S)Nr )r )r% r r r r s zBarrierProxy.resetc C s | j d d S)Nr parties)r )r )r% r r r r s zBarrierProxy.partiesc C s | j d d S)Nr n_waiting)r )r )r% r r r r s zBarrierProxy.n_waitingc C s | j d d S)Nr broken)r )r )r% r r r r s zBarrierProxy.broken)r rp r r ) r+ r- r. r rp r r r r r r r r r r r s r c @ s: e Z d Z d Z d d Z d d Z d d Z d
S) NamespaceProxyr __setattr__ __delattr__c C sB | d d k r t j | | St j | d } | d | f S)Nr rI r r )objectr )r% r
callmethodr r r __getattr__ s zNamespaceProxy.__getattr__c C sH | d d k r# t j | | | St j | d } | d | | f S)Nr rI r r )r r r )r% r r r r r r r s zNamespaceProxy.__setattr__c C sB | d d k r t j | | St j | d } | d | f S)Nr rI r r )r r r )r% r r r r r r s zNamespaceProxy.__delattr__N)r r r )r+ r- r. r r r r r r r r r s r c @ s= e Z d Z d Z d d Z d d Z e e e Z d S)
ValueProxyr r c C s
| j d S)Nr )r )r% r r r r s zValueProxy.getc C s | j d | f S)Nr )r )r% r r r r r s zValueProxy.setN)r r )r+ r- r. r r r r r r r r r r s r
BaseListProxy__add__ __contains__ __delitem__ __getitem____len____mul__ __reversed____rmul__ __setitem__rE countextendindexinsertr removereverser __imul__c @ s( e Z d Z d d Z d d Z d S) ListProxyc C s | j d | f | S)Nr, )r )r% r r r r __iadd__ s zListProxy.__iadd__c C s | j d | f | S)Nr1 )r )r% r r r r r1 s zListProxy.__imul__N)r+ r- r. r3 r1 r r r r r2 s r2 DictProxyr r r Zhas_keypopitem
setdefaultr
ArrayProxy PoolProxyZapplyZ apply_asyncr ZimapZimap_unorderedr mapZ map_asyncstarmapZ
starmap_asyncr Z AsyncResultIteratorc @ s( e Z d Z d d Z d d Z d S)r8 c C s | S)Nr )r% r r r r 5 s zPoolProxy.__enter__c C s | j d S)N)r )r% r r r r r r r 7 s zPoolProxy.__exit__N)r+ r- r. r r r r r r r8 4 s c @ s e Z d Z d Z d S)r a(
Subclass of `BaseManager` which supports a number of shared object types.
The types registered are those intended for the synchronization
of threads, plus `dict`, `list` and `Namespace`.
The `multiprocessing.Manager()` function creates started instances of
this class.
N)r+ r- r. r/ r r r r r > s QueueZ
JoinableQueuerg Lockra SemaphoreBoundedSemaphore ConditionBarrierPoolr r r r F)I__all__rs r` r Zqueuer r r r r r
r r r
r r r r Z
view_typesr r! Z view_typer r r: r3 r r= rH rJ rK r rd re Z XmlListenerZ XmlClientr\ r r r r r r r r r r r r
r
r r r r r! r2 r4 r7 Z
BasePoolProxyr r8 r r< rg r= ra r> r? r@ rA rB r r r r r s