3
P\ @ s d Z ddlZddlZddlmZ ddlmZ G dd deZG d d
d
eZ G d d d e Z
G d
d de Z dd Z dd dgZ
dS )zEModule implementing the Pool for :mod:``requests_toolbelt.threaded``. N )thread )queuec @ sn e Zd ZdZdddejfddZdd Zedd Z edd d
Z
d d Z d
d Z dd Z
dd Zdd ZdS )Poola> Pool that manages the threads containing sessions.
:param queue:
The queue you're expected to use to which you should add items.
:type queue: queue.Queue
:param initializer:
Function used to initialize an instance of ``session``.
:type initializer: collections.Callable
:param auth_generator:
Function used to generate new auth credentials for the session.
:type auth_generator: collections.Callable
:param int num_process:
Number of threads to create.
:param session:
:type session: requests.Session
Nc s| |d krt j pd}|dk r$td| _tj _tj _| _|pJt _
|pTt _ | _ fddt
jD _d S )Nr z)Number of processes should at least be 1.c s& g | ] }t j j j j jqS )r Z
SessionThread _new_session
_job_queue_response_queue
_exc_queue).0_)selfr /usr/lib/python3.6/pool.py
+ s z!Pool.__init__..)multiprocessing cpu_count
ValueErrorr r Queuer
r Z
_processes _identity _initializer_auth_sessionrange_pool)r job_queueZ initializerZauth_generatorZ
num_processesZsessionr )r r __init__ s
z
Pool.__init__c C s | j | j| j S )N)r r r )r r r r r 0 s zPool._new_sessionc K s4 t j }x|D ]}|j|j qW | f d|i|S )a2 Create a :class:`~Pool` from an :class:`~ThreadException`\ s.
Provided an iterable that provides :class:`~ThreadException` objects,
this classmethod will generate a new pool to retry the requests that
caused the exceptions.
:param exceptions:
Iterable that returns :class:`~ThreadException`
:type exceptions: iterable
:param kwargs:
Keyword arguments passed to the :class:`~Pool` initializer.
:returns: An initialized :class:`~Pool` object.
:rtype: :class:`~Pool`
r )r r putrequest_kwargs)cls
exceptionskwargsr excr r r from_exceptions3 s
zPool.from_exceptionsc K s^ ddi}|j |pi tj }x,|D ]$}|j }|j d|i |j| q$W | f d|i|S )a Create a :class:`~Pool` from an iterable of URLs.
:param urls:
Iterable that returns URLs with which we create a pool.
:type urls: iterable
:param dict request_kwargs:
Dictionary of other keyword arguments to provide to the request
method.
:param kwargs:
Keyword arguments passed to the :class:`~Pool` initializer.
:returns: An initialized :class:`~Pool` object.
:rtype: :class:`~Pool`
methodZGETurlr )updater r copyr )r Zurlsr r! Z request_dictr r% Zjobr r r from_urlsI s
zPool.from_urlsc c s" x | j }|dkrP |V qW dS )zoIterate over all the exceptions in the pool.
:returns: Generator of :class:`~ThreadException`
N)
get_exception)r r" r r r r b s
zPool.exceptionsc C s8 y| j j \}}W n tjk
r( dS X t||S dS )zSGet an exception from the pool.
:rtype: :class:`~ThreadException`
N)r
get_nowaitr EmptyThreadException)r requestr" r r r r) m s
zPool.get_exceptionc C s8 y| j j \}}W n tjk
r( dS X t||S dS )zPGet a response from the pool.
:rtype: :class:`~ThreadResponse`
N)r
r* r r+ ThreadResponse)r r- responser r r get_responsey s
zPool.get_responsec c s" x | j }|dkrP |V qW dS )zmIterate over all the responses in the pool.
:returns: Generator of :class:`~ThreadResponse`
N)r0 )r Zrespr r r responses s
zPool.responsesc C s x| j D ] }|j qW dS )z*Join all the threads to the master thread.N)r join)r Zsession_threadr r r join_all s z
Pool.join_all)N)__name__
__module__ __qualname____doc__requestsZSessionr r classmethodr# r( r r) r0 r1 r3 r r r r r s r c @ s e Zd ZdZdd ZdS ) ThreadProxyNc C s4 t j}|| jkr&|| | j}t||S || |S dS )z/Proxy attribute accesses to the proxied object.N)object__getattribute__attrs proxied_attrgetattr)r attrgetr/ r r r __getattr__ s
zThreadProxy.__getattr__)r4 r5 r6 r> rB r r r r r: s r: c @ s( e Zd ZdZdZeddgZdd ZdS )r. a> A wrapper around a requests Response object.
This will proxy most attribute access actions to the Response object. For
example, if you wanted the parsed JSON from the response, you might do:
.. code-block:: python
thread_response = pool.get_response()
json = thread_response.json()
r/ r c C s || _ || _d S )N)r r/ )r r r/ r r r r s zThreadResponse.__init__N)r4 r5 r6 r7 r> frozensetr= r r r r r r. s r. c @ s( e Zd ZdZdZeddgZdd ZdS )r, a= A wrapper around an exception raised during a request.
This will proxy most attribute access actions to the exception object. For
example, if you wanted the message from the exception, you might do:
.. code-block:: python
thread_exc = pool.get_exception()
msg = thread_exc.message
exceptionr c C s || _ || _d S )N)r rD )r r rD r r r r s zThreadException.__init__N)r4 r5 r6 r7 r> rC r= r r r r r r, s r, c C s | S )Nr )Z session_objr r r r s r )r7 r r8 r Z_compatr r; r r: r. r, r __all__r r r r s