U
fY @ s d dl mZ d dlZd dlZd dlZd dlZd dlmZ d dlm Z ddl
m Z m Z ddl
mZ ddlmZ dd lmZmZmZ dd
lmZmZmZmZ dd lmZ dd l m Z dd
l m Z ddl m!Z! ddl"m#Z# ddl$m%Z%m&Z& ej'rd dl(Z(d dl)m*Z* dddgZ+e,e-Z.dZ/dZ0G dd dej1Z2ddddd d Z3e4e3e2e4e3e2d Z5eed Z6G d d deZ7G d d de7Z8d!d"dd#d$dZ9dS )% ) annotationsN)
TracebackType)urljoin )HTTPHeaderDictRecentlyUsedContainer)RequestMethods) ProxyConfig)HTTPConnectionPoolHTTPSConnectionPoolport_by_scheme)LocationValueError
MaxRetryErrorProxySchemeUnknownURLSchemeUnknown)BaseHTTPResponse)_TYPE_SOCKET_OPTIONS) connection_requires_http_tunnel)Retry)Timeout)Url parse_url)Self PoolManager ProxyManagerproxy_from_url) key_file cert_file cert_reqsca_certs ca_cert_data ssl_versionZssl_minimum_versionZssl_maximum_version ca_cert_dir ssl_context key_passwordserver_hostnamei @ c @ s e Zd ZU dZded< ded< ded< ded< d ed
< d ed < d
ed< ded< ded< ded< ded< ded< ded< ded< ded< ded< ded < d ed < ded < d ed!< d"ed#< d ed$< d%ed&< d'ed(< d ed)< d*ed+< ded,< ded-< ded.< d/S )0PoolKeyz
All known keyword arguments that could be provided to the pool manager, its
pools, or the underlying connections.
All custom key schemes should include the fields in this key at a minimum.
str
key_schemekey_host
int | Nonekey_portz Timeout | float | int | None key_timeoutzRetry | bool | int | None key_retriesz bool | None key_blockztuple[str, int] | Nonekey_source_address
str | None key_key_filekey_key_password
key_cert_file
key_cert_reqs key_ca_certszstr | bytes | NoneZkey_ca_cert_datazint | str | Nonekey_ssl_versionzssl.TLSVersion | NoneZkey_ssl_minimum_versionZkey_ssl_maximum_versionkey_ca_cert_dirssl.SSLContext | Nonekey_ssl_context key_maxsizez!frozenset[tuple[str, str]] | None key_headers
Url | None
key__proxykey__proxy_headersProxyConfig | Nonekey__proxy_configz_TYPE_SOCKET_OPTIONS | Nonekey_socket_optionskey__socks_optionszbool | str | Nonekey_assert_hostnamekey_assert_fingerprintkey_server_hostname
key_blocksizeN)__name__
__module__ __qualname____doc____annotations__ rL rL D/opt/alt/python38/lib/python3.8/site-packages/urllib3/poolmanager.pyr& 8 s<
r& z
type[PoolKey]dict[str, typing.Any]) key_classrequest_contextreturnc C s | }|d |d< |d |d< dD ],}||kr,|| dk r,t|| ||< q,|d}|dk rxt||d< t| D ]}|||d| < q| j D ]}||krd||< q|ddkrt
|d< | f |S )a
Create a pool key out of a request context dictionary.
According to RFC 3986, both the scheme and host are case-insensitive.
Therefore, this function normalizes both before constructing the pool
key for an HTTPS request. If you wish to change this behaviour, provide
alternate callables to ``key_fn_by_scheme``.
:param key_class:
The class to use when constructing the key. This should be a namedtuple
with the ``scheme`` and ``host`` keys at a minimum.
:type key_class: namedtuple
:param request_context:
A dictionary-like object that contain the context for a request.
:type request_context: dict
:return: A namedtuple that can be used as a connection pool key.
:rtype: PoolKey
schemehost)headers_proxy_headers_socks_optionsNsocket_optionskey_rF ) copylower frozensetitemsgettuplelistkeyspop_fields_DEFAULT_BLOCKSIZE)rO rP contextkey socket_optsfieldrL rL rM _default_key_normalizer_ s"
rh httphttpsc s e Zd ZU dZdZded< dZded< d@dd d
d d fd
d
ZddddZdddddddZ dAddddd d d d Z
d dd d!Z dBd#d$d#dd d%d&d'Z d(d d)d*d+Z
d,d(d d-d.d/ZdCddd d0d1d2Zdd(d3d4d5Zd6d7d8d9d:ZdDddd7d
dd?Z ZS )Er aV
Allows for arbitrary requests while transparently keeping track of
necessary connection pools for you.
:param num_pools:
Number of connection pools to cache before discarding the least
recently used pool.
:param headers:
Headers to include with all requests, unless other headers are given
explicitly.
:param \**connection_pool_kw:
Additional parameters are used to create fresh
:class:`urllib3.connectionpool.ConnectionPool` instances.
Example:
.. code-block:: python
import urllib3
http = urllib3.PoolManager(num_pools=2)
resp1 = http.request("GET", "https://google.com/")
resp2 = http.request("GET", "https://google.com/mail")
resp3 = http.request("GET", "https://yahoo.com/")
print(len(http.pools))
# 2
Nr< proxyr? proxy_config
int typing.Mapping[str, str] | None
typing.AnyNone) num_poolsrT connection_pool_kwrQ c s4 t | || _| t|| _t| _t | _d S N)super__init__rt r poolspool_classes_by_schemekey_fn_by_schemerY )selfrs rT rt __class__rL rM rw s
zPoolManager.__init__r )rQ c C s | S ru rL r{ rL rL rM __enter__ s zPoolManager.__enter__ztype[BaseException] | NonezBaseException | NonezTracebackType | Noneztyping.Literal[False])exc_typeexc_valexc_tbrQ c C s | dS )NF)clear)r{ r r r rL rL rM __exit__ s zPoolManager.__exit__r' dict[str, typing.Any] | Noner
)rR rS portrP rQ c C st | j | }|dkr | j }|ddkr2t|d< dD ]}||d q6|dkrftD ]}||d qT|||f|S )a
Create a new :class:`urllib3.connectionpool.ConnectionPool` based on host, port, scheme, and
any additional pool keyword arguments.
If ``request_context`` is provided, it is provided as keyword arguments
to the pool class used. This method is used to actually create the
connection pools handed out by :meth:`connection_from_url` and
companion methods. It is intended to be overridden for customization.
N blocksize)rR rS r rj )ry rt rY r] rc ra SSL_KEYWORDS)r{ rR rS r rP pool_clsre kwrL rL rM _new_pool s
zPoolManager._new_poolc C s | j dS )z
Empty our store of pools and direct them all to close.
This will not affect in-flight connections, but they will not be
re-used after completion.
N)rx r r~ rL rL rM r s zPoolManager.clearrj r0 r* rS r rR pool_kwargsrQ c C sT |s t d| |}|p d|d< |s:t|d d}||d< ||d< | |S )a
Get a :class:`urllib3.connectionpool.ConnectionPool` based on the host, port, and scheme.
If ``port`` isn't given, it will be derived from the ``scheme`` using
``urllib3.connectionpool.port_by_scheme``. If ``pool_kwargs`` is
provided, it is merged with the instance's ``connection_pool_kw``
variable and used to create the new connection pool, if one is
needed.
zNo host specified.rj rR P r rS )r
_merge_pool_kwargsr r] rZ connection_from_context)r{ rS r rR r rP rL rL rM connection_from_host s
z PoolManager.connection_from_hostrN )rP rQ c C sX d|kr t dt |d |d }| j|}|sBt|||}| j||dS )z
Get a :class:`urllib3.connectionpool.ConnectionPool` based on the request context.
``request_context`` must at least contain the ``scheme`` key and its
value must be a key in ``key_fn_by_scheme`` instance variable.
strictzdThe 'strict' parameter is no longer needed on Python 3+. This will raise an error in urllib3 v2.1.0.rR rP ) warningswarnDeprecationWarningra rZ rz r] r connection_from_pool_key)r{ rP rR pool_key_constructorpool_keyrL rL rM r 1 s
z#PoolManager.connection_from_contextr& )r rP rQ c
C sl | j jZ | j |}|r*|W 5 Q R S |d }|d }|d }| j||||d}|| j |< W 5 Q R X |S )a
Get a :class:`urllib3.connectionpool.ConnectionPool` based on the provided pool key.
``pool_key`` should be a namedtuple that only contains immutable
objects. At a minimum it must have the ``scheme``, ``host``, and
``port`` fields.
rR rS r r )rx lockr] r )r{ r rP poolrR rS r rL rL rM r J s
z$PoolManager.connection_from_pool_key)urlr rQ c C s t |}| j|j|j|j|dS )a
Similar to :func:`urllib3.connectionpool.connection_from_url`.
If ``pool_kwargs`` is not provided and a new pool needs to be
constructed, ``self.connection_pool_kw`` is used to initialize
the :class:`urllib3.connectionpool.ConnectionPool`. If ``pool_kwargs``
is provided, it is used instead. Note that if a new pool does not
need to be created for the request, the provided ``pool_kwargs`` are
not used.
)r rR r )r r rS r rR )r{ r r urL rL rM connection_from_urld s
z PoolManager.connection_from_url)overriderQ c C sV | j }|rR| D ]:\}}|dkrHz
||= W qP tk
rD Y qPX q|||< q|S )a
Merge a dictionary of override values for self.connection_pool_kw.
This does not modify self.connection_pool_kw and returns a new dict.
Any keys in the override dictionary with a value of ``None`` are
removed from the merged dictionary.
N)rt rY r\ KeyError)r{ r base_pool_kwargsre valuerL rL rM r v s
z PoolManager._merge_pool_kwargsr bool)
parsed_urlrQ c C s" | j dkrdS t| j | j|j S )z
Indicates if the proxy requires the complete destination URL in the
request. Normally this is only needed when not using an HTTP CONNECT
tunnel.
NF)rl r rm rR )r{ r rL rL rM !_proxy_requires_url_absolute_form s
z-PoolManager._proxy_requires_url_absolute_formTr methodr redirectr rQ c K s t |}|jdkr"tjdtdd | j|j|j|jd}d|d< d|d< d |krZ| j|d < | |rv|j
||f|}n|j
||j f|}|o| }|s|S t
||}|jd
krd }d|d < t|d |d < |d
} t| tstj| |d} | jrF||sF|d }
|d D ]"} | | jkr|
| d q|
|d < z| j||||d} W n, tk
r | j r| | Y S X | |d
< ||d< t d|| | | j
||f|S )aN
Same as :meth:`urllib3.HTTPConnectionPool.urlopen`
with custom cross-host redirect logic and only sends the request-uri
portion of the ``url``.
The given ``url`` parameter must be absolute, such that an appropriate
:class:`urllib3.connectionpool.ConnectionPool` can be chosen for it.
Na URLs without a scheme (ie 'https://') are deprecated and will raise an error in a future version of urllib3. To avoid this DeprecationWarning ensure all URLs start with 'https://' or 'http://'. Read more in this issue: https://github.com/urllib3/urllib3/issues/2920 )category
stacklevel)r rR Fassert_same_hostr rT i/ GETbodyretries)r )response_poolzRedirecting %s -> %s) r rR r r r r rS r rT r urlopen request_uriget_redirect_locationr statusr Z_prepare_for_method_changer]
isinstancer from_intremove_headers_on_redirect is_same_hostrY rZ ra incrementr raise_on_redirect
drain_connloginfo) r{ r r r r r connr redirect_locationr new_headersheaderrL rL rM r s\
zPoolManager.urlopen)rn N)N)Nrj N)N)T)rG rH rI rJ rl rK rm rw r r r r r r r r r r r
__classcell__rL rL r| rM r s.
! & c s e Zd ZdZd!dddddd d
d d d
d
fdd
Zd"d dd ddd fdd
Zd#ddddddZd$ddd d d d fd d
Z ZS )%r a>
Behaves just like :class:`PoolManager`, but sends all requests through
the defined proxy, using the CONNECT method for HTTPS URLs.
:param proxy_url:
The URL of the proxy to be used.
:param proxy_headers:
A dictionary containing headers that will be sent to the proxy. In case
of HTTP they are being sent with each request, while in the
HTTPS/CONNECT case they are sent only once. Could be used for proxy
authentication.
:param proxy_ssl_context:
The proxy SSL context is used to establish the TLS connection to the
proxy when using HTTPS proxies.
:param use_forwarding_for_https:
(Defaults to False) If set to True will forward requests to the HTTPS
proxy to be made on behalf of the client instead of creating a TLS
tunnel via the CONNECT method. **Enabling this flag means that request
and response headers and content will be visible from the HTTPS proxy**
whereas tunneling keeps request and response headers and content
private. IP address, target hostname, SNI, and port are always visible
to an HTTPS proxy even when this flag is disabled.
:param proxy_assert_hostname:
The hostname of the certificate to verify against.
:param proxy_assert_fingerprint:
The fingerprint of the certificate to verify against.
Example:
.. code-block:: python
import urllib3
proxy = urllib3.ProxyManager("https://localhost:3128/")
resp1 = proxy.request("GET", "https://google.com/")
resp2 = proxy.request("GET", "https://httpbin.org/")
print(len(proxy.pools))
# 1
resp3 = proxy.request("GET", "https://httpbin.org/")
resp4 = proxy.request("GET", "https://twitter.com/")
print(len(proxy.pools))
# 3
rn NFr' ro rp r8 r z"None | str | typing.Literal[False]r0 rq rr )
proxy_urlrs rT
proxy_headersproxy_ssl_contextuse_forwarding_for_httpsproxy_assert_hostnameproxy_assert_fingerprintrt rQ c
s t |tr&|j d|j d|j }
n|}
t|
} | jdkrFt| j| jsft| jd} | j | d} | | _
|pri | _ || _ t
||||| _| j
| d< | j | d< | j| d< t j||f| d S ) Nz://:ri r )r _proxyrU
_proxy_config)r r
rR rS r r r r r] _replacerl r r r rm rv rw )
r{ r rs rT r r r r r rt Z
str_proxy_urlrl r r| rL rM rw # s,
zProxyManager.__init__rj r* r r
r c s< |dkr t j||||dS t j| jj| jj| jj|dS )Nrk )r )rv r rl rS r rR )r{ rS r rR r r| rL rM r L s z!ProxyManager.connection_from_hostztyping.Mapping[str, str])r rT rQ c C s0 ddi}t |j}|r ||d< |r,|| |S )z
Sets headers needed by proxies: specifically, the Accept and Host
headers. Only sets headers not provided by the user.
Acceptz*/*Host)r netlocupdate)r{ r rT headers_r rL rL rM _set_proxy_headers\ s
z ProxyManager._set_proxy_headersTr r c sR t |}t| j| j|js8|d| j}| |||d< t j ||fd|i|S )z@Same as HTTP(S)ConnectionPool.urlopen, ``url`` must be absolute.rT r )
r r rl rm rR r] rT r rv r )r{ r r r r r rT r| rL rM r m s
zProxyManager.urlopen)rn NNNFNN)Nrj N)N)T) rG rH rI rJ rw r r r r rL rL r| rM r s" 9 &, r' rq )r r rQ c K s t f d| i|S )Nr )r )r r rL rL rM r | s ):
__future__r functoolsloggingtypingr typesr urllib.parser _collectionsr r Z_request_methodsr
connectionr connectionpoolr
r r
exceptionsr
r r r r r Zutil.connectionr Z
util.proxyr Z
util.retryr Z util.timeoutr Zutil.urlr r
TYPE_CHECKINGsslZtyping_extensionsr __all__ getLoggerrG r r rc
NamedTupler& rh partialrz ry r r r rL rL rL rM sH
'<
L