3
Pf/l @ s d Z ddlZddlZddlZddlZddlZddlZddlZddlZddl Z ddl
Z
ddl Z ddl m Z ddl
mZ ddlmZ ddlmZ ddlmZmZmZmZmZmZmZmZm Z m Z m Z m Z m Z m!Z! dd l"m#Z# dd
l$m%Z% dd l&m'Z'm(Z(m)Z)m*Z* dfZ+ej, Z-dddZ.ej/ dkr0dd Z0dd Zdd Z1dd Z2dgdd Z3d d Z4d d Z5d!d" Z6d#d$ Z7d%d& Z8dhd'd(Z9d)d* Z:d+d, Z;d-d. Zd3d4 Z?d5d6 Z@eAdiZBd9d: ZCd;d< ZDd=d> ZEd?d@ ZFdAdB ZGdCdD ZHejIdEdF ZJdGdH ZKdjdIdJZLdKdL ZMdkdNdOZNdPdQ ZOdRdS ZPdTjQdUZReRdV ZSeRdW ZTdXdY ZUdZd[ ZVd\d] ZWejXd^ZYejXd_ZZd`da Z[dbdc Z\ddde Z]dS )lz
requests.utils
~~~~~~~~~~~~~~
This module provides utility functions that are used within Requests
that are also useful for external consumption.
N ) __version__)certs)to_native_string)parse_http_list)quoteurlparsebytesstr OrderedDictunquote
getproxies proxy_bypass
urlunparse
basestring
integer_typesis_py3proxy_bypass_environmentgetproxies_environment)cookiejar_from_dict)CaseInsensitiveDict)
InvalidURL
InvalidHeaderFileModeWarningUnrewindableBodyError.netrc_netrcP i )ZhttpZhttpsZWindowsc
C s t rdd l}ndd l}y2|j|jd}|j|dd }|j|dd }W n tk
r\ dS X | sj| rndS |jd}xX|D ]P}|dkrd| krd S |jdd
}|jd d }|jd
d}t j
|| t j r~d S q~W dS )Nr z;Software\Microsoft\Windows\CurrentVersion\Internet SettingsZ ProxyEnableZ
ProxyOverrideF;z.Tz\.*z.*?) r winreg_winregOpenKeyHKEY_CURRENT_USERZ QueryValueExOSErrorsplitreplacerematchI)hostr" ZinternetSettingsZ proxyEnableZ
proxyOverrideZtest r- /usr/lib/python3.6/utils.pyproxy_bypass_registry. s2
r/ c C s t rt| S t| S dS )zReturn True, if the host should be bypassed.
Checks proxy settings gathered from the environment, if specified,
or the registry.
N)r r r/ )r, r- r- r. r O s r c C s t | dr| j } | S )z/Returns an internal sequence dictionary update.items)hasattrr0 )dr- r- r. dict_to_sequence[ s
r3 c C s2 d }d}t | dr t| }nbt | dr.| j}nPt | dr~y | j }W n tjk
rZ Y n$X tj|j}d| jkr~t j
dt t | dry | j }W n$ t
tfk
r |d k r|}Y n\X t | do|d kry&| jdd | j }| j|pd W n t
tfk
r d}Y nX |d kr$d}td|| S )
Nr __len__lenfilenoba% Requests has determined the content-length for this request using the binary size of the file: however, the file has been opened in text mode (i.e. without the 'b' flag in the mode). This may lead to an incorrect content-length. In Requests 3.0, support will be removed for files in text mode.tellseek )r1 r5 r6 ioUnsupportedOperationosfstatst_sizemodewarningswarnr r8 r& IOErrorr9 max)oZ total_lengthZcurrent_positionr6 r- r- r. super_lend s@
rF Fc C s yddl m }m} d}xJtD ]B}ytjjdj|}W n tk
rJ dS X tjj|r |}P q W |dkrndS t | }d}t
| t r|j d}|j
j|d } y6||j| }
|
r|
d rdnd} |
| |
d fS W n |tfk
r |r Y nX W n ttfk
r Y nX dS ) z;Returns the Requests tuple auth for a given url from netrc.r )netrcNetrcParseErrorNz~/{0} :asciir r: )rG rH NETRC_FILESr= path
expanduserformatKeyErrorexistsr
isinstancer
decodenetlocr' ZauthenticatorsrC ImportErrorAttributeError) urlZ raise_errorsrG rH Z
netrc_pathflocZriZsplitstrr, r Zlogin_ir- r- r. get_netrc_auth s8
rY c C sB t | dd}|r>t|tr>|d dkr>|d dkr>tjj|S dS )z0Tries to guess the filename of the given object.nameNr )getattrrQ r r= rL basename)objrZ r- r- r. guess_filename s ra c C s. | dkr dS t | ttttfr&tdt| S )a Take an object and test to see if it can be represented as a
dictionary. Unless it can not be represented as such, return an
OrderedDict, e.g.,
::
>>> from_key_val_list([('key', 'val')])
OrderedDict([('key', 'val')])
>>> from_key_val_list('string')
ValueError: need more than 1 value to unpack
>>> from_key_val_list({'key': 'val'})
OrderedDict([('key', 'val')])
:rtype: OrderedDict
Nz+cannot encode objects that are not 2-tuples)rQ r
r boolint
ValueErrorr )valuer- r- r. from_key_val_list s
rf c C sB | dkr dS t | ttttfr&tdt | tjr:| j } t | S )a Take an object and test to see if it can be represented as a
dictionary. If it can be, return a list of tuples, e.g.,
::
>>> to_key_val_list([('key', 'val')])
[('key', 'val')]
>>> to_key_val_list({'key': 'val'})
[('key', 'val')]
>>> to_key_val_list('string')
ValueError: cannot encode objects that are not 2-tuples.
:rtype: list
Nz+cannot encode objects that are not 2-tuples)
rQ r
r rb rc rd collectionsMappingr0 list)re r- r- r. to_key_val_list s rj c C s\ g }xRt | D ]F}|dd |dd ko4dkn rJt|dd }|j| qW |S )a Parse lists as described by RFC 2068 Section 2.
In particular, parse comma-separated lists where the elements of
the list may include quoted-strings. A quoted-string could
contain a comma. A non-quoted string could have quotes in the
middle. Quotes are removed automatically after parsing.
It basically works like :func:`parse_set_header` just that items
may appear multiple times and case sensitivity is preserved.
The return value is a standard :class:`list`:
>>> parse_list_header('token, "quoted value"')
['token', 'quoted value']
To create a header from the :class:`list` again, use the
:func:`dump_header` function.
:param value: a string with a list header.
:return: :class:`list`
:rtype: list
Nr "r] r] )_parse_list_headerunquote_header_valueappend)re resultitemr- r- r. parse_list_header s (rq c C s| i }xrt | D ]f}d|kr$d||< q|jdd\}} | dd | dd koVdkn rlt| dd } | ||< qW |S )a^ Parse lists of key, value pairs as described by RFC 2068 Section 2 and
convert them into a python dict:
>>> d = parse_dict_header('foo="is a fish", bar="as well"')
>>> type(d) is dict
True
>>> sorted(d.items())
[('bar', 'as well'), ('foo', 'is a fish')]
If there is no value for a key it will be `None`:
>>> parse_dict_header('key_without_value')
{'key_without_value': None}
To create a header from the :class:`dict` again, use the
:func:`dump_header` function.
:param value: a string with a dict header.
:return: :class:`dict`
:rtype: dict
=Nr rk r] r] )rl r' rm )re ro rp rZ r- r- r. parse_dict_header1 s ( rs c C s^ | rZ| d | d ko dkn rZ| dd
} | sF| dd dkrZ| j ddj ddS | S ) zUnquotes a header value. (Reversal of :func:`quote_header_value`).
This does not use the real unquoting but what browsers are actually
using for quoting.
:param value: the header value to unquote.
:rtype: str
r r rk Nr: z\\\z\"r] r] )r( )re Z is_filenamer- r- r. rm T s
$ rm c C s" i }x| D ]}|j ||j< q
W |S )zReturns a key/value dictionary from a CookieJar.
:param cj: CookieJar object to extract cookies from.
:rtype: dict
)re rZ )cj cookie_dictZcookier- r- r. dict_from_cookiejarm s
rw c C s
t || S )zReturns a CookieJar from a key/value dictionary.
:param cj: CookieJar to insert cookies into.
:param cookie_dict: Dict of key/values to insert into CookieJar.
:rtype: CookieJar
)r )ru rv r- r- r. add_dict_to_cookiejar| s rx c C sT t jdt tjdtjd}tjdtjd}tjd}|j| |j| |j| S )zlReturns encodings from given content string.
:param content: bytestring to extract encodings from.
zIn requests 3.0, get_encodings_from_content will be removed. For more information, please see the discussion on issue #2266. (This warning should only appear once.)z!])flagsz+]z$^<\?xml.*?encoding=["\']*(.+?)["\'>])rA rB DeprecationWarningr) compiler+ findall)contentZ
charset_reZ pragma_reZxml_rer- r- r. get_encodings_from_content s
r~ c C sF | j d}|sdS tj|\}}d|kr6|d jdS d|krBdS dS )z}Returns encodings from given HTTP Header Dict.
:param headers: dictionary to extract encoding from.
:rtype: str
z content-typeNcharsetz'"textz
ISO-8859-1)getcgiZ parse_headerstrip)headersZ content_typeparamsr- r- r. get_encoding_from_headers s
r c c sr |j dkr"x| D ]
}|V qW dS tj|j dd}x | D ]}|j|}|r:|V q:W |jddd}|rn|V dS )zStream decodes a iterator.Nr( )errors T)final)encodingcodecsgetincrementaldecoderrR )iteratorrrp decoderchunkrvr- r- r. stream_decode_response_unicode s
r c c sL d}|dks|dkr t | }x*|t | k rF| ||| V ||7 }q W dS )z Iterate over slices of a string.r N)r5 )stringZ slice_lengthposr- r- r. iter_slices s r c C sv t jdt g }t| j}|rJy t| j|S tk
rH |j| Y nX yt| j|ddS t k
rp | jS X dS )zReturns the requested content back in unicode.
:param r: Response object to get unicode content from.
Tried:
1. charset from content-type
2. fall back and replace all unicode characters
:rtype: str
zIn requests 3.0, get_unicode_from_response will be removed. For more information, please see the discussion on issue #2266. (This warning should only appear once.)r( )r N)
rA rB rz r r r
r} UnicodeErrorrn TypeError)r Ztried_encodingsr r- r- r. get_unicode_from_response s
r Z4ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyzz0123456789-._~c
C s | j d}xtdt|D ]}|| dd }t|dkr|j rytt|d}W n tk
rt td| Y nX |tkr||| dd ||< qd|| ||< qd|| ||< qW dj |S ) zUn-escape any percent-escape sequences in a URI that are unreserved
characters. This leaves all reserved, illegal and non-ASCII bytes encoded.
:rtype: str
%r r r: z%Invalid percent-escape sequence: '%s'N )
r' ranger5 isalnumchrrc rd r UNRESERVED_SETjoin)uripartsihcr- r- r. unquote_unreserved s
r c C s: d}d}yt t| |dS tk
r4 t | |dS X dS )zRe-quote the given URI.
This function passes the given URI through an unquote/quote cycle to
ensure that it is fully and consistently quoted.
:rtype: str
z!#$%&'()*+,/:;=?@[]~z!#$&'()*+,/:;=?@[]~)ZsafeN)r r r )r Zsafe_with_percentZsafe_without_percentr- r- r. requote_uri
s r c C sl t jdtj| d }|jd\}}t jdtjtt|d }t jdtj|d |@ }||@ ||@ kS )zThis function allows you to check if an IP belongs to a network subnet
Example: returns True if ip = 192.168.1.1 and net = 192.168.1.0/24
returns False if ip = 192.168.1.1 and net = 192.168.100.0/24
:rtype: bool
z=Lr /)structunpacksocket inet_atonr' dotted_netmaskrc )ipZnetZipaddrZnetaddrbitsZnetmaskZnetworkr- r- r. address_in_network# s
r c C s& ddd| > d A }t jtjd|S )zConverts mask from /xx format to xxx.xxx.xxx.xxx
Example: if mask is 24 function returns 255.255.255.0
:rtype: str
l r z>I)r Z inet_ntoar pack)maskr r- r- r. r 2 s r c C s* yt j| W n t jk
r$ dS X dS )z
:rtype: bool
FT)r r error)Z string_ipr- r- r. is_ipv4_address= s
r c C s | j ddkryt| jdd }W n tk
r8 dS X |dk sJ|dkrNdS ytj| jdd W q tjk
r| dS X ndS dS )zV
Very simple check of the cidr format in no_proxy variable.
:rtype: bool
r r Fr r T)countrc r' rd r r r )Zstring_networkr r- r- r.
is_valid_cidrH s r c c sT |dk }|r"t jj| }|t j| < z
dV W d|rN|dkrDt j| = n
|t j| < X dS )zSet the environment variable 'env_name' to 'value'
Save previous value, yield, and then restore the previous value stored in
the environment variable 'env_name'.
If 'value' is None, do nothingN)r= environr )Zenv_namere Z
value_changedZ old_valuer- r- r. set_environ` s
r c C s dd }|}|dkr |d}t | j}|rdd |jddjd D }|jd
d }t|rxb|D ](}t|r~t||rd S qb||krbd S qbW n0x.|D ]&}|j|s|jd
d j|rd S qW td|2 y t |}W n t
t j fk
r d
}Y nX W dQ R X |r d S d
S )zL
Returns whether we should bypass proxies or not.
:rtype: bool
c S s t jj| pt jj| j S )N)r= r r upper)kr- r- r. | s z'should_bypass_proxies..Nno_proxyc s s | ]}|r|V qd S )Nr- ).0r, r- r- r. s z(should_bypass_proxies.. r ,:r TF)
r rS r( r' r r r endswithr r r r Zgaierror) rV r Z get_proxyZ no_proxy_argrS r Zproxy_ipr, Zbypassr- r- r. should_bypass_proxiesv s4
r c C s t | |dri S t S dS )zA
Return a dict of environment proxies.
:rtype: dict
)r N)r r
)rV r r- r- r. get_environ_proxies s r c C sv |pi }t | }|jdkr.|j|j|jdS |jd |j |jd|j dg}d}x |D ]}||krX|| }P qXW |S )zSelect a proxy for the url, if applicable.
:param url: The url being for the request
:param proxies: A dictionary of schemes or schemes and hosts to proxy URLs
Nallz://zall://)r Zhostnamer scheme)rV ZproxiesZurlpartsZ
proxy_keysproxyZ proxy_keyr- r- r. select_proxy s
r python-requestsc C s d| t f S )zO
Return a string representing the default user agent.
:rtype: str
z%s/%s)r )rZ r- r- r. default_user_agent s r c C s t t djddddS )z9
:rtype: requests.structures.CaseInsensitiveDict
z, gzipdeflatez*/*z
keep-alive)z
User-AgentzAccept-EncodingZAcceptZ
Connection)r r )r r r r- r- r- r. default_headers s
r c C s g }d}xt jd| D ]}y|jdd\}}W n tk
rL |d }}Y nX d|jdi}xP|jdD ]B}y|jd\}} W n tk
r P Y nX | j|||j|< qhW |j| qW |S ) zReturn a dict of parsed link headers proxies.
i.e. Link: ; rel=front; type="image/jpeg",; rel=back;type="image/jpeg"
:rtype: list
z '"z, * '"rr )r) r' rd r rn ) re ZlinksZ
replace_charsvalrV r linkZparamkeyr- r- r. parse_header_links s r rJ r: c C s | dd }|t jt jfkr dS |dd t jkr6dS |dd t jt jfkrRdS |jt}|dkrhd S |dkr|ddd tkrd
S |d dd tkrd S |dkr|dd t krd
S |d d t krdS dS )z
:rtype: str
N zutf-32r z utf-8-sigr: zutf-16r zutf-8z utf-16-ber z utf-16-lez utf-32-bez utf-32-le)
r BOM_UTF32_LE BOM_UTF32_BEBOM_UTF8 BOM_UTF16_LE BOM_UTF16_BEr _null_null2_null3)dataZsampleZ nullcountr- r- r. guess_json_utf s*
r c C s8 t | |\}}}}}}|s$|| }}t||||||fS )zGiven a URL that may or may not have a scheme, prepend the given scheme.
Does not replace a present scheme with the one provided as an argument.
:rtype: str
)r r )rV Z
new_schemer rS rL r queryfragmentr- r- r. prepend_scheme_if_needed1 s
r c C sB t | }yt|jt|jf}W n ttfk
r< d}Y nX |S )z{Given a url with authentication components, extract them into a tuple of
username,password.
:rtype: (str,str)
r )r r )r r ZusernameZpasswordrU r )rV ZparsedZauthr- r- r. get_auth_from_urlB s
r s ^\S[^\r\n]*$|^$z^\S[^\r\n]*$|^$c C sf | \}}t |trt}nt}y|j|s4td| W n* tk
r` td||t|f Y nX dS )zVerifies that header value is a string which doesn't contain
leading whitespace or return characters. This prevents unintended
header injection.
:param header: tuple, in the format (name, value).
z7Invalid return character or leading space in header: %sz>Value for header {%s: %s} must be of type str or bytes, not %sN)rQ r _CLEAN_HEADER_REGEX_BYTE_CLEAN_HEADER_REGEX_STRr* r r type)headerrZ re Zpatr- r- r. check_header_validityW s
r c C sF t | \}}}}}}|s"|| }}|jddd }t|||||dfS )zW
Given a url remove the fragment and the authentication part.
:rtype: str
@r r r] )r rsplitr )rV r rS rL r r r r- r- r.
urldefragauthl s
r c C s` t | jdd}|dk rTt| jtrTy|| j W q\ ttfk
rP tdY q\X ntddS )zfMove file pointer back to its recorded starting position
so it can be read again on redirect.
r9 Nz;An error occurred when rewinding request body for redirect.z+Unable to rewind request body for redirect.)r^ ZbodyrQ Z_body_positionr rC r& r )Zprepared_requestZ body_seekr- r- r. rewind_body} s r )r r )F)FzBABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-._~)N)r )^__doc__r r rg
contextlibr; r= platformr) r r rA r r r Z_internal_utilsr compatr rl r r r r
r r r
r r r r r r r Zcookiesr Z
structuresr
exceptionsr r r r rK whereZDEFAULT_CA_BUNDLE_PATHZ
DEFAULT_PORTSsystemr/ r3 rF rY ra rf rj rq rs rm rw rx r~ r r r r frozensetr r r r r r r contextmanagerr r r r r r r encoder r r r r r r{ r r r r r r- r- r- r. s @
! =
3 #
% 9
"