3
\meCm @ s d dl Z d dlZd dlZd dlZd dlZd dlZd dlmZmZm Z d dl
m Z d dl m
Z
d dlmZ d dlmZ d dlmZ d dlmZmZmZmZmZmZmZ d d l m Z m Z m Z d dl Z d d
l m!Z! d d l"m#Z# ej$e%Z&d Z'd
d Z(dd Z)ed5 dddZ*G dd dZ+G dd de+Z,G dd dZ-G dd de.Z/d d Z0d6e1e-d"d#d$Z2d7ed%ef e3e4ej5ee6 ed&d'd(Z7d8ed%ef ee3 e6e4ed+d,d-Z8dde&j9ddd ddd!d d)f eee e4ee eeee4ge4f e3e1e6d.d/d0Z:G d1d2 d2Z;d9d3d4Z t t| dd}|d r6|d r6|d |d< d|d< t|S )NZhttp)scheme )listr r )urlZ
parsed_url r /usr/lib/python3.6/url_helper.py _cleanurl$ s
r c G s( dd }| }x|D ]}|||}qW |S )Nc S sP t t| }|d }|r,|jd r,|d7 }|tt|dd7 }||d< t|S )Nr /z/:)Zsafe)r r endswithr strr )r add_onZ
url_parsedpathr r r combine_single/ s z#combine_url..combine_singler )baseZadd_onsr% r r# r r r combine_url. s
r' FileResponse UrlResponse)returnc K s | j } | jdrd| } | j jdr|jdr>tjd| | tdd }y"t|d}|j }W dQ R X W nH t k
r } z,|j
}|j
t krt }t
||d| d|W Y dd}~X nX t||d S t| f|S dS )
a0 Wrapper function around readurl to allow passing a file path as url.
When url is not a local file path, passthrough any kwargs to readurl.
In the case of parameter passthrough to readurl, default values for some
parameters. See: call-signature of readurl in this module for param docs.
r z file://%szfile://dataz'Unable to post data to file resource %sNrb)causecodeheadersr )contents)lstrip
startswithlowergetLOGwarninglenopenreadIOErrorerrnor r UrlErrorr( readurl)r kwargsZ file_pathfpr0 er. r r r read_file_or_url> s"
$ rA c @ s& e Zd Zd ddZdd Zdd ZdS )
StringResponse c C s || _ i | _|| _d | _d S )N)r. r/ r0 r )selfr0 r. r r r __init__^ s zStringResponse.__init__c O s | j dkrdS dS )NrC FT)r. )rD argsr> r r r okd s
zStringResponse.okc C s | j jdS )Nzutf-8)r0 decode)rD r r r __str__i s zStringResponse.__str__N)rC )__name__
__module__ __qualname__rE rG rI r r r r rB ] s
rB c @ s e Zd ZdddZdS )r( rC c C s t j| ||d || _d S )N)r. )rB rE r )rD r$ r0 r. r r r rE n s zFileResponse.__init__N)rC )rJ rK rL rE r r r r r( m s c @ s e Zd ZejdddZeedddZedd Z dd
d Z
ed d
Z edd Z dd Z
dee eee dddZdS )r) )responsec C s
|| _ d S )N) _response)rD rM r r r rE t s zUrlResponse.__init__)r* c C s | j jd krdS | j jS )N )rN content)rD r r r r0 w s zUrlResponse.contentsc C s | j jS )N)rN r )rD r r r r } s zUrlResponse.urlFc C s2 d}|r d}d| j ko |k n r*dS dS d S )Ni, i rC TF)r. )rD Z redirects_okupperr r r rG s zUrlResponse.okc C s | j jS )N)rN r/ )rD r r r r/ s zUrlResponse.headersc C s | j jS )N)rN status_code)rD r r r r. s zUrlResponse.codec C s | j jS )N)rN text)rD r r r rI s zUrlResponse.__str__r )
chunk_sizedecode_unicoder* c c s | j j||E dH dS )at Iterates over the response data.
When stream=True is set on the request, this avoids reading the content
at once into memory for large responses.
:param chunk_size: Number of bytes it should read into memory.
:param decode_unicode: If True, content will be decoded using the best
available encoding based on the response.
N)rN iter_content)rD rT rU r r r rV s zUrlResponse.iter_contentN)F)r F)rJ rK rL requestsZResponserE propertybytesr0 r rG r/ r. rI r intboolr rV r r r r r) s s
c @ s e Zd ZdddZdS )r< Nc C s< t j| t| || _|| _|| _| jd kr2i | _|| _d S )N)r: rE r" r- r. r/ r )rD r- r. r/ r r r r rE s
zUrlError.__init__)NNN)rJ rK rL rE r r r r r< s r< c C s i }t | j}|dkr~|r~d|kr8|d r8|d |d< nd|d< d|krfd|krf|d |d g|d< nd|kr~t|d |d< |S )NZhttpsZca_certsZverifyTZ cert_fileZkey_fileZcert)r r r" )r ssl_detailsZssl_argsr r r r
_get_ssl_args s
r] r TFr )streamr* c s" t | } | |d}|jt| | |
|d< |s:|r6dnd}||d< |dk rpt|tr^||d< ntt|d|d< |dkr|g }d }|rtt|d d }d
d tj i} r|j | |ṡ fd d
}|}|r||d< |dkrd}g }x*|
rt
nt d|D ]}|| |d< i }x|j D ]t\}|dkr6q |dkr|rfdd|D }|rt
j||< x$|D ]}t|| |< qrW n||< q W y|rtjd||
rdn|| | | dkrtj } | }|jf |} W dQ R X | r| j tjd| | jt| j|d t| S tjk
r } zt| tj rzt | drzt | j drz|j t | | j j| j j!| d n"|j t | | d t| tj"rP | r| |j
|d rP |
r|dks|d |k r|dkr|rtjd| t#j$| W Y dd} ~ X nX q W |d dS ) a Wrapper around requests.Session to read the url and retry if necessary
:param url: Mandatory url to request.
:param data: Optional form data to post the URL. Will set request_method
to 'POST' if present.
:param timeout: Timeout in seconds to wait for a response. May be a tuple
if specifying (connection timeout, read timeout).
:param retries: Number of times to retry on exception if exception_cb is
None or exception_cb returns True for the exception caught. Default is
to fail with 0 retries on exception.
:param sec_between: Default 1: amount of seconds passed to time.sleep
between retries. None or -1 means don't sleep.
:param headers: Optional dict of headers to send during request
:param headers_cb: Optional callable returning a dict of values to send as
headers during request
:param headers_redact: Optional list of header names to redact from the log
:param ssl_details: Optional dict providing key_file, ca_certs, and
cert_file keys for use on in ssl connections.
:param check_status: Optional boolean set True to raise when HTTPError
occurs. Default: True.
:param allow_redirects: Optional boolean passed straight to Session.request
as 'allow_redirects'. Default: True.
:param exception_cb: Optional callable which accepts the params
msg and exception and returns a boolean True if retries are permitted.
:param session: Optional exiting requests.Session instance to reuse.
:param infinite: Bool, set True to retry indefinitely. Default: False.
:param log_req_resp: Set False to turn off verbose debug messages.
:param request_method: String passed as 'method' to Session.request.
Typically GET, or POST. Default: POST if data is provided, GET
otherwise.
:param stream: if False, the response content will be immediately
downloaded.
)r r^ allow_redirectsZPOSTZGETmethodNtimeoutr r z
User-Agentz
Cloud-Init/%sc s S )Nr )r )r/ r r _cb s zreadurl.._cbr+ r/ c s g | ]} j |r|qS r )r4 ).0k)vr r
' s zreadurl..z'[%s/%s] open '%s' with %s configurationinfinitez(Read from %s (%s, %sb) after %s attemptsrM rR )r. r/ r )r z1Please wait %s seconds while we wait to try againrh rh )%r updater]
isinstancetuplemaxfloatrZ r Zversion_stringr rangeitemscopydeepcopyr r5 debugrW ZSessionZrequestZraise_for_statusrR r7 rP r) r ZRequestExceptionZ HTTPErrorhasattrrM appendr< r/ ZSSLErrortimesleep) r r+ ra ZretriesZ sec_betweenr/
headers_cbheaders_redactr\ check_statusr_ exception_cbZsessionrg Z log_req_resprequest_methodr^ Zreq_argsZ manual_triesZ def_headersrb ZexcpsiZfiltered_req_argsrd Zmatched_headerskeyZsessrr@ r )r/ re r r= s 4
"r= .)funcaddrra eventdelayr* c C s |r|j |drdS | ||S )z Execute func with optional delay)ra N)wait)r r ra r r r r r _run_func_with_delayu s r 333333?
)r addresses
stagger_delayra r* c s d}d}d}g }t j tt|d zy fddt|D }xTt|dD ]D} || }| j }
|
r~|
}|j| qV| j }|rVj ||fS qVW |rt
j d|| |nt
j d| t
dW n. tk
r t
j d d
j|d
j| Y nX W d jd d X ||fS )
a execute multiple callbacks in parallel
Run blocking func against two different addresses staggered with a
delay. The first call to return successfully is returned from this
function and remaining unfinished calls are cancelled if they have not
yet started
N) max_workersc s, i | ]$\}}| j t|| d qS ))r r ra r r )Zsubmitr )rc r| r )executorr is_doner ra r r
s z dual_stack..)ra z.default_sleep_timec S s$ | dkr dS | dkp"t j | | kS )z4Check if time is up based on start time and max waitNFr )ru )max_wait
start_timer r r timeup s z wait_for_url..timeupc S sf | j s*d| j }tt|| j| j|d}n4| j sVd| j }tt|| j| j|d}nd}d}||fS )z?Map requests response code/contents to internal "UrlError" typezempty response [%s])r. r/ r zbad status code [%s]r N)r0 r. r< r r/ rG )rM r reasonurl_excr r r handle_url_response s"
z)wait_for_url..handle_url_responsec
s d}d}y*| |\}} ||\}}|s0||fS W nZ t k
r` } zd| }| }W Y dd} ~ X n. tk
r } zd| }| }W Y dd} ~ X nX ttj | }
rd nd} d||
| |f } || |r|| |d dS ) z:Execute request, handle response, optionally log exceptionr Nzrequest error [%s]zunexpected error [%s]z%ssZ unlimitedz Calling '%s' failed [%s/%s]: %s)msgr )r< ExceptionrZ ru )
Z
url_reader_cburlsr exc_cblog_cbr r rM r r@ Z
time_takenZ max_wait_strZ
status_msg)r r r r read_url_handle_exceptions. s. z0wait_for_url..read_url_handle_exceptionsc s$ t | d kri n | |ddS )NF)r/ rx ra ry r{ )r= )r ra )rw rx r{ r r read_url_cbN s z!wait_for_url..read_url_cbc s fdd}xnD ]f}t j }dkrb| r6dS dk rb rb| | krbt| | ||| ||}|r|S qW dS )z|iterate over list of urls, request each one and handle responses
and thrown exceptions individually per url
c s | | fS )Nr )r )r ra r r url_reader_serial] s z@wait_for_url..read_url_serial..url_reader_serialr N)ru rZ )r ra r r r r Znowout)loop_nr r r r r )ra r read_url_serialX s
z%wait_for_url..read_url_serialc s, t t |d}|| ||}|r(|S dS )zpass list of urls to dual_stack which sends requests in parallel
handle response and exceptions of the first endpoint to respond
)r ra N)r r )r ra r r Zurl_reader_parallelr )r r r r r r read_url_parallels s z'wait_for_url..read_url_parallelr Nr z1Please wait %s seconds while we wait to try againz$Timed out, no response from urls: %sF)FN)rZ ru r0 r5 rr rv r )r r ra r rw rx r rz r r{ r r r r r r Z do_read_urlZcalculate_sleep_timerM r Zaddressr ) r r rw rx r r r r r{ r r r wait_for_url s6 4
r c @ s^ e Zd ZdddZdd Zdd Zd d
Zd d Zd
d Zdd Z dd Z
dd Z dd Z dS )OauthUrlHelperN/run/oauth_skew.jsonc C st || _ |p d| _|| _|| _|| _d| _d| _| j| j| j f}t|sNd| _nt|s^t d| j
}|pli | _ d S )Nr Tr FzBall or none of token_key, token_secret, or consumer_key can be set) consumer_keyconsumer_secret token_key token_secretskew_data_file _do_oauthskew_change_limitanyallr read_skew_file skew_data)rD r r r r r Zrequiredoldr r r rE s
zOauthUrlHelper.__init__c C s8 | j r4tjj| j r4t| j dd }tj|S Q R X d S )Nr~ )mode)r osr$ isfiler8 jsonload)rD r? r r r r s z OauthUrlHelper.read_skew_filec
C sT | j s
d S | j }|d kr i }|||< t| j dd}|jtj| W d Q R X d S )Nw)r )r r r8 writer dumps)rD hostvalueZcurr? r r r update_skew_file s z OauthUrlHelper.update_skew_filec C s t |to |jdkp |jdks"d S d|jkr>tjd|j d S |jd }ytjt|}W n. t k
r } ztjd|| d S d }~X nX t
|tj }t |j j
}| jj|d}t|| | jkr| j|| tjd|| || j|< d S )Ni i datez$Missing header 'date' in %s responsez#Failed to convert datetime '%s': %sr z$Setting oauth clockskew for %s to %d)rj r< r. r/ r5 r6 ru Zmktimer r rZ r r netlocr r4 absr r ) rD r r r Z remote_timer@ Zskewr Zold_skewr r r rz s(
zOauthUrlHelper.exception_cbc C sZ | j s
i S d }t|j}| jr>|| jkr>ttj | j| }t|| j| j| j | j
|dS )N)r r r r r timestamp) r r r r rZ ru
oauth_headersr r r r )rD r r r r r r rw s
zOauthUrlHelper.headers_cbc C s6 t | j|jd|d< t | j|jd|d< |||S )Nrw rz )r _headers_cbr4
_exception_cb)rD Z wrapped_funcrF r> r r r _wrapped s
zOauthUrlHelper._wrappedc O s | j t||S )N)r r )rD rF r> r r r r s zOauthUrlHelper.wait_for_urlc O s | j t||S )N)r r= )rD rF r> r r r r= s zOauthUrlHelper.readurlc C s* d }z|r|||}W d | j || X |S )N)rz )rD Zextra_exception_cbr r retr r r r s z OauthUrlHelper._exception_cbc C s$ i }|r||}|j | j| |S )N)ri rw )rD Zextra_headers_cbr r/ r r r r s
zOauthUrlHelper._headers_cb)NNNNr )
rJ rK rL rE r r rz rw r r r= r r r r r r r s
r c C sz ydd l j}W n, tk
r: } ztd|W Y d d }~X nX |rJt|}nd }|j|||||j|d}|j| \} }
} |
S )Nr z oauth support is not available)Z
client_secretZresource_owner_keyZresource_owner_secretZsignature_methodr )Zoauthlib.oauth1oauth1 ImportErrorNotImplementedErrorr" ZClientZSIGNATURE_PLAINTEXTZsign) r r r r r r r r@ ZclientZ_uriZsigned_headersZ_bodyr r r r & s
r )r( r) )NNr r NNNNTTNNFTr F)N)r r )N)=rp r Zloggingr r ru concurrent.futuresr r r Z email.utilsr r; r functoolsr Z http.clientr itertoolsr Ztypingr
r r r
r r r Z urllib.parser r r rW r Z cloudinitr Z getLoggerrJ r5 r r r' rA rB r( r) r: r< r] r[ r= r" rZ r rm r r rr r r r r r r r s~ $
1 ( P8 F|