U
ʗReMO @ sL d Z ddlZddlZddlZddlZddlZddlZddlZddl Z ddl
Z ddl Z ddl
ZddlmZ ddlmZ ddlmZmZmZmZmZmZmZmZm Z m Z m Z ddl m Z ddl!m"Z" ddl#m$Z$m%Z% dd l&m'Z' dd
l(m)Z) dd l*m+Z+ dd l,m-Z- dd
l.m/Z/ ddl0m1Z1 ddl2m3Z3m4Z4 ddl5m6Z6 ddl7m8Z8m9Z9m:Z: er\ddlm;Z; neZ?ej@jAjBZCeeDeDf ZEeDeeD dddZFG dd deGZHe"ddddZIG d d d eGZJeDe-dd d d ZKeDe-e"d d!d"ZLeEeeD d#d$d%ZMeDeDd&d'd(ZNeDeDd&d)d*ZOe Pd+e jQZReDeSeDd,d-d.ZTeDeDdd/d0ZUeeDeeD f eDeDee) d1d2d3ZVG d4d5 d5ZWG d6d7 d7e;ZXeXeXd8d9d:ZYeYd;ee) d<d=d>ZZG d?d; d;Z[G d@dA dAeZ\dQe)e eDeGf eedB ddCdDdEZ]dRe"eSe[dGdHdIZ^dSe)ee- ed; dJdKdLZ_G dMdN dNeZ`G dOdP dPZadS )TzO
The main purpose of this module is to expose LinkCollector.collect_sources().
N)
HTMLParser)Values)
TYPE_CHECKINGCallableDictIterableListMutableMapping
NamedTupleOptionalSequenceTupleUnion)requests)Response)
RetryErrorSSLError)NetworkConnectionError)Link) SearchScope)
PipSession)raise_for_status)is_archive_file)pairwiseredact_auth_from_url)vcs )CandidatesFromPage
LinkSource build_source)Protocolurlreturnc C s6 t jD ]*}| |r| t| dkr| S qdS )zgLook for VCS schemes in the URL.
Returns the matched VCS scheme, or None if there's no match.
z+:N)r schemeslower
startswithlen)r" scheme r) /builddir/build/BUILDROOT/alt-python38-pip-22.2.1-2.el8.x86_64/opt/alt/python38/lib/python3.8/site-packages/pip/_internal/index/collector.py_match_vcs_scheme: s
r+ c s& e Zd Zeedd fdd Z ZS )_NotAPIContentN) content_type request_descr# c s t || || _|| _d S N)super__init__r- r. )selfr- r. __class__r) r* r1 F s z_NotAPIContent.__init__)__name__
__module__ __qualname__strr1
__classcell__r) r) r3 r* r, E s r, )responser# c C s6 | j dd}| }|dr$dS t|| jjdS )z
Check the Content-Type header to ensure the response contains a Simple
API Response.
Raises `_NotAPIContent` if the content type is not a valid content-type.
Content-TypeUnknown)z text/htmlz#application/vnd.pypi.simple.v1+html#application/vnd.pypi.simple.v1+jsonN)headersgetr% r& r, requestmethod)r: r- content_type_lr) r) r* _ensure_api_headerL s rC c @ s e Zd ZdS )_NotHTTPN)r5 r6 r7 r) r) r) r* rD b s rD )r" sessionr# c C sF t j| \}}}}}|dkr$t |j| dd}t| t| dS )z
Send a HEAD request to the URL, and ensure the response contains a simple
API Response.
Raises `_NotHTTP` if the URL is not available for a HEAD request, or
`_NotAPIContent` if the content type is not a valid content type.
> httphttpsT)allow_redirectsN)urllibparseurlsplitrD headr rC )r" rE r( netlocpathqueryfragmentrespr) r) r* _ensure_api_responsef s rR c C sz t t| jrt| |d tdt| |j| ddddgddd }t | t
| td
t| |j d d |S )
aY Access an Simple API response with GET, and return the response.
This consists of three parts:
1. If the URL looks suspiciously like an archive, send a HEAD first to
check the Content-Type is HTML or Simple API, to avoid downloading a
large file. Raise `_NotHTTP` if the content type cannot be determined, or
`_NotAPIContent` if it is not HTML or a Simple API.
2. Actually perform the request. Raise HTTP exceptions on network failures.
3. Check the Content-Type header to make sure we got a Simple API response,
and raise `_NotAPIContent` otherwise.
rE zGetting page %sz, r= z*application/vnd.pypi.simple.v1+html; q=0.1ztext/html; q=0.01z max-age=0)Acceptz
Cache-Control)r> zFetched page %s as %sr; r< ) r r filenamerR loggerdebugr r? joinr rC r> )r" rE rQ r) r) r* _get_simple_responsex s,
rY )r> r# c C s< | r8d| kr8t j }| d |d< |d}|r8t|S dS )z=Determine if we have any encoding information in our headers.r; z content-typecharsetN)emailmessageMessage get_paramr8 )r> mrZ r) r) r* _get_encoding_from_headers s
r` )partr# c C s t jt j| S )zP
Clean a "part" of a URL path (i.e. after splitting on "@" characters).
)rI rJ quoteunquotera r) r) r* _clean_url_path_part s re c C s t jt j| S )z
Clean the first part of a URL path that corresponds to a local
filesystem path (i.e. the first part after splitting on "@" characters).
)rI r@ pathname2url url2pathnamerd r) r) r* _clean_file_url_path s
rh z(@|%2F))rN
is_local_pathr# c C s^ |r
t }nt}t| }g }tt|dgD ]$\}}||| || q.d |S )z*
Clean the path portion of a URL.
)
rh re _reserved_chars_resplitr itertoolschainappendupperrX )rN ri
clean_funcparts
cleaned_partsto_cleanreservedr) r) r* _clean_url_path s
rv c C s6 t j| }|j }t|j|d}t j|j|dS )z
Make sure a link is fully quoted.
For example, if ' ' occurs in the URL, it will be replaced with "%20",
and without double-quoting other characters.
)ri )rN )rI rJ urlparserM rv rN
urlunparse_replace)r" resultri rN r) r) r* _clean_link s r{ )element_attribspage_urlbase_urlr# c C sL | d}|sdS ttj||}| d}| d}t||||d}|S )zW
Convert an anchor element's attributes in a simple repository page to a Link.
hrefNzdata-requires-pythonz data-yanked)
comes_fromrequires_python
yanked_reason)r? r{ rI rJ urljoinr )r| r} r~ r r" pyrequirer linkr) r) r* _create_link_from_element s
r c @ s: e Zd ZdddddZeedddZed d
d ZdS ) CacheablePageContent IndexContentNpager# c C s |j s
t|| _d S r/ )cache_link_parsingAssertionErrorr r2 r r) r) r* r1 s
z CacheablePageContent.__init__)otherr# c C s t |t| o | jj|jjkS r/ )
isinstancetyper r" )r2 r r) r) r* __eq__ s zCacheablePageContent.__eq__r# c C s t | jjS r/ )hashr r" r2 r) r) r* __hash__" s z CacheablePageContent.__hash__) r5 r6 r7 r1 objectboolr intr r) r) r) r* r s r c @ s e Zd Zdee dddZdS )
ParseLinksr r c C s d S r/ r) r r) r) r* __call__' s zParseLinks.__call__N)r5 r6 r7 r r r r) r) r) r* r & s r )fnr# c sL t jddttt d fdd t dtt d fdd }|S )
z
Given a function that parses an Iterable[Link] from an IndexContent, cache the
function's result (keyed by CacheablePageContent), unless the IndexContent
`page` has `page.cache_link_parsing == False`.
N)maxsize)cacheable_pager# c s t | jS r/ )listr )r )r r) r* wrapper2 s z*with_cached_index_content..wrapperr r c s | j rt| S t | S r/ )r r r )r r r r) r* wrapper_wrapper6 s z2with_cached_index_content..wrapper_wrapper) functools lru_cacher r r wraps)r r r) r r* with_cached_index_content+ s
r r r c
c s | j }|drt| j}|dg D ]r}|d}|dkrDq,|d}|rbt|tsbd}n|sjd}t t
t j
| j|| j|d||di d V q,t| j}| jpd
}|| j| | j}|jp|} |jD ]"}
t|
|| d } | dkrq| V qdS ) z\
Parse a Simple API's Index Content, and yield its anchor elements as Link objects.
r= filesr" Nyankedrj zrequires-pythonhashes)r r r r zutf-8)r} r~ )r- r% r& jsonloadscontentr? r r8 r r{ rI rJ r r" HTMLLinkParserencodingfeeddecoder~ anchorsr ) r rB datafilefile_urlr parserr r" r~ anchorr r) r) r* parse_links? sB
r c @ s< e Zd ZdZd
eeee eeddddZeddd Z dS ) r z5Represents one response (or page), along with its URLTN)r r- r r" r r# c C s" || _ || _|| _|| _|| _dS )am
:param encoding: the encoding to decode the given content.
:param url: the URL from which the HTML was downloaded.
:param cache_link_parsing: whether links parsed from this page's url
should be cached. PyPI index urls should
have this set to False, for example.
N)r r- r r" r )r2 r r- r r" r r) r) r* r1 q s
zIndexContent.__init__r c C s
t | jS r/ )r r" r r) r) r* __str__ s zIndexContent.__str__)T)
r5 r6 r7 __doc__bytesr8 r r r1 r r) r) r) r* r n s c sn e Zd ZdZedd fdd Zeeeeee f ddddZ eeeee f ee d d
d Z
Z S ) r zf
HTMLParser that keeps the first base HREF and a list of all anchor
elements' attributes.
Nr! c s$ t jdd || _d | _g | _d S )NT)convert_charrefs)r0 r1 r" r~ r )r2 r" r3 r) r* r1 s zHTMLLinkParser.__init__)tagattrsr# c C sH |dkr,| j d kr,| |}|d k rD|| _ n|dkrD| jt| d S )Nbasea)r~ get_hrefr ro dict)r2 r r r r) r) r* handle_starttag s
z HTMLLinkParser.handle_starttag)r r# c C s" |D ]\}}|dkr| S qd S )Nr r) )r2 r namevaluer) r) r* r s
zHTMLLinkParser.get_href) r5 r6 r7 r r8 r1 r r
r r r r9 r) r) r3 r* r s "r ).N)r reasonmethr# c C s |d krt j}|d| | d S )Nz%Could not fetch URL %s: %s - skipping)rV rW )r r r r) r) r* _handle_get_simple_fail s r T)r: r r# c C s& t | j}t| j| jd || j|dS )Nr; )r r" r )r` r> r r r" )r: r r r) r) r* _make_index_content s
r )r rE r# c
C s |d krt d| jddd }t|}|r@td|| d S tj|\}}}}}}|dkrt j
tj
|r|ds|d7 }tj|d}td | zt||d
}W nF tk
r td | Y n4 tk
r
} ztd | |j|j W 5 d }~X Y n tk
r6 } zt| | W 5 d }~X Y n tk
rb } zt| | W 5 d }~X Y n tk
r } z$d
} | t|7 } t| | tjd W 5 d }~X Y nd t j k
r } zt| d| W 5 d }~X Y n0 t j k
r t| d Y nX t || j dS d S )Nz?_get_html_page() missing 1 required keyword argument: 'session'#r r zICannot look at %s URL %s because it does not support lookup as web pages.r /z
index.htmlz# file: URL is directory, getting %srS z`Skipping page %s because it looks like an archive, and cannot be checked by a HTTP HEAD request.zSkipping page %s because the %s request got Content-Type: %s. The only supported Content-Types are application/vnd.pypi.simple.v1+json, application/vnd.pypi.simple.v1+html, and text/htmlz4There was a problem confirming the ssl certificate: )r zconnection error: z timed out)r )! TypeErrorr" rl r+ rV warningrI rJ rw osrN isdirr@ rg endswithr rW rY rD r, r. r- r r r r r8 infor ConnectionErrorTimeoutr r )
r rE r"
vcs_schemer( _rN rQ excr r) r) r* _get_index_content s^
r c @ s. e Zd ZU eee ed< eee ed< dS )CollectedSources
find_links
index_urlsN)r5 r6 r7 r r r __annotations__r) r) r) r* r s
r c @ sx e Zd ZdZeeddddZedeee d ddd Z
e e e
d
d d Zeee d
ddZe
eedddZdS )
LinkCollectorz
Responsible for collecting Link objects from all configured locations,
making network requests as needed.
The class's main method is its collect_sources() method.
N)rE search_scoper# c C s || _ || _d S r/ )r rE )r2 rE r r) r) r* r1 s zLinkCollector.__init__F)rE optionssuppress_no_indexr# c C s` |j g|j }|jr8|s8tdddd |D g }|jp@g }tj||d}t ||d}|S )z
:param session: The Session to use to make requests.
:param suppress_no_index: Whether to ignore the --no-index option
when constructing the SearchScope object.
zIgnoring indexes: %s,c s s | ]}t |V qd S r/ )r ).0r" r) r) r* ' s z'LinkCollector.create..r r )rE r )
index_urlextra_index_urlsno_indexrV rW rX r r creater )clsrE r r r r r link_collectorr) r) r* r s"
zLinkCollector.creater c C s | j jS r/ )r r r r) r) r* r 8 s zLinkCollector.find_links)locationr# c C s t || jdS )z>
Fetch an HTML page containing package links.
rS )r rE )r2 r r) r) r* fetch_response< s z LinkCollector.fetch_response) project_namecandidates_from_pager# c s t fddj|D }t fddjD }ttj rdd t
||D }t | d| dg| }t
d| tt|t|d S )
Nc 3 s$ | ] }t | jjd d dV qdS )Fr page_validator
expand_dirr Nr rE is_secure_originr locr r2 r) r* r H s z0LinkCollector.collect_sources..c 3 s$ | ] }t | jjd d dV qdS )Tr Nr r r r) r* r R s c S s* g | ]"}|d k r|j d k rd|j qS )Nz* )r )r sr) r) r*
^ s
z1LinkCollector.collect_sources..z' location(s) to search for versions of :
r ) collections OrderedDictr get_index_urls_locationsvaluesr rV isEnabledForloggingDEBUGrm rn r' rW rX r r )r2 r r index_url_sourcesfind_links_sourceslinesr) r r* collect_sourcesB s&
z LinkCollector.collect_sources)F)r5 r6 r7 r r r r1 classmethodr r r propertyr r8 r r r r r r r r r) r) r) r* r s( r )N)T)N)br r
email.messager[ r rm r r r re urllib.parserI urllib.requestxml.etree.ElementTreexml html.parserr optparser typingr r r r r r r
r r r
r pip._vendorr Zpip._vendor.requestsr Z pip._vendor.requests.exceptionsr r pip._internal.exceptionsr pip._internal.models.linkr !pip._internal.models.search_scoper pip._internal.network.sessionr pip._internal.network.utilsr pip._internal.utils.filetypesr pip._internal.utils.miscr r pip._internal.vcsr sourcesr r r r r getLoggerr5 rV etree ElementTreeElement HTMLElementr8 ResponseHeadersr+ Exceptionr, rC rD rR rY r` re rh compile
IGNORECASErk r rv r{ r r r r r r r r r r r r r) r) r) r* s 4
? .
D