1j? d Z ddlZddlZddlZddlZddlZddlZddlZddl Z
ddl Z
ddl m
Z
ddlmZ ddlmZ ddlmZmZmZmZmZmZmZmZmZm Z m Z ddl m Z ddl m!Z! dd l"m#Z#m$Z$ dd
l%m&Z& dd l'm(Z( dd l)m*Z* dd
l+m,Z, ddl-m.Z. ddl/m0Z0 ddl1m2Z2 ddl3m4Z4 ddl5m6Z6m7Z7m8Z8 ej9 e: Z;ee G d de? Z@de!ddfdZA G d d e? ZBdezO
The main purpose of this module is to expose LinkCollector.collect_sources().
N) dataclass)
HTMLParser)Values) CallableDictIterableListMutableMapping
NamedTupleOptionalProtocolSequenceTupleUnion)requests)Response)
RetryErrorSSLError)NetworkConnectionError)Link) SearchScope)
PipSession)raise_for_status)is_archive_fileredact_auth_from_url)vcs )CandidatesFromPage
LinkSource build_sourceurlreturnc t j D ]D}| | r| t | dv r|c S EdS )zgLook for VCS schemes in the URL.
Returns the matched VCS scheme, or None if there's no match.
z+:N)r schemeslower
startswithlen)r" schemes o/builddir/build/BUILD/cloudlinux-venv-1.0.11/venv/lib/python3.11/site-packages/pip/_internal/index/collector.py_match_vcs_schemer+ 3 sW
+ 99;; ! !& ) ) c#f++.>$.F.FMMM 4 c , e Zd Zdededdf fd Z xZS )_NotAPIContent content_type request_descr# Nc h t || || _ || _ d S N)super__init__r/ r0 )selfr/ r0 __class__s r* r4 z_NotAPIContent.__init__? s3
|444 ( (r, )__name__
__module__ __qualname__strr4
__classcell__r6 s @r* r. r. > sR )S ) ) ) ) ) ) ) ) ) ) ) )r, r. responsec | j dd }| }| d rdS t || j j )z
Check the Content-Type header to ensure the response contains a Simple
API Response.
Raises `_NotAPIContent` if the content type is not a valid content-type.
Content-TypeUnknown)z text/htmlz#application/vnd.pypi.simple.v1+html#application/vnd.pypi.simple.v1+jsonN)headersgetr& r' r. requestmethod)r= r/ content_type_ls r* _ensure_api_headerrG E sg #'' BBL!''))N
x'7'>
?
??r, c e Zd ZdS )_NotHTTPN)r7 r8 r9 r, r* rI rI [ s Dr, rI sessionc t j | \ }}}}}|dvrt | | d }t | t
| dS )z
Send a HEAD request to the URL, and ensure the response contains a simple
API Response.
Raises `_NotHTTP` if the URL is not available for a HEAD request, or
`_NotAPIContent` if the content type is not a valid content type.
> httphttpsT)allow_redirectsN)urllibparseurlsplitrI headr rG )r" rK r) netlocpathqueryfragmentresps r* _ensure_api_responserY _ sq -3L,A,A#,F,F)FFD%
&&&jj <<T< 2 2DTt r, c t t | j rt | | t dt
| | | d g d dd }t | t | t dt
| |j d d
|S ) aY Access an Simple API response with GET, and return the response.
This consists of three parts:
1. If the URL looks suspiciously like an archive, send a HEAD first to
check the Content-Type is HTML or Simple API, to avoid downloading a
large file. Raise `_NotHTTP` if the content type cannot be determined, or
`_NotAPIContent` if it is not HTML or a Simple API.
2. Actually perform the request. Raise HTTP exceptions on network failures.
3. Check the Content-Type header to make sure we got a Simple API response,
and raise `_NotAPIContent` otherwise.
rK zGetting page %sz, )rA z*application/vnd.pypi.simple.v1+html; q=0.1ztext/html; q=0.01z max-age=0)Acceptz
Cache-Control)rB zFetched page %s as %sr? r@ ) r r filenamerY loggerdebugr rC joinr rG rB )r" rK rX s r* _get_simple_responsera q s tCyy)** 3 S'2222
LL"$8$=$=>>> ;; ii ( )+
D4 T t
LL S!! 33 Kr, rB c | rSd| v rOt j }| d |d< | d }|rt | S dS )z=Determine if we have any encoding information in our headers.r? z content-typecharsetN)emailmessageMessage get_paramr: )rB mrc s r* _get_encoding_from_headersri s` >W,, M ! ! # # #N 3.++i(( w<< 4r, c 2 e Zd Zd dZdedefdZdefdZdS )
CacheablePageContentpage IndexContentr# Nc & |j sJ || _ d S r2 )cache_link_parsingrl r5 rl s r* r4 z CacheablePageContent.__init__ s &&&& r, otherc p t |t | o| j j |j j k S r2 )
isinstancetyperl r" )r5 rq s r* __eq__zCacheablePageContent.__eq__ s* %d,,P %*.1PPr, c 4 t | j j S r2 )hashrl r" r5 s r* __hash__z CacheablePageContent.__hash__ s DIM"""r, )rl rm r# N) r7 r8 r9 r4 objectboolru intry rJ r, r* rk rk sk QF Qt Q Q Q Q## # # # # # #r, rk c * e Zd Zdddee fdZdS )
ParseLinksrl rm r# c d S r2 rJ rp s r* __call__zParseLinks.__call__ s r, N)r7 r8 r9 r r r rJ r, r* r~ r~ s* C^CCCCCCCr, r~ fnc t j d dt dt t f fd t j dddt t f fd }|S ) z
Given a function that parses an Iterable[Link] from an IndexContent, cache the
function's result (keyed by CacheablePageContent), unless the IndexContent
`page` has `page.cache_link_parsing == False`.
N)maxsizecacheable_pager# c > t | j S r2 )listrl )r r s r* wrapperz*with_cached_index_content..wrapper s BB~*++,,,r, rl rm c r | j r t | S t | S r2 )ro rk r )rl r r s r* wrapper_wrapperz2with_cached_index_content..wrapper_wrapper s; " 77/5566 6BBtHH~~ r, ) functools lru_cacherk r r wraps)r r r s ` @r* with_cached_index_contentr s &&&- 4 -d - - - - - '&- _R n d
r, rl rm c # K | j }| d rUt j | j }| dg D ]#}t j || j }| |V $dS t | j }| j pd}| | j
| | j }|j p|}|j D ] } t j | || }| |V !dS )z\
Parse a Simple API's Index Content, and yield its anchor elements as Link objects.
rA filesNzutf-8)page_urlbase_url)r/ r& r' jsonloadscontentrC r from_jsonr" HTMLLinkParserencodingfeeddecoder anchors from_element)
rl rF datafilelinkparserr r" r anchors
r* parse_linksr s! &,,..N !FGG z$,''HHWb)) D>$11D|JJJJ
DH
%
%F}'H
KK ##H--...
(C%#H. #III <
r, T)frozenc b e Zd ZU dZeed< eed< ee ed< eed< dZe ed< defd Z
d
S ) rm a Represents one response (or page), along with its URL.
:param encoding: the encoding to decode the given content.
:param url: the URL from which the HTML was downloaded.
:param cache_link_parsing: whether links parsed from this page's url
should be cached. PyPI index urls should
have this set to False, for example.
r r/ r r" Tro r# c * t | j S r2 )r r" rx s r* __str__zIndexContent.__str__
s #DH---r, N) r7 r8 r9 __doc__bytes__annotations__r: r ro r{ r rJ r, r* rm rm s| NNNsm HHH ####. . . . . . .r, c e Zd ZdZdeddf fd Zdedeeeee f ddfdZ deeeee f dee fd Z
xZ S )
r zf
HTMLParser that keeps the first base HREF and a list of all anchor
elements' attributes.
r" r# Nc v t d || _ d | _ g | _ d S )NT)convert_charrefs)r3 r4 r" r r )r5 r" r6 s r* r4 zHTMLLinkParser.__init__ s7
$///'+
79 r, tagattrsc |dk r)| j "| | }| || _ d S d S |dk r)| j t | d S d S )Nbasea)r get_hrefr appenddict)r5 r r hrefs r* handle_starttagz HTMLLinkParser.handle_starttag sn &==T] 2==''D $
CZZ L U , , , , , Zr, c * |D ]\ }}|dk r|c S d S )Nr rJ )r5 r namevalues r* r zHTMLLinkParser.get_href& s1 KD%v~~ tr, ) r7 r8 r9 r r: r4 r r r r r r; r< s @r* r r s
:C :D : : : : : :-3 -tE#x}:L4M/N -SW - - - -d5hsm);#< = (3- r, r r reasonmeth).Nc < | t j } |d| | d S )Nz%Could not fetch URL %s: %s - skipping)r^ r_ )r r r s r* _handle_get_simple_failr - s+
||D 0$?????r, ro c | t | j }t | j | j d || j | S )Nr? )r r" ro )ri rB rm r r" )r= ro r s r* _make_index_contentr 7 sG *(*:;;H ( L- r, c | j dd d }t | }|r t d|| d S t
j | \ }}}}}}|dk rt j
t
j | rU|
d s|dz
}t
j |d }t d| t! || }t# || j
S # t& $ r t d | Y n
t( $ r1}t d | |j |j Y d }~nd }~wt. $ r}t1 | | Y d }~nd }~wt2 $ r}t1 | | Y d }~nd }~wt4 $ r:}d
} | t7 | z
} t1 | | t j Y d }~nRd }~wt: j $ r }t1 | d| Y d }~n(d }~wt: j $ r t1 | d Y nw xY wd S )N#r r zICannot look at %s URL %s because it does not support lookup as web pages.r /z
index.htmlz# file: URL is directory, getting %sr[ )ro z`Skipping page %s because it looks like an archive, and cannot be checked by a HTTP HEAD request.zSkipping page %s because the %s request got Content-Type: %s. The only supported Content-Types are application/vnd.pypi.simple.v1+json, application/vnd.pypi.simple.v1+html, and text/htmlz4There was a problem confirming the ssl certificate: )r zconnection error: z timed out) r" splitr+ r^ warningrP rQ urlparseosrU isdirrD url2pathnameendswithurljoinr_ ra r ro rI r. r0 r/ r r r r r: infor ConnectionErrorTimeout)
r rK r"
vcs_schemer) _rU rX excr s
r* _get_index_contentr D s
(..a
#C #3''J W
t &| 4 4S 9 9 FAtQ1
BGMM&.*E*Ed*K*K L L ||C 3JC
l""3 55 :C@@@ U#C999: #4D *-....*-......r, r c
e Zd ZdZdededdfdZe ddeded e dd fd
Z
e de e
fd Zd edee fd
Zde
dedefdZdS )
LinkCollectorz
Responsible for collecting Link objects from all configured locations,
making network requests as needed.
The class's main method is its collect_sources() method.
rK search_scoper# Nc " || _ || _ d S r2 )r rK )r5 rK r s r* r4 zLinkCollector.__init__ s
) r, Foptionssuppress_no_indexc |j g|j z }|j r<|s:t dd d |D g }|j pg }t j |||j }t || }|S )z
:param session: The Session to use to make requests.
:param suppress_no_index: Whether to ignore the --no-index option
when constructing the SearchScope object.
zIgnoring indexes: %s,c 3 4 K | ]}t | V d S r2 r ).0r" s r* z'LinkCollector.create.. s+ IIs-c22IIIIIIr, )r r no_index)rK r )
index_urlextra_index_urlsr r^ r_ r` r r creater )clsrK r r r r r link_collectors r* r zLinkCollector.create s '(7+CC
$5 LL&IIjIIIII
J '-2
")!! %
'%
r, c | j j S r2 )r r rx s r* r zLinkCollector.find_links s ++r, locationc . t || j S )z>
Fetch an HTML page containing package links.
r[ )r rK )r5 r s r* fetch_responsez LinkCollector.fetch_response s "(DLAAAAr, project_namecandidates_from_pagec z t j fd j D }t j fd j D }t t j red t j || D }t | d dg|z }t
d | t t! | t! | S )Nc 3 V K | ]#}t |j j d d V $dS )Fr page_validator
expand_dirro r Nr! rK is_secure_originr locr r r5 s r* r z0LinkCollector.collect_sources.. s_
4
4
%9 #| < #( )
4
4
4
4
4
4
r, c 3 V K | ]#}t |j j d d V $dS )Tr Nr r s r* r z0LinkCollector.collect_sources.. s_
5
5
%9 #| < #' )
5
5
5
5
5
5
r, c 4 g | ]}||j d|j S )Nz* )r )r ss r*
z1LinkCollector.collect_sources.. s6 =QV%7 QV
%7%7%7r, z' location(s) to search for versions of :
)r r ) collections OrderedDictr get_index_urls_locationsvaluesr r^ isEnabledForloggingDEBUG itertoolschainr( r_ r` r r )r5 r r index_url_sourcesfind_links_sourcesliness ``` r* collect_sourcesz LinkCollector.collect_sources s ( 3
4
4
4
4
4
4
(AA,OO
4
4
4
&(( ) 4
5
5
5
5
5
5
5
5
5
&(( w} - -
+ " );=NOO E u:: 3 3#/3 3 3 E
LL5)) * * * .// -..
r, )F)r7 r8 r9 r r r r4 classmethodr r{ r propertyr r: r r r rm r r r r rJ r, r* r r s' "
#(
[ B ,DI , , , X,Bt B 0F B B B B ,
,
1,
,
,
,
,
,
,
r, r r2 )T)Rr r
email.messagerd r r r r r urllib.parserP urllib.request dataclassesr html.parserr optparser typingr r r r r
r r r
r r r pip._vendorr pip._vendor.requestsr pip._vendor.requests.exceptionsr r pip._internal.exceptionsr pip._internal.models.linkr !pip._internal.models.search_scoper pip._internal.network.sessionr pip._internal.network.utilsr pip._internal.utils.filetypesr pip._internal.utils.miscr pip._internal.vcsr sourcesr r r! getLoggerr7 r^ r: ResponseHeadersr+ Exceptionr. rG rI rY ra ri rk r~ r r rm r r r{ r r r r rJ r, r* r s ! ! ! ! ! ! " " " " " " ! ) ) ) ) ) ) @ @ @ @ @ @ @ @ ; ; ; ; ; ; * * * * * * 9 9 9 9 9 9 4 4 4 4 4 4 8 8 8 8 8 8 9 9 9 9 9 9 9 9 9 9 9 9 ! ! ! ! ! ! A A A A A A A A A A 8 $ $ c*3 8C= ) ) ) ) )Y ) ) )@ @d @ @ @ @, y c J 4 $<c <J <8 < < < <~ HSM # # # # # # # #D D D D D D D D* ( n $ 8 $. . . . . . . .( Z > +/@ @
@ #y. !@ 8I&
'@
@ @ @ @ 48
,0
:T :z :h~>V : : : :z/ / / / /z / / /
h
h
h
h
h
h
h
h
h
h
r,