2bix@ D S r SSKrSSKrSSKrSSKrSSKrSSKrSSKrSSK r
SSK r
SSK J
r
SSKJr SSKJrJrJrJrJrJrJrJrJrJrJr SSK J r SSK J r SSK J!r!J"r" SS K#J$r$ SS
K%J&r& SS K'J(r( SS K)J*r* SS
K+J,r, SSK-J.r. SSK/J0r0 SSK1J2r2 SSK3J4r4J5r5J6r6 \( a SSKJ7r7 O\8r7\Rr " \:5 r;\\<\<4 r=S\ " S S\?5 r@S\ SS4S jrA " S S \?5 rBS\$.FM c 8 ^ \ rS rSrS\S\SS4U 4S jjrSrU =r$ )_NotAPIContentB content_type request_descr# Nc < > [ TU ] X5 Xl X l g N)super__init__r0 r1 )selfr0 r1 __class__s r* r5 _NotAPIContent.__init__C s
4 ( (r, )r0 r1 )__name__
__module__ __qualname____firstlineno__strr5 __static_attributes__
__classcell__r7 s @r* r. r. B s" )S ) ) ) )r, r. responsec U R R SS5 nUR 5 nUR S5 ( a g[ XR
R 5 e)z
Check the Content-Type header to ensure the response contains a Simple
API Response.
Raises `_NotAPIContent` if the content type is not a valid content-type.
Content-TypeUnknown)z text/htmlz#application/vnd.pypi.simple.v1+html#application/vnd.pypi.simple.v1+jsonN)headersgetr& r' r. requestmethod)rA r0 content_type_ls r* _ensure_api_headerrK I s^ ##'' BL!'')N
'7'7'>'>
??r, c \ rS rSrSrg)_NotHTTP_ N)r9 r: r; r< r> rO r, r* rM rM _ s r, rM sessionc [ R R U 5 u p#pEnUS; a
[ 5 eUR U SS9n[ U5 [
U5 g)z
Send a HEAD request to the URL, and ensure the response contains a simple
API Response.
Raises `_NotHTTP` if the URL is not available for a HEAD request, or
`_NotAPIContent` if the content type is not a valid content type.
> httphttpsT)allow_redirectsN)urllibparseurlsplitrM headr rK )r" rP r) netlocpathqueryfragmentresps r* _ensure_api_responser^ c sR -3LL,A,A#,F)FD
&&j <<T< 2DTt r, c [ [ U 5 R 5 ( a [ XS9 [ R S[
U 5 5 UR U SR / SQ5 SS.S9n[ U5 [ U5 [ R S[
U 5 UR R S S
5 5 U$ ) a5 Access an Simple API response with GET, and return the response.
This consists of three parts:
1. If the URL looks suspiciously like an archive, send a HEAD first to
check the Content-Type is HTML or Simple API, to avoid downloading a
large file. Raise `_NotHTTP` if the content type cannot be determined, or
`_NotAPIContent` if it is not HTML or a Simple API.
2. Actually perform the request. Raise HTTP exceptions on network failures.
3. Check the Content-Type header to make sure we got a Simple API response,
and raise `_NotAPIContent` otherwise.
rP zGetting page %sz, )rE z*application/vnd.pypi.simple.v1+html; q=0.1ztext/html; q=0.01z max-age=0)Acceptz
Cache-Control)rF zFetched page %s as %srC rD ) r r filenamer^ loggerdebugr rG joinr rK rF )r" rP r] s r* _get_simple_responserf u s tCy))** S2
LL"$8$=> ;; ii( )+
D4 T t
LL S! 3 Kr, rF c U ( aO SU ; aI [ R R 5 nU S US' UR S5 nU( a [ U5 $ g)z=Determine if we have any encoding information in our headers.rC z content-typecharsetN)emailmessageMessage get_paramr= )rF mrh s r* _get_encoding_from_headersrn sK >W, MM ! ! # #N 3.++i( w< r, c > \ rS rSrS S jrS\S\4S jrS\4S jr Sr
g)
CacheablePageContent r# Nc 6 UR ( d eXl g r3 )cache_link_parsingpager6 rt s r* r5 CacheablePageContent.__init__ s &&&& r, otherc [ U[ U 5 5 =( a- U R R UR R :H $ r3 )
isinstancetypert r" )r6 rw s r* __eq__CacheablePageContent.__eq__ s- %d,P %**..1PPr, c @ [ U R R 5 $ r3 )hashrt r" r6 s r* __hash__ CacheablePageContent.__hash__ s DIIMM""r, )rt )rt IndexContentr# N) r9 r: r; r< r5 objectboolr{ intr r> rO r, r* rp rp s) QF Qt Q## #r, rp c , \ rS rSrSSS\\ 4S jrSrg)
ParseLinks rt r r# c g r3 rO ru s r* __call__ParseLinks.__call__ s r, rO N)r9 r: r; r< r r r r> rO r, r* r r s ^ r, r fnc ^ ^ [ R " SS9S[ S[ [ 4U 4S jj5 m[ R
" T 5 SSS[ [ 4U U4S jj5 nU$ ) z
Given a function that parses an Iterable[Link] from an IndexContent, cache the
function's result (keyed by CacheablePageContent), unless the IndexContent
`page` has `page.cache_link_parsing == False`.
N)maxsizecacheable_pager# c : > [ T" U R 5 5 $ r3 )listrt )r r s r* wrapper*with_cached_index_content..wrapper s B~**+,,r, rt r c j > U R ( a T" [ U 5 5 $ [ T" U 5 5 $ r3 )rs rp r )rt r r s r* wrapper_wrapper2with_cached_index_content..wrapper_wrapper s+ " "/56 6BtH~ r, ) functools lru_cacherp r r wraps)r r r s ` @r* with_cached_index_contentr sl &- 4 -d - '- __R n d
r, rt r c # ~ # U R R 5 nUR S5 ( ac [ R " U R
5 nUR
S/ 5 H, n[ R " X0R 5 nUc M( Uv M. g[ U R 5 nU R =( d SnUR U R
R U5 5 U R nUR =( d UnUR H! n [ R " XUS9nUc M Uv M# g7f)zT
Parse a Simple API's Index Content, and yield its anchor elements as Link objects.
rE filesNzutf-8)page_urlbase_url)r0 r& r' jsonloadscontentrG r from_jsonr" HTMLLinkParserencodingfeeddecoder anchors from_element)
rt rJ datafilelinkparserr r" r anchors
r* parse_linksr s &&,,.N !FGGzz$,,'HHWb)D>>$1D|J *
DHH
%F}}'H
KK ##H-.
((C%#H.. I <
!s D;D=c
T \ rS rSrSr S
S\S\S\\ S\S\SS 4 S
jjr S\4S jr
S r g )r i z5Represents one response (or page), along with its URLr r0 r r" rs r# Nc @ Xl X l X0l X@l XPl g)a=
:param encoding: the encoding to decode the given content.
:param url: the URL from which the HTML was downloaded.
:param cache_link_parsing: whether links parsed from this page's url
should be cached. PyPI index urls should
have this set to False, for example.
N)r r0 r r" rs )r6 r r0 r r" rs s r* r5 IndexContent.__init__ s (
"4 r, c , [ U R 5 $ r3 )r r" r s r* __str__IndexContent.__str__ s #DHH--r, )rs r r0 r r" T) r9 r: r; r< __doc__bytesr= r r r5 r r> rO r, r* r r sY ? $(
55 5 3- 5
5 !
5
5*. .r, c ^ \ rS rSrSrS\SS4U 4S jjrS\S\\\\ \ 4 SS4S jr
S\\\\ \ 4 S\ \ 4S
jr S r U =r
$ ) r i zZ
HTMLParser that keeps the first base HREF and a list of all anchor
elements' attributes.
r" r# Nc H > [ TU ] SS9 Xl S U l / U l g )NT)convert_charrefs)r4 r5 r" r r )r6 r" r7 s r* r5 HTMLLinkParser.__init__# s&
$/'+
79 r, tagattrsc US:X a) U R c U R U5 nUb X0l g g US:X a% U R R [ U5 5 g g )Nbasea)r get_hrefr appenddict)r6 r r hrefs r* handle_starttag HTMLLinkParser.handle_starttag* sR &=T]] 2=='D $
CZ LL U , r, c . U H u p#US:X d M
Us $ g )Nr rO )r6 r namevalues r* r HTMLLinkParser.get_href2 s KDv~ ! r, )r r r" )r9 r: r; r< r r= r5 r r r r r r> r? r@ s @r* r r sx
:C :D :-3 -tE#x}:L4M/N -SW -d5hsm);#< = (3- r, r r reasonmeth).Nc < Uc [ R nU" SX5 g )Nz%Could not fetch URL %s: %s - skipping)rc rd )r r r s r* _handle_get_simple_failr 9 s
||| 0$?r, rs c [ U R 5 n[ U R U R S UU R US9$ )NrC )r r" rs )rn rF r r r" )rA rs r s r* _make_index_contentr C sE *(*:*:;H ( LL- r, c U R R SS5 S n[ U5 nU( a [ R SUU 5 g [
R R U5 u pEn nUS:X a [ R R [
R R U5 5 ( aQ UR S5 ( d US-
n[
R R US5 n[ R SU5 [! X!S 9n[# XpR$ S
9$ ! [& a [ R S U 5 g [( a6 n[ R S U UR* UR, 5 S nAg S nAf[. a n[1 X5 S nAg S nAf[2 a n[1 X5 S nAg S nAf[4 a2 nS
n U [7 U5 -
n [1 X [ R8 S9 S nAg S nAf[: R< a n[1 U SU 35 S nAg S nAf[: R> a [1 U S5 g f = f)N#r r zICannot look at %s URL %s because it does not support lookup as web pages.r /z
index.htmlz# file: URL is directory, getting %sr` )rs z`Skipping page %s because it looks like an archive, and cannot be checked by a HTTP HEAD request.zSkipping page %s because the %s request got Content-Type: %s. The only supported Content-Types are application/vnd.pypi.simple.v1+json, application/vnd.pypi.simple.v1+html, and text/htmlz4There was a problem confirming the ssl certificate: )r zconnection error: z timed out) r" splitr+ rc warningrU rV urlparseosrZ isdirrH url2pathnameendswithurljoinrd rf r rs rM r. r1 r0 r r r r r= infor ConnectionErrorTimeout)
r rP r"
vcs_schemer) _rZ r] excr s
r* _get_index_contentr P s
((..a
#C #3'J W
&|| 4 4S 9 FtQ1
BGGMM&..*E*Ed*K L L ||C 3JC
ll""3 5 :C@ U#C9: #4 rO r, r* r r s" *-..*-..r, r c
\ rS rSrSrS\S\SS4S jr\ SS\S\ S \
SS 4S
jj5 r \ S\
\ 4S j5 rS \S\\ 4S
jrS\S\S\4S jrSrg)
LinkCollectori z
Responsible for collecting Link objects from all configured locations,
making network requests as needed.
The class's main method is its collect_sources() method.
rP search_scoper# Nc X l Xl g r3 r rP )r6 rP r s r* r5 LinkCollector.__init__ s
) r, optionssuppress_no_indexc D UR /UR - nUR ( a5 U( d. [ R SSR S U 5 5 5 / nUR =( d / n[ R " UUUR S9n[ UUS9nU$ )z
:param session: The Session to use to make requests.
:param suppress_no_index: Whether to ignore the --no-index option
when constructing the SearchScope object.
zIgnoring indexes: %s,c 3 8 # U H n[ U5 v M g 7fr3 r ).0r" s r* 'LinkCollector.create.. s Ijs-c22js )r r no_index)rP r )
index_urlextra_index_urlsr rc rd re r r creater )clsrP r r r r r link_collectors r* r LinkCollector.create s ''(7+C+CC
$5 LL&IjII
J ''-2
"))!! %%
'%
r, c . U R R $ r3 )r r r s r* r LinkCollector.find_links s +++r, locationc ( [ XR S9$ )z.
Fetch an HTML page containing package links.
r` )r rP )r6 r s r* fetch_response LinkCollector.fetch_response s "(LLAAr, project_namecandidates_from_pagec ^ ^ [ R " UU 4S jT R R U5 5 5 R 5 n[ R " UU 4S jT R
5 5 R 5 n[ R [ R 5 ( a [ R " XC5 Vs/ s H& nUc M UR c M SUR 3PM( nn[ U5 SU S3/U- n[ R SR U5 5 [! [# U5 [# U5 S9$ s snf )Nc 3 f ># U H& n[ UTTR R S S S9v M( g7f)Fr page_validator
expand_dirrs Nr rP is_secure_originr locr r6 s r* r 0LinkCollector.collect_sources.. s< 4
P
%9 #|| < < #(
P .1c 3 f ># U H& n[ UTTR R S S S9v M( g7f)Tr Nr
r s r* r r s: 5
'
%9 #|| < < #'
'r z* z' location(s) to search for versions of :
)r r ) collections OrderedDictr get_index_urls_locationsvaluesr rc isEnabledForloggingDEBUG itertoolschainr r( rd re r r )r6 r r index_url_sourcesfind_links_sourcessliness ` ` r* collect_sources LinkCollector.collect_sources s8 ( 3 3 4
((AA,O 4
&( ) 4 4 5
5
&( w}} - - # );OOA %&VV "QVVH
O
u:, ##/.3 E
LL5) * ./ -.
s E E Er )F)r9 r: r; r< r r r r5 classmethodr r r propertyr r= r r r r r r r r r> rO r, r* r r s "
#(
B ,DI , ,Bt B 0F B *
*
1*
*
r, r r3 r )Rr r
email.messageri r r r r r urllib.parserU urllib.request html.parserr optparser typingr r r r r r
r r r
r r pip._vendorr pip._vendor.requestsr pip._vendor.requests.exceptionsr r pip._internal.exceptionsr pip._internal.models.linkr !pip._internal.models.search_scoper pip._internal.network.sessionr pip._internal.network.utilsr pip._internal.utils.filetypesr pip._internal.utils.miscr pip._internal.vcsr sourcesr r r r! r getLoggerr9 rc r= ResponseHeadersr+ Exceptionr. rK rM r^ rf rn rp r r r r r r r r r r r rO r, r* r9 s) " ! ) @ ; * 9 4 8 9 9 ! A A H 8 $ c*3 8C= )Y )@ @d @, y c J 4 $<c <J <8 <~ HSM # #
* ( n $ 8. .8Z > +/@
@ #y. !@ 8I&
'@
@ 48
,0
:T :z :h~>V :z/z /
g
g
r,