Rec @ s d Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l m Z d d l
m Z m Z d d l
m Z d d l m Z m Z d d l m Z d d l m Z d d l m Z d d
l m Z d d l m Z d d l m Z d d
l m Z d d l! m" Z" d d l# m$ Z$ m% Z% d d l& m' Z' d d l( m) Z) m* Z* d d l+ m, Z, m- Z- e' rd d l. m/ Z/ d d l0 m1 Z1 m2 Z2 m3 Z3 m4 Z4 m5 Z5 m6 Z6 m7 Z7 m8 Z8 d d l9 Z: d d l; m< Z< d d l= m> Z> e: j? j@ jA ZB e4 eC eC f ZD n e jE eF ZG d ZH d ZI d eJ f d YZK d ZL d eJ f d YZM d ZN d ZO d ZP d! ZQ d" ZR d# ZS e jT d$ e jU ZV d% ZW d& ZX d' ZY d( eZ f d) YZ[ d* Z\ e\ d+ Z] d, eZ f d- YZ^ d d. Z` ea d/ Zb d d0 Zc d1 Zd ee d2 Zf d3 eZ f d4 YZg d5 eZ f d6 YZh d S(7 sM
The main purpose of this module is to expose LinkCollector.collect_links().
iN( t OrderedDict( t html5libt requests( t unescape( t
RetryErrort SSLError( t parse( t request( t NetworkConnectionError( t Link( t SearchScope( t raise_for_status( t lru_cache( t ARCHIVE_EXTENSIONS( t pairwiset redact_auth_from_url( t MYPY_CHECK_RUNNING( t path_to_urlt url_to_path( t is_urlt vcs( t Values( t Callablet Iterablet Listt MutableMappingt Optionalt Sequencet Tuplet Union( t Response( t
PipSessionc C sG x@ t j D]5 } | j j | r
| t | d k r
| Sq
Wd S( sg Look for VCS schemes in the URL.
Returns the matched VCS scheme, or None if there's no match.
s +:N( R t schemest lowert
startswitht lent None( t urlt scheme( ( s /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyt _match_vcs_scheme8 s +c C s7 t | j } x! t D] } | j | r t Sq Wt S( s2 Return whether the URL looks like an archive.
( R t filenameR
t endswitht Truet False( R% R( t bad_ext( ( s /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyt _is_url_like_archiveD s
t _NotHTMLc B s e Z d Z RS( c C s/ t t | j | | | | _ | | _ d S( N( t superR. t __init__t content_typet request_desc( t selfR1 R2 ( ( s /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyR0 P s ( t __name__t
__module__R0 ( ( ( s /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyR. O s c C sF | j j d d } | j j d sB t | | j j n d S( s Check the Content-Type header to ensure the response contains HTML.
Raises `_NotHTML` if the content type is not text/html.
s Content-Typet s text/htmlN( t headerst getR! R" R. R t method( t responseR1 ( ( s /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyt _ensure_html_headerW s t _NotHTTPc B s e Z RS( ( R4 R5 ( ( ( s /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyR< b s c C si t j | \ } } } } } | d d h k r< t n | j | d t } t | t | d S( s Send a HEAD request to the URL, and ensure the response contains HTML.
Raises `_NotHTTP` if the URL is not available for a HEAD request, or
`_NotHTML` if the content type is not text/html.
t httpt httpst allow_redirectsN( t urllib_parset urlsplitR< t headR* R R; ( R% t sessionR& t netloct patht queryt fragmentt resp( ( s /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyt _ensure_html_responsef s
c C sp t | r t | d | n t j d t | | j | d i d d 6d d 6} t | t | | S( s Access an HTML page with GET, and return the response.
This consists of three parts:
1. If the URL looks suspiciously like an archive, send a HEAD first to
check the Content-Type is HTML, to avoid downloading a large file.
Raise `_NotHTTP` if the content type cannot be determined, or
`_NotHTML` if it is not HTML.
2. Actually perform the request. Raise HTTP exceptions on network failures.
3. Check the Content-Type header to make sure we got HTML, and raise
`_NotHTML` otherwise.
RC s Getting page %sR7 s text/htmlt Accepts max-age=0s
Cache-Control( R- RI t loggert debugR R8 R R; ( R% RC RH ( ( s /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyt _get_html_responsew s
c C sF | rB d | k rB t j | d \ } } d | k rB | d Sn d S( sB Determine if we have any encoding information in our headers.
s Content-Typet charsetN( t cgit parse_headerR$ ( R7 R1 t params( ( s /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyt _get_encoding_from_headers s
c C s= x6 | j d D]% } | j d } | d k r | Sq W| S( s Determine the HTML document's base URL.
This looks for a ```` tag in the HTML document. If present, its href
attribute denotes the base URL of anchor tags in the document. If there is
no such tag (or if it does not have a valid href attribute), the HTML
file's URL is used as the base URL.
:param document: An HTML document representation. The current
implementation expects the result of ``html5lib.parse()``.
:param page_url: The URL of the HTML document.
s .//baset hrefN( t findallR8 R$ ( t documentt page_urlt baseRS ( ( s /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyt _determine_base_url s
c C s t j t j | S( sP
Clean a "part" of a URL path (i.e. after splitting on "@" characters).
( R@ t quotet unquote( t part( ( s /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyt _clean_url_path_part s c C s t j t j | S( s
Clean the first part of a URL path that corresponds to a local
filesystem path (i.e. the first part after splitting on "@" characters).
( t urllib_requestt pathname2urlt url2pathname( R[ ( ( s /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyt _clean_file_url_path s s (@|%2F)c C s | r t } n t } t j | } g } xO t t j | d g D]2 \ } } | j | | | j | j qF Wd j | S( s*
Clean the path portion of a URL.
R6 (
R` R\ t _reserved_chars_ret splitR t itertoolst chaint appendt uppert join( RE t
is_local_patht
clean_funct partst
cleaned_partst to_cleant reserved( ( s /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyt _clean_url_path s (c C sG t j | } | j } t | j d | } t j | j d | S( s
Make sure a link is fully quoted.
For example, if ' ' occurs in the URL, it will be replaced with "%20",
and without double-quoting other characters.
Rh RE ( R@ t urlparseRD Rn RE t
urlunparset _replace( R% t resultRh RE ( ( s /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyt _clean_link s
c C s | j d } | s d St t j | | } | j d } | rR t | n d } | j d } | r| t | } n t | d | d | d | } | S( sJ
Convert an anchor element in a simple repository page to a Link.
RS s data-requires-pythons data-yankedt
comes_fromt requires_pythont
yanked_reasonN( R8 R$ Rs R@ t urljoinR R ( t anchorRV t base_urlRS R% t pyrequireRv t link( ( s /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyt _create_link_from_element s t CacheablePageContentc B s# e Z d Z d Z d Z RS( c C s | j s t | | _ d S( N( t cache_link_parsingt AssertionErrort page( R3 R ( ( s /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyR0 , s c C s+ t | t | o* | j j | j j k S( N( t
isinstancet typeR R% ( R3 t other( ( s /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyt __eq__1 s c C s t | j j S( N( t hashR R% ( R3 ( ( s /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyt __hash__6 s ( R4 R5 R0 R R ( ( ( s /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyR} + s c sC t d d f d t j f d } | S( s
Given a function that parses an Iterable[Link] from an HTMLPage, cache the
function's result (keyed by CacheablePageContent), unless the HTMLPage
`page` has `page.cache_link_parsing == False`.
t maxsizec s t | j S( N( t listR ( t cacheable_page( t fn( s /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyt wrapperE s c s) | j r t | St | S( N( R~ R} R ( R ( R R ( s /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyt wrapper_wrapperJ s N( R R$ t functoolst wraps( R R ( ( R R s /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyt with_cached_html_pages; s
!c c s t j | j d | j d t } | j } t | | } xF | j d D]5 } t | d | d | } | d k ry qI n | VqI Wd S( sP
Parse an HTML document, and yield its anchor elements as Link objects.
t transport_encodingt namespaceHTMLElementss .//aRV Ry N(
R R t contentt encodingR+ R% RX RT R| R$ ( R RU R% Ry Rx R{ ( ( s /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyt parse_linksT s t HTMLPagec B s# e Z d Z e d Z d Z RS( s' Represents one page, along with its URLc C s( | | _ | | _ | | _ | | _ d S( sm
:param encoding: the encoding to decode the given content.
:param url: the URL from which the HTML was downloaded.
:param cache_link_parsing: whether links parsed from this page's url
should be cached. PyPI index urls should
have this set to False, for example.
N( R R R% R~ ( R3 R R R% R~ ( ( s /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyR0 p s c C s
t | j S( N( R R% ( R3 ( ( s /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyt __str__ s ( R4 R5 t __doc__R* R0 R ( ( ( s /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyR m s c C s, | d k r t j } n | d | | d S( Ns% Could not fetch URL %s: %s - skipping( R$ RK RL ( R{ t reasont meth( ( s /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyt _handle_get_page_fail s c C s1 t | j } t | j d | d | j d | S( NR R% R~ ( RR R7 R R R% ( R: R~ R ( ( s /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyt _make_html_page s c
C s, | d k r t d n | j j d d d } t | } | r] t j d | | d St j | \ } } } } } } | d k r t j
j t j
| r | j d s | d 7} n t j | d } t j d | n y t | d
| } Wnt k
r t j d | n t k
rM} t j d | | j | j n t k
rl} t | | n t k
r} t | | n t k
r} d
} | t | 7} t | | d t j n_ t j k
r} t | d j | n4 t j k
rt | d n Xt | d | j! Sd S( Ns? _get_html_page() missing 1 required keyword argument: 'session't #i i sI Cannot look at %s URL %s because it does not support lookup as web pages.t filet /s
index.htmls# file: URL is directory, getting %sRC s` Skipping page %s because it looks like an archive, and cannot be checked by a HTTP HEAD request.si Skipping page %s because the %s request got Content-Type: %s.The only supported Content-Type is text/htmls4 There was a problem confirming the ssl certificate: R s connection error: {}s timed outR~ (" R$ t TypeErrorR% Rb R' RK t warningR@ Ro t osRE t isdirR] R_ R) Rw RL RM R< R. R2 R1 R R R R t strt infoR t ConnectionErrort formatt TimeoutR R~ (
R{ RC R% t
vcs_schemeR& t _RE RH t excR ( ( s /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyt _get_html_page sP
!'
c C s t t j | S( sQ
Return a list of links, with duplicates removed and ordering preserved.
( R R t fromkeys( t links( ( s /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyt _remove_duplicate_links s c sh g g f d } x=| D]5} t j j | } | j d } | sX | r.| rg | } n t | } t j j | r | r t j j | } xY t j | D] } | t j j | | q Wq+| r j | q+t
j d | qZt j j | r| | qZt
j d | q% t
| rJ j | q% t
j d | q% W f S( s
Divide a list of locations into two groups: "files" (archives) and "urls."
:return: A pair of lists (files, urls).
c sL t | } t j | d t d d k r; j | n
j | d S( Nt stricti s text/html( R t mimetypest
guess_typeR+ Re ( RE R% ( t filest urls( s /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyt sort_path s s file:s( Path '%s' is ignored: it is a directory.s: Url '%s' is ignored: it is neither a file nor a directory.sQ Url '%s' is ignored. It is either a non-existing path or lacks a specific scheme.( R RE t existsR" R R t realpatht listdirRg Re RK R t isfileR ( t locationst
expand_dirR R% Rh t is_file_urlRE t item( ( R R s /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyt group_locations s<
t CollectedLinksc B s e Z d Z d Z RS( s
Encapsulates the return value of a call to LinkCollector.collect_links().
The return value includes both URLs to project pages containing package
links, as well as individual package Link objects collected from other
sources.
This info is stored separately as:
(1) links from the configured file locations,
(2) links from the configured find_links, and
(3) urls to HTML project pages, as described by the PEP 503 simple
repository API.
c C s | | _ | | _ | | _ d S( s
:param files: Links from file locations.
:param find_links: Links from find_links.
:param project_urls: URLs to HTML project pages, as described by
the PEP 503 simple repository API.
N( R t
find_linkst project_urls( R3 R R R ( ( s /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyR0 , s
( R4 R5 R R0 ( ( ( s /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyR s t
LinkCollectorc B sJ e Z d Z d Z e e d Z e d Z d Z d Z
RS( s
Responsible for collecting Link objects from all configured locations,
making network requests as needed.
The class's main method is its collect_links() method.
c C s | | _ | | _ d S( N( t search_scopeRC ( R3 RC R ( ( s /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyR0 G s c C s | j g | j } | j rO | rO t j d d j d | D g } n | j p[ g } t j d | d | } t d | d | } | S( s
:param session: The Session to use to make requests.
:param suppress_no_index: Whether to ignore the --no-index option
when constructing the SearchScope object.
s Ignoring indexes: %st ,c s s | ] } t | Vq d S( N( R ( t .0R% ( ( s /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pys \ s R t
index_urlsRC R (
t index_urlt extra_index_urlst no_indexRK RL Rg R R
t createR ( t clsRC t optionst suppress_no_indexR R R t link_collector( ( s /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyR P s c C s
| j j S( N( R R ( R3 ( ( s /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyR k s c C s t | d | j S( s>
Fetch an HTML page containing package links.
RC ( R RC ( R3 t location( ( s /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyt
fetch_pagep s c C s] | j } | j | } t | \ } } t | j d t \ } } g t j | | D] } t | ^ qX } g | j D] } t | d ^ qz }
g t j d | D d | D D] } | j j | r | ^ q } t
| } d j t | | g }
x$ | D] } |
j
d j | qWt j d j |
t d | d |
d
| S( s Find all available links for the given project name.
:return: All the Link objects (unfiltered), as a CollectedLinks object.
R s -fc s s! | ] } t | d t Vq d S( R~ N( R R+ ( R R% ( ( s /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pys s c s s | ] } t | Vq d S( N( R ( R R% ( ( s /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pys s s, {} location(s) to search for versions of {}:s * {}s
R R R ( R t get_index_urls_locationsR R R* Rc Rd R RC t is_secure_originR R R# Re RK RL Rg R ( R3 t project_nameR t index_locationst index_file_loct
index_url_loct fl_file_loct
fl_url_locR% t
file_linkst find_link_linksR{ t
url_locationst lines( ( s /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyt
collect_linksw s* +%
( R4 R5 R R0 t classmethodR+ R t propertyR R R ( ( ( s /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyR > s (i R RO R Rc t loggingR R t ret collectionsR t pip._vendorR R t pip._vendor.distlib.compatR t pip._vendor.requests.exceptionsR R t pip._vendor.six.moves.urllibR R@ R R] t pip._internal.exceptionsR t pip._internal.models.linkR t! pip._internal.models.search_scopeR
t pip._internal.network.utilsR t pip._internal.utils.compatR t pip._internal.utils.filetypesR
t pip._internal.utils.miscR R t pip._internal.utils.typingR t pip._internal.utils.urlsR R t pip._internal.vcsR R t optparseR t typingR R R R R R R R t xml.etree.ElementTreet xmlt pip._vendor.requestsR t pip._internal.network.sessionR t etreet ElementTreet Elementt HTMLElementR t ResponseHeaderst getLoggerR4 RK R' R- t ExceptionR. R; R< RI RM RR RX R\ R` t compilet
IGNORECASERa Rn Rs R| t objectR} R R R R$ R R* R R R R+ R R R ( ( ( s /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyt st :
3
9 ;#