Re @ s0 d d l Z d d l m Z d d l Z d d l Z d d l Z d d l Z d d l Z y d d l Z Wn e k
r d d l
Z Yn Xd d l Z d d l m
Z
d d l m Z m Z m Z m Z m Z m Z m Z m Z m Z m Z m Z m Z m Z m Z m Z d d l m Z m! Z! m" Z" d d l# m$ Z$ m% Z% d d l& m' Z' m( Z( m) Z) m* Z* m+ Z+ m, Z, m- Z- m. Z. m/ Z/ d d l0 m1 Z1 m2 Z2 d d
l3 m4 Z4 m5 Z5 e j6 e7 Z8 e j9 d Z: e j9 d e j; Z< e j9 d
Z= d Z> d d d Z? Gd d d e Z@ Gd d d eA ZB Gd d d eB ZC Gd d d eB ZD Gd d d eA ZE Gd d d eB ZF Gd d d eB ZG Gd d d eB ZH Gd! d" d" eB ZI Gd# d$ d$ eB ZJ eJ eH eF d% d& d' d( d) ZK eK jL ZL e j9 d* ZM Gd+ d, d, eA ZN d S)- N)BytesIO )DistlibException)urljoinurlparse
urlunparse url2pathname pathname2urlqueuequoteunescape string_types build_openerHTTPRedirectHandler text_typeRequest HTTPErrorURLError) DistributionDistributionPath make_dist)MetadataMetadataInvalidError) cached_propertyparse_credentials ensure_slashsplit_filenameget_project_dataparse_requirementparse_name_and_version ServerProxynormalize_name)
get_schemeUnsupportedVersionError)Wheel
is_compatiblez^(\w+)=([a-f0-9]+)z;\s*charset\s*=\s*(.*)\s*$z text/html|application/x(ht)?mlzhttps://pypi.org/pypic C sG | d k r t } t | d d } z | j SWd | d Xd S)z
Return all distribution names known by an index.
:param url: The URL of the index.
:return: A list of all known distribution names.
Ntimeoutg @close)
DEFAULT_INDEXr
list_packages)urlclient r, /builddir/build/BUILDROOT/alt-python35-pip-20.2.4-5.el8.x86_64/opt/alt/python35/lib/python3.5/site-packages/pip/_vendor/distlib/locators.pyget_all_distribution_names) s r. c @ s0 e Z d Z d Z d d Z e Z Z Z d S)RedirectHandlerzE
A class to work around a bug in some Python 3.2.x releases.
c C s d } x% d D] } | | k r
| | } Pq
W| d k r> d St | } | j d k r t | j | } t | d r | j | | n
| | |
Clear any errors which may have been logged.
N)rb )r9 r, r, r- clear_errors s zLocator.clear_errorsc C s | j j d S)N)rU clear)r9 r, r, r- clear_cache s zLocator.clear_cachec C s | j S)N)_scheme)r9 r, r, r- _get_scheme s zLocator._get_schemec C s
| | _ d S)N)rf )r9 valuer, r, r- _set_scheme s zLocator._set_schemec C s t d d S)a=
For a given project, get a dictionary mapping available versions to Distribution
instances.
This should be implemented in subclasses.
If called from a locate() request, self.matcher will be set to a
matcher for the requirement to satisfy, otherwise it will be None.
z Please implement in the subclassN)NotImplementedError)r9 namer, r, r- _get_project s
zLocator._get_projectc C s t d d S)zJ
Return all the distribution names known to this locator.
z Please implement in the subclassN)rj )r9 r, r, r- get_distribution_names s z Locator.get_distribution_namesc C sj | j d k r! | j | } nE | | j k r@ | j | } n& | j | j | } | | j | <| S)z
For a given project, get a dictionary mapping available versions to Distribution
instances.
This calls _get_project to do all the work, and just implements a caching layer on top.
N)rU rl rc )r9 rk r` r, r, r- get_project s
zLocator.get_projectc C s t | } t j | j } d } | j d } | j | j } | rc t t | | j } | j d k d | j
k | | | | f S)zu
Give an url a score which can be used to choose preferred URLs
for a given project release.
Tz.whlhttpszpypi.org) r posixpathbasenamepathendswithdownloadable_extensionsr% r$
wheel_tagsr4 netloc)r9 r* trq
compatibleis_wheelZis_downloadabler, r, r- score_url s zLocator.score_urlc C su | } | rq | j | } | j | } | | k r< | } | | k r^ t j d | | n t j d | | | S)a{
Choose one of two URLs where both are candidates for distribution
archives for the same version of a distribution (for example,
.tar.gz vs. zip).
The current implementation favours https:// URLs over http://, archives
from PyPI over those from other locations, wheel compatibility (if a
wheel) and then the archive name.
zNot replacing %r with %rzReplacing %r with %r)rz loggerdebug)r9 url1url2r` s1s2r, r, r-
prefer_url s
zLocator.prefer_urlc C s
t | | S)zZ
Attempt to split a filename in project name, version and Python version.
)r )r9 filename project_namer, r, r- r s zLocator.split_filenamec C s d d } d } t | \ } } } } } }
|
j j d rX t j d | |
t j |
} | r | j \ } }
n d \ } }
| } | r | d d k r | d d } | j d ry t | } t
| | j s t j d | n | d k rd
} n | | j | } | rd | j d | j
d
| j d t | | | | | d f d d j d d | j D i } Wqt k
r} z t j d | WYd d } ~ XqXn | j | j st j d | n t j | } } x | j D] } | j | r| d t | } | j | | } | s]t j d | nk | \ } } } | s| | | rd | d | d
| d t | | | | | d f i } | r| | d .same_projectNzegg=z %s: version hint in fragment: %rr /z.whlzWheel not compatible: %sTrk versionr r* r2 zpython-versionz, c S s/ g | ]% } d j t | d d q S). N)joinlist).0vr, r, r-
s z8Locator.convert_url_to_download_info..zinvalid path for wheel: %szNot downloadable: %sz No match for project/version: %sz %s_digest)NNr )r lower
startswithr{ r| HASHER_HASHmatchgroupsrs r$ r% ru rk r r r r pyver Exceptionwarningrt rp rq lenr )r9 r* r r r` r4 rv rr paramsqueryfragmalgodigestZorigpathwheelincludera r extrw rk r r r, r, r- convert_url_to_download_info sj
&
z$Locator.convert_url_to_download_infoc C s d } d | k rJ | d } x+ d D]# } | | k r# | | | f } Pq# W| s x5 d D]- } d | } | | k rW | | | f } PqW W| S)z
Get a digest from a dictionary by looking at a "digests" dictionary
or keys of the form 'algo_digest'.
Returns a 2-tuple (algo, digest) if found, else None. Currently
looks only for SHA256, then MD5.
Ndigestssha256md5z %s_digest)r r )r r r, )r9 infor` r r r@ r, r, r- _get_digest1 s
zLocator._get_digestc C s | j d } | j d } | | k r@ | | } | j } n! t | | d | j } | j } | j | | _ } | d } | | d | <| j | d k r | j | j | | _ | d j | t j
| | | _ | | | d } t | } | d k r. t d | t | j } | j | j | _ } t j d | t | j | j
| j } t | d k rg } | j
} x | D] }
|
d
k r q ye | j |
s t j d | |
n< | s | |
j r| j |
n t j d |
| j Wq t k
rMt j d | |
Yq Xq Wt | d
k ryt | d | j } | rt j d | | d } | | } | r1| j r| j | _ | j d i j | t | _ i } | j d i }
x+ | j D] } | |
k r|
| | | = 1.0, < 2.0, != 1.3)'
:param prereleases: If ``True``, allow pre-release versions
to be located. Otherwise, pre-release versions
are not returned.
:return: A :class:`Distribution` instance, or ``None`` if no such
distribution could be located.
NzNot a valid requirement: %rzmatcher: %s (%s)r r r z%s did not match %rz%skipping pre-release version %s of %szerror matching %s with %rr r@ zsorted list: %s)r r r )r r r" r4 rW requirementr{ r| typerB rn rk r Z
version_classr
is_prereleaser] r r sortedr@ extrasr\ r
download_urlsr )r9 r prereleasesr` rr4 rW versionsslistZvclskr dsdr* r, r, r- locate_ sT
$ zLocator.locate)rJ rK rL rM rN rO )rP rQ rR )rS )rR )rB rC rD rE source_extensionsbinary_extensionsexcluded_extensionsru rt rZ rb rc re rg ri propertyr4 rl rm rn rz r r r r r r r, r, r, r- rI V s.
J rI c s@ e Z d Z d Z f d d Z d d Z d d Z S)PyPIRPCLocatorz
This locator uses XML-RPC to locate distributions. It therefore
cannot be used with simple mirrors (that only mirror file content).
c s8 t t | j | | | _ t | d d | _ d S)z
Initialise an instance.
:param url: The URL to use for XML-RPC.
:param kwargs: Passed to the superclass constructor.
r& g @N)superr rZ base_urlr r+ )r9 r* kwargs) __class__r, r- rZ s zPyPIRPCLocator.__init__c C s t | j j S)zJ
Return all the distribution names known to this locator.
)r r+ r) )r9 r, r, r- rm s z%PyPIRPCLocator.get_distribution_namesc C sq d i d i i } | j j | d } xC| D];} | j j | | } | j j | | } t d | j } | d | _ | d | _ | j d | _ | j d g | _
| j d | _ t | } | r. | d
} | d | _
| j | | _ | | _ | | | [^"]*)"|'(?P[^']*)'|(?P[^>\s
]*))\s+)?
href\s*=\s*(?:"(?P[^"]*)"|'(?P[^']*)'|(?P[^>\s
]*))
(\s+rel\s*=\s*(?:"(?P[^"]*)"|'(?P[^']*)'|(?P[^>\s
]*)))?
z!]+)c C sJ | | _ | | _ | _ | j j | j } | rF | j d | _ d S)zk
Initialise an instance with the Unicode page contents and the URL they
came from.
r N)r r r* _basesearchgroup)r9 r r* r r, r, r- rZ s
z
Page.__init__z[^a-z0-9$&+,/:;=?@.#%_\\|-]c C s d d } t } x | j j | j D] } | j d } | d py | d py | d py | d py | d py | d } | d
p | d p | d } t | j | } t | } | j j d
d | } | j
| | f q+ Wt | d d d d d } | S)z
Return the URLs of all the links on a page together with information
about their "rel" attribute, for determining which ones to treat as
downloads and which ones to queue for further scraping.
c S s@ t | \ } } } } } } t | | t | | | | f S)zTidy up an URL.)r r r )r* r4 rv rr r r r r, r, r- clean4 s zPage.links..cleanr2 Zrel1Zrel2Zrel3Zrel4Zrel5Zrel6r} r~ Zurl3c S s d t | j d S)Nz%%%2xr )ordr )r r, r, r- B s z Page.links..r@ c S s | d S)Nr r, )rw r, r, r- r F s reverseT) r _hreffinditerr groupdictr r r _clean_resubr r )r9 r r` r r relr* r, r, r- links- s z
Page.linksN)rB rC rD rE recompileISXr r rZ r r r r, r, r, r- r s r c s e Z d Z d Z d e j d d d d d d i Z d d f d
d Z d d
Z d d Z d d Z
e j d e j
Z d d Z d d Z d d Z d d Z d d Z e j d Z d d Z S) SimpleScrapingLocatorz
A locator which scrapes HTML pages to locate downloads for a distribution.
This runs multiple threads to do the I/O; performance is at least as good
as pip's PackageFinder, which works in an analogous fashion.
deflategzipc C s t j d t t j S)Nfileobj)r GzipFiler r r )br, r, r- r T s z SimpleScrapingLocator.nonec C s | S)Nr, )r r, r, r- r U s N
c s t t | j | t | | _ | | _ i | _ t | _ t j
| _ t | _ d | _
| | _ t j | _ t j | _ d | _ d S)a
Initialise an instance.
:param url: The root URL to use for scraping.
:param timeout: The timeout, in seconds, to be applied to requests.
This defaults to ``None`` (no timeout specified).
:param num_workers: The number of worker threads you want to do I/O,
This defaults to 10.
:param kwargs: Passed to the superclass.
FN)r r rZ r r r& _page_cacher _seenr
rX _to_fetch
_bad_hostsskip_externals num_workers threadingRLock_lock_gplockplatform_check)r9 r* r& r r )r r, r- rZ X s
z SimpleScrapingLocator.__init__c C sc g | _ xS t | j D]B } t j d | j } | j d | j | j j | q Wd S)z
Threads are created only when get_project is called, and terminate
before it returns. They are there primarily to parallelise I/O (i.e.
fetching web pages).
targetTN) _threadsranger r Thread_fetch setDaemonstartr] )r9 irw r, r, r- _prepare_threadss s
z&SimpleScrapingLocator._prepare_threadsc C sO x! | j D] } | j j d q
Wx | j D] } | j q. Wg | _ d S)zu
Tell all the threads to terminate (by sending a sentinel value) and
wait for them to do so.
N)r r r r )r9 rw r, r, r-
_wait_threads s
z#SimpleScrapingLocator._wait_threadsc C s d i d i i } | j | | _ | | _ t | j d t | } | j j | j j | j z1 t
j d | | j j
| | j j Wd | j X| ` Wd QRX| S)Nr r z%s/z Queueing %s)r r` r r r r r rd r r r{ r| r r r r )r9 rk r` r* r, r, r- rl s
z"SimpleScrapingLocator._get_projectz<\b(linux_(i\d86|x86_64|arm\w+)|win(32|_amd64)|macosx_?\d+)\bc C s | j j | S)zD
Does an URL refer to a platform-specific download?
)platform_dependentr )r9 r* r, r, r- _is_platform_dependent s z,SimpleScrapingLocator._is_platform_dependentc
C sw | j r! | j | r! d } n | j | | j } t j d | | | rs | j | j | j | Wd QRX| S)a%
See if an URL is a suitable download for a project.
If it is, register information in the result dictionary (for
_get_project) about the specific version it's for.
Note that the return value isn't actually used other than as a boolean
value.
Nzprocess_download: %s -> %s) r r r r r{ r| r r r` )r9 r* r r, r, r- _process_download s
z'SimpleScrapingLocator._process_downloadc
C s t | \ } } } } } } | j | j | j | j rG d } n | j rl | j | j rl d } n | j | j s d } ny | d
k r d } nd | d k r d } nO | j | r d } n7 | j d d d } | j
d
k r d } n d } t j d | | | | | S)z
Determine whether a link URL from a referring page and with a
particular "rel" attribute should be queued for scraping.
Fhomepagedownloadhttpro ftp:r r localhostTz#should_queue: %s (%s) from %s -> %s)r
r )r ro r )
r rs r r r r r r r splitr r{ r| )
r9 linkZreferrerr r4 rv rr _r` hostr, r, r-
_should_queue s*
z#SimpleScrapingLocator._should_queuec C s6 x/| j j } zy | r | j | } | d k r< w x | j D] \ } } | | j k rF y\ | j j | | j | r | j | | | r t j d | | | j j
| WqF t k
r YqF XqF WWn; t k
r} z | j
j
t | WYd d } ~ Xn XWd | j j X| s Pq Wd S)z
Get a URL to fetch from the work queue, get the HTML page, examine its
links for download candidates and candidates for further scraping.
This is a handy method to run in a thread.
NzQueueing %s from %s)r r\ get_pager r r r r r{ r| r r r rY r r_ )r9 r* pager r ra r, r, r- r s,
-z SimpleScrapingLocator._fetchc C s t | \ } } } } } } | d k rW t j j t | rW t t | d } | | j k r | j | } t j d | | nv| j
d d d } d } | | j k r t j d | | n5t | d d
d i } z y/t j d | | j
j | d
| j } t j d | | j }
|
j d d } t j | r| j } | j }
|
j d } | r| j | } | |
}
d } t j | } | r| j d } y |
j | }
Wn! t k
r|
j d }
Yn Xt |
| } | | j | zAccept-encodingidentityz Fetching %sr& z
Fetched %sz Content-Typer2 zContent-Encodingzutf-8zlatin-1i zFetch failed: %s: %s)$r osrr isdirr r r r r{ r| r r r rV r r& r r\ HTML_CONTENT_TYPEr geturlr decodersCHARSETr r r UnicodeErrorr r r< r r r r r )r9 r* r4 rv rr r r` r r: r r> content_typeZ final_urlr encodingdecoderr ra r, r, r- r sZ $
%
)*z SimpleScrapingLocator.get_pagez]*>([^<]+)[\w-]+)\s*\(\s*(==\s*)?(?P[^)]+)\)$c @ ss e Z d Z d Z d d d Z d d Z d d Z d d
Z d d Z d
d Z d d d d Z
d S)DependencyFinderz0
Locate dependencies for distributions.
Nc C s( | p t | _ t | j j | _ d S)zf
Initialise an instance, using the specified locator
to locate distributions.
N)default_locatorr r" r4 )r9 r r, r, r- rZ 1 s zDependencyFinder.__init__c C s t j d | | j } | | j | <| | j | | j f = 1.0) while C requires (B >= 1.1).
For successful replacement, ``provider`` must meet all the requirements
which ``other`` fulfills.
:param provider: The provider we are trying to replace with.
:param other: The provider we're trying to replace.
:param problems: If False is returned, this will contain what
problems prevented replacement. This is currently
a tuple of the literal string 'cantreplace',
``provider``, ``other`` and the set of requirements
that ``provider`` couldn't fulfill.
:return: True if we can replace ``other`` with ``provider``, else
False.
Z cantreplaceFT)
reqtsr rT r r r frozensetrR r rO ) r9 rU otherproblemsZrlist unmatchedrQ rW r` r, r, r- try_to_replace s"
#
z DependencyFinder.try_to_replaceFc C s i | _ i | _ i | _ i | _ t | p0 g } d | k rh | j d | t d d d g O} t | t r | } } t j d | nH | j
j | d | } } | d k r t d | t j d | d
| _
t } t | g } t | g } x| r| j } | j } | | j k rI| j | n, | j | }
|
| k ru| j | |
| | j | j B} | j } t }
| r| | k rx; d D]3 } d | } | | k r|
t | d | O}
qW| | B|
B} x| D]} | j | } | st j d | | j
j | d | } | d k rj| rj| j
j | d d
} | d k rt j d | | j d | f nz | j | j } } | | f | j k r| j | | j | | | k r| | k r| j | t j d | j xt | D]l } | j } | | j k r]| j j | t j | q | j | }
|
| k r | j | |
| q WqWqWt | j j } x9 | D]1 } | | k | _ | j rt j d | j qWt j d | | | f S)a
Find a distribution and all distributions it depends on.
:param requirement: The requirement specifying the distribution to
find, or a Distribution instance.
:param meta_extras: A list of meta extras such as :test:, :build: and
so on.
:param prereleases: If ``True``, allow pre-release versions to be
returned - otherwise, don't return prereleases
unless they're all that's available.
Return a set of :class:`Distribution` instances and a set of
problems.
The distributions returned should be such that they have the
:attr:`required` attribute set to ``True`` if they were
from the ``requirement`` passed to ``find()``, and they have the
:attr:`build_time_dependency` attribute set to ``True`` unless they
are post-installation dependencies of the ``requirement``.
The problems should be a tuple consisting of the string
``'unsatisfied'`` and the requirement which couldn't be satisfied
by any distribution known to the locator.
z:*:z:test:z:build:z:dev:zpassed %s as requirementr NzUnable to locate %rz
located %sTtestbuilddevz:%s:z %s_requireszNo providers found for %rzCannot satisfy %rZ unsatisfiedzAdding %s to install_distsz#%s is a build-time dependency only.zfind done for %s)r] r^ r_ ) rM rK rJ rW r rP r; r r{ r| r r r requestedr r@ rO r\ Z run_requiresZ
meta_requiresZbuild_requiresgetattrrV r r Zname_and_versionr valuesZbuild_time_dependency)r9 r Z meta_extrasr r r rZ todoZ
install_distsrk rY ZireqtsZsreqtsZereqtsr@ ra Z all_reqtsr Z providersrU nr rN rK r, r, r- find s
"
zDependencyFinder.find) rB rC rD rE rZ rO rR rT rV r\ re r, r, r, r- rH , s (rH )Or ior r loggingr rp r r ImportErrordummy_threadingr' r2 r compatr r r r r r
r r r
r r r7 r r r r Zdatabaser r r r r r utilr r r r r r r r r! r r" r# r r$ r% getLoggerrB r{ r r r r! r r( r. r/ objectrI r r r r r) r4 r: r? rI r NAME_VERSION_RErH r, r, r, r- sV
d @ F0E:A& [