Rea~ @ s d d l m Z m Z m Z d d l m Z d d l m Z m Z d d l Z d d l
Z
d d l m Z m
Z
d d l m Z d d l m Z m Z m Z m Z d d l m Z d d
l m Z e d d e D Z e d
d e D Z e d d e D Z e e d d g BZ d Z e j re dE d k rWe j d d k s]t e
j! e d dF e" d d Z# n e
j! e Z# d d d d d d d d d d d d d d! d" d# d$ d% d& d' d( d) d* d+ d, d- d. d/ d0 d1 d2 d3 h Z$ e
j! d4 Z% i Z& Gd5 d6 d6 e' Z( d7 d8 Z) Gd9 d: d: e' Z* Gd; d< d< e* Z+ Gd= d> d> e, Z- Gd? d@ d@ e' Z. GdA dB dB e' Z/ dC dD Z0 d S)G )absolute_importdivisionunicode_literals) text_type) http_clienturllibN)BytesIOStringIO) webencodings )EOFspaceCharacters asciiLettersasciiUppercase)_ReparseException)_utilsc C s g | ] } | j d q S)ascii)encode).0item r /builddir/build/BUILDROOT/alt-python35-pip-20.2.4-5.el8.x86_64/opt/alt/python35/lib/python3.5/site-packages/pip/_vendor/html5lib/_inputstream.py
s r c C s g | ] } | j d q S)r )r )r r r r r r s c C s g | ] } | j d q S)r )r )r r r r r r s > t | j | | k rd | t | j | 8} | d 7} q' W| | g | _ d S)Nr r )_bufferedBytesAssertionErrorr# r r )r! r$ offsetir r r seekF s zBufferedStream.seekc C sp | j s | j | S| j d t | j k r_ | j d t | j d k r_ | j | S| j | Sd S)Nr r r )r _readStreamr r# _readFromBuffer)r! bytesr r r readO s
zBufferedStream.readc C s t d d | j D S)Nc S s g | ] } t | q Sr )r# )r r r r r r Y s z1BufferedStream._bufferedBytes..)sumr )r! r r r r' X s z BufferedStream._bufferedBytesc C sJ | j j | } | j j | | j d d 7.z3Cannot set an encoding with a unicode input, set %r)
isinstancer HTTPResponser responseaddbasefphasattrr/ r TypeErrorHTMLUnicodeInputStreamHTMLBinaryInputStream)sourcekwargs isUnicode encodingsr r r HTMLInputStream} s
rO c @ s e Z d Z d Z d Z d d Z d d Z d d Z d d
Z d d Z d
d Z
d d d Z d d Z d d Z
d d d Z d d Z d S)rI zProvides a unicode stream of characters to the HTMLTokenizer.
This class takes care of character encoding and removing or replacing
incorrect byte-sequences and also provides column and line tracking.
i ( c C s t j s d | _ n- t d d k r6 | j | _ n | j | _ d g | _ t d d f | _ | j | | _
| j d S)a Initialises the HTMLInputStream.
HTMLInputStream(source, [encoding]) -> Normalized stream from source
for use by html5lib.
source can be either a file-object, local filename or a string.
The optional encoding parameter must be a string that indicates
the encoding. If specified, that encoding will be used,
regardless of any BOM or later declaration (such as in a meta
element)
Nu r r zutf-8certain) r supports_lone_surrogatesreportCharacterErrorsr# characterErrorsUCS4characterErrorsUCS2newLineslookupEncoding charEncoding
openStream
dataStreamreset)r! rK r r r r" s z HTMLUnicodeInputStream.__init__c C sC d | _ d | _ d | _ g | _ d | _ d | _ d | _ d S)N r )r% chunkSize chunkOffseterrors prevNumLines prevNumCols_bufferedCharacter)r! r r r rZ s z HTMLUnicodeInputStream.resetc C s( t | d r | } n t | } | S)zvProduces a file object from source.
source can be either a file object, local filename or a string.
r/ )rG r )r! rK r r r r rX s z!HTMLUnicodeInputStream.openStreamc C st | j } | j d d | } | j | } | j d d | } | d k r\ | j | } n | | d } | | f S)N
r r r )r% countr_ rfindr` )r! r) r% nLines positionLine lastLinePospositionColumnr r r _position s
z HTMLUnicodeInputStream._positionc C s& | j | j \ } } | d | f S)z:Returns (line, col) of the current position in the stream.r )ri r] )r! linecolr r r r s z HTMLUnicodeInputStream.positionc C sI | j | j k r" | j s" t S| j } | j | } | d | _ | S)zo Read one character from the stream or queue if available. Return
EOF when EOF is reached.
r )r] r\ readChunkr r% )r! r] charr r r rm s
zHTMLUnicodeInputStream.charNc C sI | d k r | j } | j | j \ | _ | _ d | _ d | _ d | _ | j j | } | j r | j | } d | _ n
| s d St
| d k r t | d } | d k s d | k o d k n r | d
| _ | d d } | j r | j | | j
d d } | j
d
d } | | _ t
| | _ d S)Nr[ r Fr
i i z
rb
Tr r r )_defaultChunkSizeri r\ r_ r` r% r] rY r/ ra r# ordrR replace)r! r\ r2 lastvr r r rl s0
(
z HTMLUnicodeInputStream.readChunkc C s: x3 t t t j | D] } | j j d q Wd S)Nzinvalid-codepoint)ranger# invalid_unicode_refindallr^ r1 )r! r2 _r r r rS s "z*HTMLUnicodeInputStream.characterErrorsUCS4c C s d } x t j | D] } | r% q t | j } | j } t j | | | d r t j | | | d } | t k r | j j
d d } q | d k r | d k r | t | d k r | j j
d q d } | j j
d q Wd S)NF zinvalid-codepointTi i r ) ru finditerrq groupstartr isSurrogatePairsurrogatePairToCodepointnon_bmp_invalid_codepointsr^ r1 r# )r! r2 skipmatch codepointr$ char_valr r r rT # s z*HTMLUnicodeInputStream.characterErrorsUCS2Fc
C se y t | | f } Wn t k
r x& | D] } t | d k s+ t q+ Wd j d d | D } | sy d | } t j d | } t | | f .z^%sz[%s]+N)charsUntilRegExKeyErrorrq r( r4 recompiler r% r] r\ endr1 rl )
r!
charactersoppositecharsr regexr6 mr rr r r
charsUntil: s2
& z!HTMLUnicodeInputStream.charsUntilc C sl | t k rh | j d k r= | | j | _ | j d 7_ n+ | j d 8_ | j | j | k sh t d S)Nr r )r r] r% r\ r( )r! rm r r r ungeti s z HTMLUnicodeInputStream.unget)r; r< r= r> rp r" rZ rX ri r rm rl rS rT r r r r r r rI s & /rI c @ s e Z d Z d Z d d d d d d d d Z d d Z d d
Z d d d Z d
d Z d d Z d d Z
d S)rJ zProvides a unicode stream of characters to the HTMLTokenizer.
This class takes care of character encoding and removing or replacing
incorrect byte-sequences and also provides column and line tracking.
Nz windows-1252Tc C s | j | | _ t j | | j d | _ d | _ | | _ | | _ | | _ | | _ | | _
| j | | _ | j d d k s t
| j d S)a Initialises the HTMLInputStream.
HTMLInputStream(source, [encoding]) -> Normalized stream from source
for use by html5lib.
source can be either a file-object, local filename or a string.
The optional encoding parameter must be a string that indicates
the encoding. If specified, that encoding will be used,
regardless of any BOM or later declaration (such as in a meta
element)
i d r N)rX rawStreamrI r" numBytesMetanumBytesChardetoverride_encodingtransport_encodingsame_origin_parent_encodinglikely_encodingdefault_encodingdetermineEncodingrW r( rZ )r! rK r r r r r
useChardetr r r r" s z HTMLBinaryInputStream.__init__c C s3 | j d j j | j d | _ t j | d S)Nr rr )rW
codec_info streamreaderr rY rI rZ )r! r r r rZ s "zHTMLBinaryInputStream.resetc C s` t | d r | } n t | } y | j | j Wn t k
r[ t | } Yn X| S)zvProduces a file object from source.
source can be either a file object, local filename or a string.
r/ )rG r r+ r& Exceptionr )r! rK r r r r rX s
z HTMLBinaryInputStream.openStreamc C s | j d f } | d d k r& | St | j d f } | d d k rO | St | j d f } | d d k rx | S| j d f } | d d k r | St | j d f } | d d k r | d j j d r | St | j d f } | d d k r| S| ry d d l m
} Wn t k
r5Yn Xg } | } xX | j s| j
j | j } t | t s{t | sP| j | | j | qHW| j t | j d } | j
j d | d k r| d f St | j d f } | d d k r | St d d f S)NrP r tentativezutf-16)UniversalDetectorencodingz windows-1252) detectBOMrV r r detectEncodingMetar name
startswithr %pip._vendor.chardet.universaldetectorr ImportErrordoner r/ r rB r. r( r1 feedcloseresultr+ r )r! chardetrW r buffersdetectorr r r r r r sR '
z'HTMLBinaryInputStream.determineEncodingc C s | j d d k s t t | } | d k r5 d S| j d k re t d } | d k s t nr | | j d k r | j d d f | _ nF | j j d | d f | _ | j t d | j d | f d S) Nr rP utf-16beutf-16lezutf-8r z Encoding changed from %s to %s)r r )rW r( rV r r r+ rZ r )r! newEncodingr r r changeEncoding s
z$HTMLBinaryInputStream.changeEncodingc
C s t j d t j d t j d t j d t j d i } | j j d } t | t sZ t
| j | d d } d } | s | j | } d } | s | j | d d } d } | r | j j | t
| S| j j d
d Sd S) zAttempts to detect at BOM at the start of the stream. If
an encoding can be determined from the BOM return the name of the
encoding otherwise return Nonezutf-8zutf-16lezutf-16bezutf-32lezutf-32be N rx r )codecsBOM_UTF8 BOM_UTF16_LE BOM_UTF16_BE BOM_UTF32_LE BOM_UTF32_BEr r/ rB r. r( getr+ rV )r! bomDictstringr r+ r r r r s$
z HTMLBinaryInputStream.detectBOMc C s} | j j | j } t | t s* t t | } | j j d | j } | d k ry | j d k ry t
d } | S)z9Report the encoding declared by the meta element
r Nutf-16beutf-16lezutf-8)r r ) r r/ r rB r. r( EncodingParserr+ getEncodingr rV )r! r parserr r r r r 3 s z(HTMLBinaryInputStream.detectEncodingMeta) r; r< r= r> r" rZ rX r r r r r r r r rJ z s ( > "rJ c @ s e Z d Z d Z d d Z d d Z d d Z d d Z d
d Z d d
Z d d Z
d d Z e e e
Z
d d Z e e Z e d d Z d d Z d d Z d d Z d S)
EncodingByteszString-like object with an associated position and various extra methods
If the position is ever greater than the string length then an exception is
raisedc C s+ t | t s t t j | | j S)N)rB r. r( __new__lower)r! valuer r r r F s zEncodingBytes.__new__c C s
d | _ d S)Nr r )ri )r! r r r r r" J s zEncodingBytes.__init__c C s | S)Nr )r! r r r __iter__N s zEncodingBytes.__iter__c C sS | j d } | _ | t | k r/ t n | d k rA t | | | d S)Nr r )ri r#
StopIterationrH )r! pr r r __next__Q s zEncodingBytes.__next__c C s
| j S)N)r )r! r r r nextY s zEncodingBytes.nextc C sY | j } | t | k r$ t n | d k r6 t | d | _ } | | | d S)Nr r )ri r# r rH )r! r r r r previous] s zEncodingBytes.previousc C s( | j t | k r t | | _ d S)N)ri r# r )r! r r r r setPositionf s zEncodingBytes.setPositionc C s9 | j t | k r t | j d k r1 | j Sd Sd S)Nr )ri r# r )r! r r r getPositionk s
zEncodingBytes.getPositionc C s | | j | j d S)Nr )r )r! r r r getCurrentByteu s z EncodingBytes.getCurrentBytec C sf | j } xM | t | k rX | | | d } | | k rK | | _ | S| d 7} q W| | _ d S)z Skip past a list of charactersr N)r r# ri )r! r r r r r r r z s zEncodingBytes.skipc C sf | j } xM | t | k rX | | | d } | | k rK | | _ | S| d 7} q W| | _ d S)Nr )r r# ri )r! r r r r r r skipUntil s zEncodingBytes.skipUntilc C s4 | j | | j } | r0 | j t | 7_ | S)zLook for a sequence of bytes at the start of a string. If the bytes
are found return True and advance the position to the byte after the
match. Otherwise return False and leave the position alone)r r r# )r! r. r6 r r r
matchBytes s zEncodingBytes.matchBytesc C sI y* | j | | j t | d | _ Wn t k
rD t Yn Xd S)zLook for the next sequence of bytes matching a given sequence. If
a match is found advance the position to the last byte of the matchr T)indexr r# ri
ValueErrorr )r! r. r r r jumpTo s
*
zEncodingBytes.jumpToN)r; r< r= r> r r" r r r r r r propertyr r currentBytespaceCharactersBytesr r r r r r r r r B s r c @ s e Z d Z d Z d d Z d d Z d d Z d d Z d
d Z d d
Z d d Z
d d Z d d Z d S)r z?Mini parser for detecting character encoding from meta elementsc C s t | | _ d | _ d S)z3string - the data to work on for encoding detectionN)r r2 r )r! r2 r r r r" s zEncodingParser.__init__c C s d | j k r d Sd | j f d | j f d | j f d | j f d | j f d | j f f } x | j D] } d } y | j j d Wn t k
r PYn XxP | D]H \ } } | j j | r y | } PWq t k
r d } PYq Xq W| sk Pqk W| j S) Ns )r2 r )r! r r r r s z EncodingParser.handleCommentc C s< | j j t k r d Sd } d } x| j } | d k rA d S| d d k r | d d k } | r4| d k r4| | _ d Sq% | d d k r | d } t | } | d k r4| | _ d Sq% | d d k r% t t | d } | j } | d k r% t | } | d k r% | r.| | _ d S| } q% Wd S) NTFr s
http-equivr s content-types charsets content) r2 r r getAttributer rV ContentAttrParserr parse)r! hasPragmapendingEncodingattrtentativeEncodingcodec
contentParserr r r r s:
zEncodingParser.handleMetac C s
| j d S)NF)handlePossibleTag)r! r r r r s z%EncodingParser.handlePossibleStartTagc C s t | j | j d S)NT)r r2 r )r! r r r r s
z#EncodingParser.handlePossibleEndTagc C s | j } | j t k r6 | r2 | j | j d S| j t } | d k r^ | j n+ | j } x | d k r | j } qm Wd S)NTr )r2 r asciiLettersBytesr r r spacesAngleBracketsr )r! endTagr2 r r r r r r s
z EncodingParser.handlePossibleTagc C s | j j d S)Nr )r2 r )r! r r r r s zEncodingParser.handleOtherc C s | j } | j t t d g B } | d k sI t | d k sI t | d k rY d Sg } g } x | d k r~ | r~ Pnz | t k r | j } Pn^ | d
k r d j | d f S| t k r | j | j n | d k r d S| j | t
| } qh W| d k r1| j d j | d f St
| | j } | d k r| } x t
| } | | k rt
| d j | d j | f S| t k r| j | j q\| j | q\Wn^ | d k rd j | d f S| t k r| j | j n | d k r!d S| j | xw t
| } | t k red j | d j | f S| t k r| j | j q1| d k rd S| j | q1Wd S) z_Return a name,value pair for the next attribute in the stream,
if one is found, or None /Nr r =r3 ' ")r N)r r )r r )
r2 r r frozensetr# r( r4 asciiUppercaseBytesr1 r r r r )r! r2 r attrName attrValue quoteCharr r r r sh $
zEncodingParser.getAttributeN)
r; r< r= r> r" r r r r r r r r r r r r r s $ r c @ s( e Z d Z d d Z d d Z d S)r c C s" t | t s t | | _ d S)N)rB r. r( r2 )r! r2 r r r r" a s zContentAttrParser.__init__c C sN y1| j j d | j j d 7_ | j j | j j d k sH d S| j j d 7_ | j j | j j d k r | j j } | j j d 7_ | j j } | j j | r | j | | j j Sd Sn] | j j } y+ | j j t | j | | j j SWn# t k
r/| j | d SYn XWn t k
rId SYn Xd S)Ns charsetr r r r )r r )r2 r r r r r r r )r! quoteMark oldPositionr r r r e s.
zContentAttrParser.parseN)r; r< r= r" r r r r r r ` s r c C s| t | t r; y | j d } Wn t k
r: d SYn X| d k rt y t j | SWqx t k
rp d SYqx Xn d Sd S)z{Return the python codec name corresponding to an encoding or None if the
string doesn't correspond to a valid encoding.r N)rB r. decodeUnicodeDecodeErrorr
lookupAttributeError)r r r r rV s
rV r r )1
__future__r r r Zpip._vendor.sixr pip._vendor.six.movesr r r r ior r pip._vendorr
constantsr r
r r r r[ r r r r r r invalid_unicode_no_surrogaterQ rc r( r evalru r~ ascii_punctuation_rer objectr rO rI rJ r. r r r rV r r r r sJ " +
J b'