z^%sz[%s]+N) charsUntilRegExKeyErrorr4 recompilerm r& rQ rP endr1 r\ ) r" Z
charactersZoppositecharsZregexr5 mrs rr r r
charsUntil@ s.
z!HTMLUnicodeInputStream.charsUntilc C s@ |d k r<| j dkr.|| j | _| jd7 _n| j d8 _ d S )Nr r
)rQ r& rP )r" r] r r r ungeto s
z HTMLUnicodeInputStream.unget)N)F)r6 r7 r8 r9 r` r# rN rL rY r! r] r\ rH rI rw rx r r r r r@ s
&
/r@ c @ sL e Zd ZdZdddZdd Zd d
Zdd d Zd
d Zdd Z dd Z
dS )rA zProvides a unicode stream of characters to the HTMLTokenizer.
This class takes care of character encoding and removing or replacing
incorrect byte-sequences and also provides column and line tracking.
N windows-1252Tc C s\ | j || _tj| | j d| _d| _|| _|| _|| _|| _ || _
| j || _ | j
dS )a Initialises the HTMLInputStream.
HTMLInputStream(source, [encoding]) -> Normalized stream from source
for use by html5lib.
source can be either a file-object, local filename or a string.
The optional encoding parameter must be a string that indicates
the encoding. If specified, that encoding will be used,
regardless of any BOM or later declaration (such as in a meta
element)
i d N)rL rawStreamr@ r# numBytesMetanumBytesChardetoverride_encodingtransport_encodingsame_origin_parent_encodinglikely_encodingdefault_encodingdetermineEncodingrK rN )r" rB r~ r r r r Z
useChardetr r r r# s z HTMLBinaryInputStream.__init__c C s& | j d jj| jd| _tj| d S )Nr rb )rK Z
codec_info streamreaderr{ rM r@ rN )r" r r r rN s zHTMLBinaryInputStream.resetc C sD t |dr|}nt|}y|j|j W n t|}Y nX |S )zvProduces a file object from source.
source can be either a file object, local filename or a string.
r. )r> r r* r' r )r" rB r r r r rL s
z HTMLBinaryInputStream.openStreamc C s | j df}|d d k r |S t| jdf}|d d k r:|S t| jdf}|d d k rX|S | j df}|d d k rt|S t| jdf}|d d k r|d jjd r|S t| jdf}|d d k r|S |rdyddl m
} W n t k
r Y nxX g }| }x6|j s.| j
j| j}|sP |j| |j| qW |j t|jd }| j
jd |d k rd|dfS t| jdf}|d d k r|S tddfS )NrE r Z tentativezutf-16)UniversalDetectorencodingz windows-1252) detectBOMrJ r~ r detectEncodingMetar name
startswithr Zchardet.universaldetectorr ImportErrordoner{ r. r} r1 Zfeedcloseresultr* r )r" ZchardetrK r ZbuffersZdetectorr r r r r r sP
z'HTMLBinaryInputStream.determineEncodingc C s t |}|d krd S |jdkr(t d}nT|| jd krH| jd df| _n4| jjd |df| _| j td| jd |f d S )Nutf-16beutf-16lezutf-8r rE z Encoding changed from %s to %s)r r )rJ r rK r{ r* rN r )r" Z newEncodingr r r changeEncoding s
z$HTMLBinaryInputStream.changeEncodingc
C s t jdt jdt jdt jdt jdi}| jjd}|j|dd }d}|sp|j|}d}|sp|j|dd }d }|r| jj | t
|S | jj d
dS dS ) zAttempts to detect at BOM at the start of the stream. If
an encoding can be determined from the BOM return the name of the
encoding otherwise return Nonezutf-8zutf-16lezutf-16bezutf-32lezutf-32be N rg r ) codecsBOM_UTF8 BOM_UTF16_LE BOM_UTF16_BE BOM_UTF32_LE BOM_UTF32_BEr{ r. getr* rJ )r" ZbomDictstringr r* r r r r s"
z HTMLBinaryInputStream.detectBOMc C sH | j j| j}t|}| j jd |j }|dk rD|jdkrDtd}|S )z9Report the encoding declared by the meta element
r Nutf-16beutf-16lezutf-8)r r )r{ r. r| EncodingParserr* getEncodingr rJ )r" r parserr r r r r 9 s z(HTMLBinaryInputStream.detectEncodingMeta)NNNNry T)T) r6 r7 r8 r9 r# rN rL r r r r r r r r rA s
(
>"rA c @ s e Zd ZdZdd Zdd Zdd Zdd Zd
d Zd d
Z dd Z
dd Z e e e
Z
dd Ze eZefddZdd Zdd Zdd Zd S )
EncodingByteszString-like object with an associated position and various extra methods
If the position is ever greater than the string length then an exception is
raisedc C s t j| |j S )N)r- __new__lower)r" valuer r r r L s zEncodingBytes.__new__c C s
d| _ d S )Nr
r )rY )r" r r r r r# P s zEncodingBytes.__init__c C s | S )Nr )r" r r r __iter__T s zEncodingBytes.__iter__c C s> | j d }| _ |t| kr"tn |dk r.t| ||d S )Nr
r )rY r$
StopIterationr? )r" pr r r __next__W s zEncodingBytes.__next__c C s | j S )N)r )r" r r r next_ s zEncodingBytes.nextc C sB | j }|t| krtn |dk r$t|d | _ }| ||d S )Nr r
)rY r$ r r? )r" r r r r previousc s zEncodingBytes.previousc C s | j t| krt|| _ d S )N)rY r$ r )r" r! r r r setPositionl s zEncodingBytes.setPositionc C s* | j t| krt| j dkr"| j S d S d S )Nr )rY r$ r )r" r r r getPositionq s
zEncodingBytes.getPositionc C s | | j | j d S )Nr
)r! )r" r r r getCurrentByte{ s z EncodingBytes.getCurrentBytec C sL | j }x:|t| k r@| ||d }||kr6|| _|S |d7 }qW || _dS )z Skip past a list of charactersr
N)r! r$ rY )r" rt r rn r r r rl s zEncodingBytes.skipc C sL | j }x:|t| k r@| ||d }||kr6|| _|S |d7 }qW || _d S )Nr
)r! r$ rY )r" rt r rn r r r skipUntil s zEncodingBytes.skipUntilc C s> | j }| ||t| }|j|}|r:| j t|7 _ |S )zLook for a sequence of bytes at the start of a string. If the bytes
are found return True and advance the position to the byte after the
match. Otherwise return False and leave the position alone)r! r$ r )r" r- r r2 r5 r r r
matchBytes s
zEncodingBytes.matchBytesc C sR | | j d j|}|dkrJ| jdkr,d| _| j|t| d 7 _dS tdS )zLook for the next sequence of bytes matching a given sequence. If
a match is found advance the position to the last byte of the matchNr
r Tr r )r! findrY r$ r )r" r- Z newPositionr r r jumpTo s
zEncodingBytes.jumpToN)r6 r7 r8 r9 r r# r r r r r r propertyr! r currentBytespaceCharactersBytesrl r r r r r r r r H s
r c @ sX e Zd ZdZdd Zdd Zdd Zdd Zd
d Zd d
Z dd Z
dd Z dd Z dS )r z?Mini parser for detecting character encoding from meta elementsc C s t || _d| _dS )z3string - the data to work on for encoding detectionN)r r2 r )r" r2 r r r r# s
zEncodingParser.__init__c
C s d| j fd| jfd| jfd| jfd| jfd| jff}x^| jD ]T}d}xD|D ]<\}}| jj|rJy | }P W qJ tk
r d}P Y qJX qJW |s)r2 r )r" r r r r s z EncodingParser.handleCommentc C s | j jtkrdS d}d }x| j }|d kr.dS |d dkr^|d dk}|r|d k r|| _dS q|d dkr|d }t|}|d k r|| _dS q|d dkrtt|d }|j }|d k rt|}|d k r|r|| _dS |}qW d S ) NTFr s
http-equivr
s content-types charsets content) r2 r r getAttributer rJ ContentAttrParserr parse)r" Z hasPragmaZpendingEncodingattrZtentativeEncodingcodecZ
contentParserr r r r s: zEncodingParser.handleMetac C s
| j dS )NF)handlePossibleTag)r" r r r r s z%EncodingParser.handlePossibleStartTagc C s t | j | jdS )NT)r r2 r )r" r r r r s
z#EncodingParser.handlePossibleEndTagc C sf | j }|jtkr(|r$|j | j dS |jt}|dkrD|j n | j }x|d k r`| j }qNW dS )NTr )r2 r asciiLettersBytesr r r spacesAngleBracketsr )r" ZendTagr2 rn r r r r r s
z EncodingParser.handlePossibleTagc C s | j jdS )Nr )r2 r )r" r r r r s zEncodingParser.handleOtherc C s | j }|jttdgB }|dkr&dS g }g }xt|dkr@|r@P nX|tkrT|j }P nD|d krjdj|dfS |tkr|j|j n|dkrdS |j| t|}q0W |dkr|j dj|dfS t| |j }|d
kr:|}xt|}||krt| dj|dj|fS |tkr*|j|j q|j| qW nJ|dkrRdj|dfS |tkrl|j|j n|dkrzdS |j| x^t|}|t
krdj|dj|fS |tkr|j|j n|dkrdS |j| qW dS ) z_Return a name,value pair for the next attribute in the stream,
if one is found, or None /r N =r3 ' ")r N)r r )r r ) r2 rl r frozensetr4 asciiUppercaseBytesr1 r r r r )r" r2 rn ZattrNameZ attrValueZ quoteCharr r r r sf
zEncodingParser.getAttributeN)
r6 r7 r8 r9 r# r r r r r r r r r r r r r s $r c @ s e Zd Zdd Zdd ZdS )r c C s
|| _ d S )N)r2 )r" r2 r r r r# f s zContentAttrParser.__init__c C s y| j jd | j jd7 _| j j | j jdks8d S | j jd7 _| j j | j jdkr| j j}| j jd7 _| j j}| j j|r| j || j j S d S nF| j j}y | j jt | j || j j S tk
r | j |d S X W n tk
r d S X d S )Ns charsetr
r r r )r r )r2 r r! rl r r r r )r" Z quoteMarkZ oldPositionr r r r j s.
zContentAttrParser.parseN)r6 r7 r8 r# r r r r r r e s r c C s` t | tr.y| jd} W n tk
r, dS X | dk rXy
tj| S tk
rT dS X ndS dS )z{Return the python codec name corresponding to an encoding or None if the
string doesn't correspond to a valid encoding.r N)r< r decodeUnicodeDecodeErrorr lookupAttributeError)r r r r rJ s
rJ r )2Z
__future__r r r Zpip._vendor.sixr r Zpip._vendor.six.movesr r r rq Z pip._vendorr Z constantsr r r
r r rO r ior r r r r r r r Z invalid_unicode_no_surrogaterF rr evalrd setrk Zascii_punctuation_rero objectr rD r@ rA r- r r r rJ r r r r sV
J g Ih 6'