Rew 1 @ s d d l m Z m Z m Z d d l m Z d d l Z d d l Z d d l m Z m
Z
m Z m Z d d l m
Z
m Z d d l m Z d d l m Z y d d l m Z Wn e k
r e Z Yn Xy d d
l m Z Wn( e k
r Gd d d e Z Yn Xe d
d e
D Z e d d e D Z e d d e D Z e e d d g BZ e j d Z e d d d d d d d d d d d d d d! d" d# d$ d% d& d' d( d) d* d+ d, d- d. d/ d0 d1 d2 d3 g Z e j d4 Z i Z! Gd5 d6 d6 e Z" d d7 d7 d8 d9 Z# Gd: d; d; e Z$ Gd< d= d= e$ Z% Gd> d? d? e& Z' Gd@ dA dA e Z( GdB dC dC e Z) dD dE Z* d S)F )absolute_importdivisionunicode_literals) text_typeN )EOFspaceCharacters asciiLettersasciiUppercase) encodingsReparseException)utils)StringIO)BytesIO)BufferedIOBasec @ s e Z d Z d S)r N)__name__
__module__ __qualname__ r r ;/tmp/pip-ztkk0jow-build/pip/_vendor/html5lib/inputstream.pyr s r c C s g | ] } | j d q S)ascii)encode).0itemr r r
s r c C s g | ] } | j d q S)r )r )r r r r r r s c C s g | ] } | j d q S)r )r )r r r r r r s > t | j | | k rd | t | j | 8} | d 7} q' W| | g | _ d S)Nr r )_bufferedBytesAssertionErrorr$ r r! )r" r% offsetir r r seekA s zBufferedStream.seekc C sp | j s | j | S| j d t | j k r_ | j d t | j d k r_ | j | S| j | Sd S)Nr r r )r _readStreamr! r$ _readFromBuffer)r" bytesr r r readJ s
zBufferedStream.readc C s t d d | j D S)Nc S s g | ] } t | q Sr )r$ )r r r r r r T s z1BufferedStream._bufferedBytes..)sumr )r" r r r r( S s z BufferedStream._bufferedBytesc C sJ | j j | } | j j | | j d d 7 Normalized stream from source
for use by html5lib.
source can be either a file-object, local filename or a string.
The optional encoding parameter must be a string that indicates
the encoding. If specified, that encoding will be used,
regardless of any BOM or later declaration (such as in a meta
element)
parseMeta - Look for a element containing encoding information
u r u [-]u0 ([-](?![-])|(? s z5HTMLUnicodeInputStream.charsUntil..z^%sz[%s]+N)charsUntilRegExKeyErrorrk r) r5 rL rM rz r& rW rV endr2 rf )
r"
charactersoppositecharsr~ regexr7 mr rr r r
charsUntil0 s2
& z!HTMLUnicodeInputStream.charsUntilc C so | d k rk | j d k r= | | j | _ | j d 7_ qk | j d 8_ | j | j | k sk t n d S)Nr r )rW r& rV r) )r" rg r r r unget_ s z HTMLUnicodeInputStream.unget)r r r r< rj r# rT rR rc r! rg rf rJ rO r r r r r r r@ s ! ( /r@ c @ sy e Z d Z d Z d d d d d Z d d Z d d Z d d d
d Z d d
Z d d Z d d Z
d S)rA zProvides a unicode stream of characters to the HTMLTokenizer.
This class takes care of character encoding and removing or replacing
incorrect byte-sequences and also provides column and line tracking.
NTc C s | j | | _ t j | | j t | d f | _ d | _ d | _ d | _ | j d d k r | j | | | _ n | j
d S)a Initialises the HTMLInputStream.
HTMLInputStream(source, [encoding]) -> Normalized stream from source
for use by html5lib.
source can be either a file-object, local filename or a string.
The optional encoding parameter must be a string that indicates
the encoding. If specified, that encoding will be used,
regardless of any BOM or later declaration (such as in a meta
element)
parseMeta - Look for a element containing encoding information
rI i d z windows-1252r N) rR rawStreamr@ r# codecNamerQ numBytesMetanumBytesChardetdefaultEncodingdetectEncodingrT )r" rB rC rD rE r r r r# x s z HTMLBinaryInputStream.__init__c C s6 t j | j d | j d | _ t j | d S)Nr rm )codecs getreaderrQ r rS r@ rT )r" r r r rT s zHTMLBinaryInputStream.resetc C sV t | d r | } n t | } y | j | j Wn t | } Yn X| S)zvProduces a file object from source.
source can be either a file object, local filename or a string.
r0 )r= r r, r' r )r" rB r r r r rR s z HTMLBinaryInputStream.openStreamc
C s | j } d } | d k r9 | r9 | j } d } n | d k r:| r:d } y y d d l m } Wn" t k
r d d l m } Yn Xg } | } x[ | j s | j j | j } t
| t s t | s Pn | j
| | j | q W| j | j d } | j j d Wq:t k
r6Yq:Xn | d k rXd } | j } n i d d 6} | j | k r| | j } n | | f S)NrI tentativer )UniversalDetectorrC z windows-1252z
iso-8859-1) detectBOMdetectEncodingMetacharade.universaldetectorr ImportErrorchardet.universaldetectordoner r0 r r> r/ r) r2 feedcloseresultr, r lower)
r" rD rE rC
confidencer buffersdetectorr encodingSubr r r r sB
z$HTMLBinaryInputStream.detectEncodingc C s | j d d k s t t | } | d k r: d } n | d k rJ d S| | j d k rv | j d d f | _ nF | j j d | j | d f | _ t d | j d | f d S)
Nr rI utf-16 utf-16-be utf-16-lezutf-8r z Encoding changed from %s to %s)r r r )rQ r) r r r, rT r )r" newEncodingr r r changeEncoding s
z$HTMLBinaryInputStream.changeEncodingc C s i d t j 6d t j 6d t j 6d t j 6d t j 6} | j j d } t | t s_ t
| j | d d } d } | s | j | } d } | s | j | d d } d } q n | j j | r | p d
| S) zAttempts to detect at BOM at the start of the stream. If
an encoding can be determined from the BOM return the name of the
encoding otherwise return Nonezutf-8z utf-16-lez utf-16-bez utf-32-lez utf-32-be N rr r )
r BOM_UTF8 BOM_UTF16_LE BOM_UTF16_BE BOM_UTF32_LE BOM_UTF32_BEr r0 r> r/ r) getr, )r" bomDictstringrC r, r r r r s
z HTMLBinaryInputStream.detectBOMc C sk | j j | j } t | t s* t t | } | j j d | j } | d k rg d } n | S)z9Report the encoding declared by the meta element
r utf-16 utf-16-be utf-16-lezutf-8)r r r ) r r0 r r> r/ r) EncodingParserr, getEncoding)r" r parserrC r r r r s z(HTMLBinaryInputStream.detectEncodingMeta) r r r r< r# rT rR r r r r r r r r rA p s ( - rA c @ s e Z d Z d Z d d Z d d Z d d Z d d Z d
d Z d d
Z d d Z
d d Z e e e
Z
d d Z e e Z e d d Z d d Z d d Z d d Z d S)
EncodingByteszString-like object with an associated position and various extra methods
If the position is ever greater than the string length then an exception is
raisedc C s+ t | t s t t j | | j S)N)r> r/ r) __new__r )r" valuer r r r & s zEncodingBytes.__new__c C s
d | _ d S)Nr r )rc )r" r r r r r# * s zEncodingBytes.__init__c C s | S)Nr )r" r r r __iter__- s zEncodingBytes.__iter__c C sV | j d } | _ | t | k r/ t n | d k rD t n | | | d S)Nr r )rc r$
StopIterationr? )r" pr r r __next__0 s zEncodingBytes.__next__c C s
| j S)N)r )r" r r r next8 s zEncodingBytes.nextc C s\ | j } | t | k r$ t n | d k r9 t n | d | _ } | | | d S)Nr r )rc r$ r r? )r" r r r r previous< s zEncodingBytes.previousc C s+ | j t | k r t n | | _ d S)N)rc r$ r )r" r! r r r setPositionE s zEncodingBytes.setPositionc C s< | j t | k r t n | j d k r4 | j Sd Sd S)Nr )rc r$ r )r" r r r getPositionJ s
zEncodingBytes.getPositionc C s | | j | j d S)Nr )r! )r" r r r getCurrentByteT s z EncodingBytes.getCurrentBytec C sf | j } xM | t | k rX | | | d } | | k rK | | _ | S| d 7} q W| | _ d S)z Skip past a list of charactersr N)r! r$ rc )r" r r r~ r r r ry Y s zEncodingBytes.skipc C sf | j } xM | t | k rX | | | d } | | k rK | | _ | S| d 7} q W| | _ d S)Nr )r! r$ rc )r" r r r~ r r r skipUntile s zEncodingBytes.skipUntilc C sT | j } | | | t | } | j | } | rP | j t | 7_ n | S)zLook for a sequence of bytes at the start of a string. If the bytes
are found return True and advance the position to the byte after the
match. Otherwise return False and leave the position alone)r! r$
startswith)r" r/ r r3 r7 r r r
matchBytesp s zEncodingBytes.matchBytesc C sn | | j d j | } | d k rd | j d k rC d | _ n | j | t | d 7_ d St d S)zLook for the next sequence of bytes matching a given sequence. If
a match is found advance the position to the last byte of the matchNr r Tr r )r! findrc r$ r )r" r/ newPositionr r r jumpTo{ s zEncodingBytes.jumpToN)r r r r< r r# r r r r r r propertyr! r currentBytespaceCharactersBytesry r r r r r r r r " s r c @ s e Z d Z d Z d d Z d d Z d d Z d d Z d
d Z d d
Z d d Z
d d Z d d Z d S)r z?Mini parser for detecting character encoding from meta elementsc C s t | | _ d | _ d S)z3string - the data to work on for encoding detectionN)r r3 rC )r" r3 r r r r# s zEncodingParser.__init__c
C s d | j f d | j f d | j f d | j f d | j f d | j f f } xw | j D]l } d } xS | D]K \ } } | j j | rk y | } PWq t k
r d } PYq Xqk qk W| sX PqX qX W| j S) Ns )r3 r )r" r r r r s z EncodingParser.handleCommentc C sE | j j t k r d Sd } d } x | j } | d k rA d S| d d k r | d d k } | r=| d k r=| | _ d Sq% | d d k r | d } t | } | d k r=| | _ d Sq% | d d k r% t t | d } | j } | d k r=t | } | d k r:| r.| | _ d S| } q:q=q% q% Wd S) NTFr s
http-equivr s content-types charsets content) r3 r r getAttributerC r ContentAttrParserr parse)r" hasPragmapendingEncodingattrtentativeEncodingcodec
contentParserr r r r s:
zEncodingParser.handleMetac C s
| j d S)NF)handlePossibleTag)r" r r r r s z%EncodingParser.handlePossibleStartTagc C s t | j | j d S)NT)r r3 r )r" r r r r s
z#EncodingParser.handlePossibleEndTagc C s | j } | j t k r9 | r5 | j | j n d S| j t } | d k ra | j n+ | j } x | d k r | j } qp Wd S)NTr )r3 r asciiLettersBytesr r r spacesAngleBracketsr )r" endTagr3 r~ r r r r r s
z EncodingParser.handlePossibleTagc C s | j j d S)Nr )r3 r )r" r r r r s zEncodingParser.handleOtherc C s | j } | j t t d g B } | d k sI t | d k sI t | d k rY d Sg } g } x | d k r~ | r~ Pnz | t k r | j } Pn^ | d
k r d j | d f S| t k r | j | j n | d k r d S| j | t
| } qh W| d k r1| j d j | d f St
| | j } | d k r| } x t
| } | | k rt
| d j | d j | f S| t k r| j | j q\| j | q\Wn^ | d k rd j | d f S| t k r| j | j n | d k r!d S| j | xw t
| } | t k red j | d j | f S| t k r| j | j q1| d k rd S| j | q1Wd S) z_Return a name,value pair for the next attribute in the stream,
if one is found, or None /Nr r =r4 ' ")r N)r r )r r )
r3 ry r frozensetr$ r) r5 asciiUppercaseBytesr2 r r r r )r" r3 r~ attrName attrValue quoteCharr r r r sh $
zEncodingParser.getAttributeN)
r r r r< r# r r r r r r r r r r r r r s $ r c @ s( e Z d Z d d Z d d Z d S)r c C s" t | t s t | | _ d S)N)r> r/ r) r3 )r" r3 r r r r# ? s zContentAttrParser.__init__c C sN y1| j j d | j j d 7_ | j j | j j d k sH d S| j j d 7_ | j j | j j d k r | j j } | j j d 7_ | j j } | j j | r | j | | j j Sd Sn] | j j } y+ | j j t | j | | j j SWn# t k
r/| j | d SYn XWn t k
rId SYn Xd S)Ns charsetr r r r )r r )r3 r r! ry r r r r )r" quoteMark oldPositionr r r r C s.
zContentAttrParser.parseN)r r r r# r r r r r r > s r c C st t | t r> y | j d } Wq> t k
r: d SYq> Xn | rl t j d | j } t j | d Sd Sd S)z{Return the python codec name corresponding to an encoding or None if the
string doesn't correspond to a valid encoding.r NrU ) r> r/ decodeUnicodeDecodeErrorascii_punctuation_rerl r r r )rC
canonicalNamer r r r e s
r )+
__future__r r r Zpip._vendor.sixr r rL constantsr r r r
r r rU r
ior r r r objectr r r r r rM rp setrx r r r rG r@ rA r/ r r r r r r r r sJ "
Jg'