Re, @ s d d l m Z m Z m Z y
e Z Wn e k
r: Yn Xd d l m Z d d l m
Z
d d l m Z d d l m Z m
Z
d d l m Z m Z m Z d d l m Z m Z d d l m Z d d
l m Z d d l m Z e e Z Gd d
d
e Z d S) )absolute_importdivisionunicode_literals)deque )spaceCharacters)entities) asciiLettersasciiUpper2Lower)digits hexDigitsEOF)
tokenTypes
tagTokenTypes)replacementCharacters)HTMLInputStream)Triec s e Z d Z d Z d d d d d d f d d Z d d Z d d Z d d
d d Z d
d Z d d Z d d Z
d d Z d d Z d d Z
d d Z d d Z d d Z d d Z d! d" Z d# d$ Z d% d& Z d' d( Z d) d* Z d+ d, Z d- d. Z d/ d0 Z d1 d2 Z d3 d4 Z d5 d6 Z d7 d8 Z d9 d: Z d; d< Z d= d> Z d? d@ Z! dA dB Z" dC dD Z# dE dF Z$ dG dH Z% dI dJ Z& dK dL Z' dM dN Z( dO dP Z) dQ dR Z* dS dT Z+ dU dV Z, dW dX Z- dY dZ Z. d[ d\ Z/ d] d^ Z0 d_ d` Z1 da db Z2 dc dd Z3 de df Z4 dg dh Z5 di dj Z6 dk dl Z7 dm dn Z8 do dp Z9 dq dr Z: ds dt Z; du dv Z< dw dx Z= dy dz Z> d{ d| Z? d} d~ Z@ d d ZA d d ZB d d ZC d d ZD d d ZE d d ZF d d ZG d d ZH d d ZI d d ZJ d d ZK d d ZL S)
HTMLTokenizera This class takes care of tokenizing HTML.
* self.currentToken
Holds the token that is currently being processed.
* self.state
Holds a reference to the method to be invoked... XXX
* self.stream
Points to HTMLInputStream object.
NTc sz t | | | | | _ | | _ | | _ | | _ d | _ g | _ | j | _ d | _ d | _
t t | j
d S)NF)r streamparserlowercaseElementNamelowercaseAttrName
escapeFlag
lastFourChars dataStatestateescape currentTokensuperr __init__)selfr encoding parseMeta
useChardetr r r ) __class__ 9/tmp/pip-ztkk0jow-build/pip/_vendor/html5lib/tokenizer.pyr % s zHTMLTokenizer.__init__c c s} t g | _ xg | j rx x6 | j j rV i t d d 6| j j j d d 6Vq! Wx | j rt | j j VqZ Wq Wd S)z This is where the magic happens.
We do our usually processing through the states and when we have a token
to return we yield the token which pauses processing until the next token
is requested.
ParseErrortyper dataN)r
tokenQueuer r errorsr poppopleft)r r% r% r& __iter__9 s * zHTMLTokenizer.__iter__c % C s t } d } | r! t } d } n g } | j j } x8 | | k rp | t k rp | j | | j j } q9 Wt d j | | } | t k r t | } | j j i t
d d 6d d 6i | d 6d 6 nd
| k o d k n s | d k r3d
} | j j i t
d d 6d d 6i | d 6d 6 nsd | k oJd k n s d | k ofd k n s d | k od k n s d | k od k n s | t d d d d d d d d d d d d! d" d# d$ d% d& d' d( d) d* d+ d, d- d. d/ d0 d1 d2 d3 d4 d5 d6 d7 d g# k rQ| j j i t
d d 6d d 6i | d 6d 6 n y t | } WnB t
k
r| d8 } t d
| d ?B t d9 | d: @B } Yn X| d; k r| j j i t
d d 6d< d 6 | j j | n | S)=zThis function returns either U+FFFD or the character based on the
decimal or hexadecimal representation. It also discards ";" if present.
If not present self.tokenQueue.append({"type": tokenTypes["ParseError"]}) is invoked.
r' r( z$illegal-codepoint-for-numeric-entityr) charAsIntdatavarsi i i u �r i i i i i i i i i i i i i i i i i i i i i i i
i
i i i i i
i
i i i i i i i i ;z numeric-entity-without-semicolon)r r r charr
appendintjoinr r* r frozensetchr
ValueErrorunget) r isHexallowedradix charStackcr2 r; vr% r% r& consumeNumberEntityI s`
+ z!HTMLTokenizer.consumeNumberEntityFc
C s d } | j j g } | d t k s] | d t d d f k s] | d k rt | | d k rt | j j | d n| d d k rpd } | j | j j | d d k r d } | j | j j n | r | d t k s | r"| d t k r"| j j | d | j | } qD| j j i t
d
d 6d d
6 | j j | j d d j | } nxF | d t k rt
j d j | sPn | j | j j qsWy2 t
j d j | d d } t | } Wn t k
rd } Yn X| d k r| d d k rG| j j i t
d
d 6d d
6 n | d d k r| r| | t k s| | t k s| | d k r| j j | j d d j | } qDt | } | j j | j | d j | | d 7} nK | j j i t
d
d 6d d
6 | j j | j d d j | } | rf| j d
d d | 7 entitiesTriehas_keys_with_prefixlongest_prefixlenKeyErrorr r r ) r allowedChar
fromAttributeoutputrF hex
entityName entityLength tokenTyper% r% r&
consumeEntity sf ) "
zHTMLTokenizer.consumeEntityc C s | j d | d d d S)zIThis method replaces the need for "entityInAttributeValueState".
rX rY TN)r_ )r rX r% r% r& processEntityInAttribute s z&HTMLTokenizer.processEntityInAttributec C s | j } | d t k r | j r< | d j t | d r' z'expected-tag-name-but-got-right-bracketrQ z<>?z'expected-tag-name-but-got-question-markzexpected-tag-namerK T)r r; markupDeclarationOpenStater closeTagOpenStater r r tagNameStater* r< r rB bogusCommentState)r r) r% r% r& ri r s6
" " zHTMLTokenizer.tagOpenStatec C s? | j j } | t k rS i t d d 6| d 6g d 6d d 6| _ | j | _ n | d k r | j j i t d d 6d d 6 | j | _ n | t
k r | j j i t d d 6d
d 6 | j j i t d d 6d d 6 | j | _ nL | j j i t d d 6d
d 6i | d 6d 6 | j j | | j | _ d S)Nrb r( ra r) Frc rx r' z*expected-closing-tag-but-got-right-bracketz expected-closing-tag-but-got-eofrQ z | j j i t d d 6d d 6 | j j | | j | _ d S)Nru r1 rQ r( rK r) T)
r r; temporaryBufferrcdataEndTagOpenStater r* r< r rB rn )r r) r% r% r& rm s " z%HTMLTokenizer.rcdataLessThanSignStatec C s{ | j j } | t k r9 | j | 7_ | j | _ n> | j j i t d d 6d d 6 | j j | | j
| _ d S)NrQ r( z | j j i t d d 6d d 6 | j j | | j | _ d S)Nru r1 rQ r( rK r) T)
r r; r rawtextEndTagOpenStater r* r< r rB rp )r r) r% r% r& ro s " z&HTMLTokenizer.rawtextLessThanSignStatec C s{ | j j } | t k r9 | j | 7_ | j | _ n> | j j i t d d 6d d 6 | j j | | j
| _ d S)NrQ r( z | j j i t d d 6d d 6 | j j | | j
| _ d S)
Nru r1 rt rQ r( z | j j i t d d 6d d 6 | j j | | j
| _ d S)NrQ r( z | j j i t d d 6d d 6 | j j
| | j | _ d S)Nru r1 rQ r( rK r) T) r r; r scriptDataEscapedEndTagOpenStater r r* r< r scriptDataDoubleEscapeStartStaterB r )r r) r% r% r& r s & " z0HTMLTokenizer.scriptDataEscapedLessThanSignStatec C su | j j } | t k r3 | | _ | j | _ n> | j j i t d d 6d d 6 | j j | | j
| _ d S)NrQ r( z
z'HTMLTokenizer.beforeAttributeValueStatec C s | j j } | d k r* | j | _ n | d k rF | j d n | d k r | j j i t d d 6d d 6 | j d d d d 7 | j j i t
d d 6d d 6 | j j | | j | _ d S) Nrx ru r' r( z$unexpected-EOF-after-attribute-valuer) z*unexpected-character-after-attribute-valueT)
r r; r r~ r rf r r
r* r< r rB r )r r) r% r% r& r * s"
z&HTMLTokenizer.afterAttributeValueStatec C s | j j } | d k r5 d | j d <| j n | t k r | j j i t d d 6d d 6 | j j | | j | _
n> | j j i t d d 6d d 6 | j j | | j | _
d S) Nrx Trc r' r( z#unexpected-EOF-after-solidus-in-tagr) z)unexpected-character-after-solidus-in-tag) r r; r rf r
r* r< r rB r r r~ )r r) r% r% r& r > s
z&HTMLTokenizer.selfClosingStartTagStatec C sc | j j d } | j d d } | j j i t d d 6| d 6 | j j | j | _ d S)Nrx rg u �Commentr( r) T) r rj replacer* r< r r; r r )r r) r% r% r& r} P s
z HTMLTokenizer.bogusCommentStatec C s0 | j j g } | d d k rv | j | j j | d! d k ri t d d 6d d 6| _ | j | _ d Sne| d" d# k rd } x> d* D]6 } | j | j j | d+ | k r d } Pq q W| ri t d d 6d d 6d d 6d d 6d d 6| _ | j | _ d Sn | d, d k r| j d k r| j j j
r| j j j
d- j | j j j k rd } xP d
d d d d d g D]6 } | j | j j | d. | k rd } PqqW| r| j
| _ d Sn | j j i t d d 6d d 6 x | r | j j | j q W| j | _ d S)/Nr r r r( r1 r) TdDoOrG CtTyYpPeEFDoctypera publicIdsystemIdcorrect[Ar' zexpected-dashes-or-doctyperR rR rR )r r r r rG r r r r r r r r r )r r r r r r rR rR rR rR )r r; r< r r commentStartStater doctypeStater tree openElements namespacedefaultNamespacecdataSectionStater* rB r, r} )r rF matchedexpectedr% r% r& rz _ sR
% z(HTMLTokenizer.markupDeclarationOpenStatec C s- | j j } | d k r* | j | _ n | d k rl | j j i t d d 6d d 6 | j d d 7 | j j i t d d 6d d 6 | j j | | j | _ d S) Nr' r( z!expected-doctype-name-but-got-eofr) Fr zneed-space-after-doctypeT) r r; r beforeDoctypeNameStater r
r* r< r r r rB )r r) r% r% r& r s
zHTMLTokenizer.doctypeStatec C s? | j j } | t k r n | d k r{ | j j i t d d 6d d 6 d | j d <| j j | j | j | _ n | d k r | j j i t d d 6d d 6 d
| j d <| j | _ nv | t
k r"| j j i t d d 6d d 6 d | j d <| j j | j | j | _ n | | j d <| j | _ d
S)Nrx r' r( z+expected-doctype-name-but-got-right-bracketr) Fr rg zinvalid-codepointu �ra z!expected-doctype-name-but-got-eofT) r r; r r* r< r r r r doctypeNameStater
)r r) r% r% r& r & s.
z$HTMLTokenizer.beforeDoctypeNameStatec C so | j j } | t k rG | j d j t | j d <| j | _ n$| d k r | j d j t | j d <| j j | j | j
| _ n | d k r | j j i t d d 6d d 6 | j d d 7<| j | _ n | t
k rZ| j j i t d d 6d d 6 d
| j d <| j d j t | j d <| j j | j | j
| _ n | j d | 7 | t k r{ | j j | | j j | j | j | _ n d S)Nrx T) r r; r* r< r r r r
rB )r r) r% r% r& r s z HTMLTokenizer.bogusDoctypeStatec C sf g } x | j | j j d | j | j j d | j j } | t k rZ Pq | d k sl t | d d d d k r | d d d | d countranger* r r r r )r r) r; nullCountr r% r% r& r s0 z HTMLTokenizer.cdataSectionState)M__name__
__module__ __qualname____doc__r r. rI r_ r` rf r rh rn rl rp rr rs ri r{ r| rm r r ro r r rq r r r r r r r r r r r r r r r r r~ r r r r r r r r r} rz r r r r r r r r r r r r r r r r r r r r r r r r% r% )r$ r& r s HP #
7 " - 3 r N)
__future__r r r unichrr@ NameError collectionsr constantsr r r r
r r r
r r r inputstreamr trier rS objectr r% r% r% r& s