Re, @ s+ d d l m Z m Z m Z d d l m Z d d l m Z m Z d d l
m Z d d l m
Z
d d l m Z d d l m Z m Z d d l m Z m Z m Z d d
l m Z m Z d d l m Z d d l m Z d d
l m Z e e Z e d k r e Z n e Z Gd d d e Z d S) )absolute_importdivisionunicode_literals)unichr)deque OrderedDict) version_info )spaceCharacters)entities) asciiLettersasciiUpper2Lower)digits hexDigitsEOF)
tokenTypes
tagTokenTypes)replacementCharacters)HTMLInputStream)Trie c s e Z d Z d Z d f d d Z d d Z d d Z d d d
d Z d d
Z d d Z d d Z
d d Z d d Z d d Z
d d Z d d Z d d Z d d Z d d! Z d" d# Z d$ d% Z d& d' Z d( d) Z d* d+ Z d, d- Z d. d/ Z d0 d1 Z d2 d3 Z d4 d5 Z d6 d7 Z d8 d9 Z d: d; Z d< d= Z d> d? Z! d@ dA Z" dB dC Z# dD dE Z$ dF dG Z% dH dI Z& dJ dK Z' dL dM Z( dN dO Z) dP dQ Z* dR dS Z+ dT dU Z, dV dW Z- dX dY Z. dZ d[ Z/ d\ d] Z0 d^ d_ Z1 d` da Z2 db dc Z3 dd de Z4 df dg Z5 dh di Z6 dj dk Z7 dl dm Z8 dn do Z9 dp dq Z: dr ds Z; dt du Z< dv dw Z= dx dy Z> dz d{ Z? d| d} Z@ d~ d ZA d d ZB d d ZC d d ZD d d ZE d d ZF d d ZG d d ZH d d ZI d d ZJ d d ZK d d ZL S)
HTMLTokenizera This class takes care of tokenizing HTML.
* self.currentToken
Holds the token that is currently being processed.
* self.state
Holds a reference to the method to be invoked... XXX
* self.stream
Points to HTMLInputStream object.
Nc sb t | | | _ | | _ d | _ g | _ | j | _ d | _ d | _ t t
| j d S)NF) r streamparser
escapeFlag
lastFourChars dataStatestateescape currentTokensuperr __init__)selfr r kwargs) __class__ /builddir/build/BUILDROOT/alt-python35-pip-20.2.4-5.el8.x86_64/opt/alt/python35/lib/python3.5/site-packages/pip/_vendor/html5lib/_tokenizer.pyr" ( s zHTMLTokenizer.__init__c c s{ t g | _ xe | j rv x4 | j j rT d t d d | j j j d i Vq! Wx | j rr | j j VqX Wq Wd S)z This is where the magic happens.
We do our usually processing through the states and when we have a token
to return we yield the token which pauses processing until the next token
is requested.
type
ParseErrordatar N)r
tokenQueuer r errorsr poppopleft)r# r& r& r' __iter__7 s ( zHTMLTokenizer.__iter__c % C s t } d } | r t } d } g } | j j } x8 | | k rm | t k rm | j | | j j } q6 Wt d j | | } | t k r t | } | j j d t
d d d d d | i i nd
| k o d k n s | d k r(d
} | j j d t
d d d d d | i i nld | k o?d k n sd | k o[d k n sd | k owd k n sd | k od k n s| t d d d d d d d d d d d d! d" d# d$ d% d& d' d( d) d* d+ d, d- d. d/ d0 d1 d2 d3 d4 d5 d6 d7 d g# k r?| j j d t
d d d d d | i i y t | } WnB t
k
r| d8 } t d
| d ?B t d9 | d: @B } Yn X| d; k r| j j d t
d d d< i | j j | | S)=zThis function returns either U+FFFD or the character based on the
decimal or hexadecimal representation. It also discards ";" if present.
If not present self.tokenQueue.append({"type": tokenTypes["ParseError"]}) is invoked.
r( r) r* z$illegal-codepoint-for-numeric-entitydatavars charAsInti i i u �r i i i i i i i i i i i i i i i i i i i i i i i
i
i i i i i
i
i i i i i i i i ;z numeric-entity-without-semicolon)r r r charr appendintjoinr r+ r frozensetchr
ValueErrorunget) r# isHexallowedradix charStackcr4 r< vr& r& r' consumeNumberEntityG sb
+
z!HTMLTokenizer.consumeNumberEntityFc
C s d } | j j g } | d t k s] | d t d d f k s] | d k rt | | d k rt | j j | d n| d d k rkd } | j | j j | d d k r d } | j | j j | r | d t k s | r | d t k r | j j | d | j | } q5| j j d
t
d d d
i | j j | j d d j | } nxC | d t k rt
j d j | sP| j | j j qnWy2 t
j d j | d d } t | } Wn t k
rd } Yn X| d k r| d d k r:| j j d
t
d d d i | d d k r| r| | t k s| | t k s| | d k r| j j | j d d j | } q5t | } | j j | j | d j | | d 7} nI | j j d
t
d d d i | j j | j d d j | } | rW| j d d d | 7r) z'expected-tag-name-but-got-right-bracketrR z<>?z'expected-tag-name-but-got-question-markzexpected-tag-namerL T)r r< markupDeclarationOpenStater closeTagOpenStater r r tagNameStater+ r= r rC bogusCommentState)r# r* r& r& r' rn w s6
zHTMLTokenizer.tagOpenStatec C s1 | j j } | t k rO d t d d | d g d d i | _ | j | _ n | d k r | j j d t d d d i | j | _ n | t
k r | j j d t d d d
i | j j d t d d d i | j | _ nH | j j d t d d d
d d | i i | j j | | j | _ d S)Nr( rd rb r* re Fr| r) z*expected-closing-tag-but-got-right-bracketz expected-closing-tag-but-got-eofrR z | j j d t d d d i n | d k rY | j | _ n | d k r | j j d t d d d i | j | _ n | d k r | j j d t d d d i | j j d t d d d
i | j | _ nG | t k r
| j
| _ n, | j j d t d d | i | j | _ d S) Nr r( rR r* rL r| rl r) zinvalid-codepointu �T) r r< r+ r= r r r rw r r r )r# r* r& r& r' r s& #
z,HTMLTokenizer.scriptDataEscapedDashDashStatec C s | j j } | d k r3 d | _ | j | _ n | t k r{ | j j d t d d d | i | | _ | j | _ n< | j j d t d d d i | j j
| | j | _ d S)Nrz r2 r( rR r* rL T) r r< r scriptDataEscapedEndTagOpenStater r r+ r= r scriptDataDoubleEscapeStartStaterC r )r# r* r& r& r' r s $ z0HTMLTokenizer.scriptDataEscapedLessThanSignStatec C ss | j j } | t k r3 | | _ | j | _ n< | j j d t d d d i | j j | | j
| _ d S)Nr( rR r* z | j j d t d d d i n8| d k ry | j j d t d d d i | j | _ n | d k r | j j d t d d d i | j | _ n | d k r| j j d t d d d i | j j d t d d d
i | j | _ ng | t k rJ| j j d t d d d i | j
| _ n, | j j d t d d | i | j | _ d S)
Nr r( rR r* rL r| rl r) zinvalid-codepointu �zeof-in-script-in-scriptT) r r< r+ r= r r r rw r r r )r# r* r& r& r' r % s, #
z2HTMLTokenizer.scriptDataDoubleEscapedDashDashStatec C ss | j j } | d k rS | j j d t d d d i d | _ | j | _ n | j j | | j | _ d S)Nrz r( rR r* r2 T)
r r< r+ r= r r scriptDataDoubleEscapeEndStater rC r )r# r* r& r& r' r > s z6HTMLTokenizer.scriptDataDoubleEscapedLessThanSignStatec C s | j j } | t t d Bk rx | j j d t d d | i | j j d k ri | j | _
q | j | _
nZ | t k r | j j d t d d | i | j | 7_ n | j j
| | j | _
d S) Nrz r| r( rR r* r T)rz r| )r r< r
r@ r+ r= r r r r r r r rC )r# r* r& r& r' r I s z,HTMLTokenizer.scriptDataDoubleEscapeEndStatec C s | j j } | t k r1 | j j t d nt| t k rf | j d j | d g | j | _ n?| d k r | j n&| d k r | j
| _ n | d k r | j j d
t d d d i | j d j | d g | j | _ n | d
k rD| j j d
t d d d i | j d j d d g | j | _ na | t
k r| j j d
t d d d i | j | _ n& | j d j | d g | j | _ d S)NTr* r2 r| rz '"rP rL r( r) z#invalid-character-in-attribute-namerl zinvalid-codepointu �z#expected-attribute-name-but-got-eof)r r rP rL )r r< r
ro r r r= attributeNameStater rk r r+ r r r )r# r* r& r& r' r Y s6
z&HTMLTokenizer.beforeAttributeNameStatec C sc | j j } d } d } | d k r6 | j | _ n| t k rw | j d d d | | j j t d 7
z'HTMLTokenizer.beforeAttributeValueStatec C s | j j } | d k r* | j | _ n | d k rF | j d n | d k r | j j d t d d d i | j d d d d 7 | t k r{ | j j | | j j | j | j | _ n d S)Nr| T) r r< r+ r= r r r r rC )r# r* r& r& r' r s z HTMLTokenizer.bogusDoctypeStatec C s\ g } x | j | j j d | j | j j d | j j } | t k rZ Pq | d k sl t | d d d d k r | d d d | d s