3
G\o # @ s d Z ddlZddlmZ ddlmZ ddlmZmZm Z m
Z
ddl m Z ddl
mZ edZejd ejZejd
ejZejd Zyed d
d W n" ek
r ejdZdZY nBX ddlmZ ejdjej ZdZddl Z e j d= ddlZe`[ejdZ ejdZ edZ!edZ"edZ#edZ$edZ%ed Z&ed Z'ed Z(ed Z)ed Z*ed!Z+ed"Z,ed#Z-ed$Z.ed%Z/ed&Z0ed'Z1ed(Z2ed)Z3ed*Z4ed+Z5ed,Z6ed-Z7ed.Z8ed/Z9ed0Z:ed1Z;ed2Zed5Z?ed6Z@ed7ZAed8ZBed9ZCed:ZDed;ZEed<ZFed=ZGed>ZHed?ZIed@ZJedAZKedBZLedCZMedDZNedEZOedFZPedGZQe!e9e%e(e1e0e4e:e,e6e-e7e+e5e'e2e)e*e.e/e"e&e#e3e$e8dHZReSdIdJ e eRD ZTejdKdLjUdMdN eVeRdOdP dQD ZWeXeGeIeHe;eLeMeNgZYeXe;eOeIeNgZZdRdS Z[dTdU Z\dVdW Z]dXdY Z^dZd[ Z_G d\d] d]e`ZaG d^d_ d_ebZceG d`da dae`ZdeG dbdc dce`Zeddde ZfG dfdg dge`ZgdS )ha
jinja2.lexer
~~~~~~~~~~~~
This module implements a Jinja / Python combination lexer. The
`Lexer` class provided by this module is used to do some preprocessing
for Jinja.
On the one hand it filters out invalid operators like the bitshift
operators we don't allow in templates. On the other hand it separates
template code and python code in expressions.
:copyright: (c) 2017 by the Jinja Team.
:license: BSD, see LICENSE for more details.
N)deque)
itemgetter)implements_iteratorintern iteritems text_type)TemplateSyntaxError)LRUCache2 z\s+z7('([^'\\]*(?:\\.[^'\\]*)*)'|"([^"\\]*(?:\\.[^"\\]*)*)")z\d+u fööz evalz[a-zA-Z_][a-zA-Z0-9_]*F) _identifierz[\w{0}]+Tzjinja2._identifierz(?z>= s rD z(%s)r< c c s | ]}t j|V qd S )N)reescape)r@ xr? r? rC s rH c C s
t | S )N)len)rG r? r? rC s rJ )keyc C sL | t krt | S tdtdtdtdtdtdtdtdt dt
d t d
t d i j
| | S ) Nzbegin of commentzend of commentr$ zbegin of statement blockzend of statement blockzbegin of print statementzend of print statementzbegin of line statementzend of line statementztemplate data / textzend of template)reverse_operatorsTOKEN_COMMENT_BEGINTOKEN_COMMENT_END
TOKEN_COMMENTTOKEN_LINECOMMENTTOKEN_BLOCK_BEGINTOKEN_BLOCK_ENDTOKEN_VARIABLE_BEGINTOKEN_VARIABLE_ENDTOKEN_LINESTATEMENT_BEGINTOKEN_LINESTATEMENT_END
TOKEN_DATA TOKEN_EOFget)
token_typer? r? rC _describe_token_type s r[ c C s | j dkr| jS t| j S )z#Returns a description of the token.r )typevaluer[ )tokenr? r? rC describe_token s
r_ c C s2 d| kr&| j dd\}}|dkr*|S n| }t|S )z0Like `describe_token` but for token expressions.r; r )splitr[ )exprr\ r] r? r? rC describe_token_expr s rc c C s t tj| S )zsCount the number of newline characters in the string. This is
useful for extensions that filter a stream.
)rI
newline_refindall)r] r? r? rC count_newlines s rf c C s t j}t| jd|| jft| jd|| jft| jd|| jfg}| jdk rp|jt| jdd|| j f | jdk r|jt| jdd|| j f d d
t |d d D S )
zACompiles all the rules from the environment into a list of rules.r$ blockZvariableNZ
linestatementz ^[ \t\v]*r' z(?:^|(?<=\S))[^\S\r\n]*c S s g | ]}|d d qS )r` Nr? )r@ rG r? r? rC rD s z!compile_rules..T)reverse)
rE rF rI comment_start_stringblock_start_stringvariable_start_stringline_statement_prefixappendline_comment_prefixsorted) environmenterulesr? r? rC
compile_rules s
rs c @ s$ e Zd ZdZefddZdd ZdS )FailurezjClass that raises a `TemplateSyntaxError` if called.
Used by the `Lexer` to specify known errors.
c C s || _ || _d S )N)message error_class)selfru clsr? r? rC __init__ s zFailure.__init__c C s | j | j||d S )N)rv ru )rw linenofilenamer? r? rC __call__ s zFailure.__call__N)__name__
__module__ __qualname____doc__r ry r| r? r? r? rC rt s rt c @ sT e Zd ZdZf Zdd edD \ZZZdd Z dd Z
d d
Z d d Z d
d Z
dS )Tokenz Token class.c c s | ]}t t|V qd S )N)propertyr )r@ rG r? r? rC rH s zToken. c C s t j| |tt||fS )N)tuple__new__r str)rx rz r\ r] r? r? rC r s z
Token.__new__c C s* | j tkrt| j S | j dkr$| jS | j S )Nr )r\ rL r] )rw r? r? rC __str__ s
z
Token.__str__c C s2 | j |krdS d|kr.|jdd| j | jgkS dS )zTest a token against a token expression. This can either be a
token type or ``'token_type:token_value'``. This can only test
against string values and types.
Tr; r` F)r\ ra r] )rw rb r? r? rC test s
z
Token.testc G s x|D ]}| j |rdS qW dS )z(Test against multiple token expressions.TF)r )rw iterablerb r? r? rC test_any s
zToken.test_anyc C s d| j | j| jf S )NzToken(%r, %r, %r))rz r\ r] )rw r? r? rC __repr__ s zToken.__repr__N)r} r~ r r __slots__rangerz r\ r] r r r r r r? r? r? rC r s
r c @ s( e Zd ZdZdd Zdd Zdd ZdS ) TokenStreamIteratorz`The iterator for tokenstreams. Iterate over the stream
until the eof token is reached.
c C s
|| _ d S )N)stream)rw r r? r? rC ry s z TokenStreamIterator.__init__c C s | S )Nr? )rw r? r? rC __iter__ s z TokenStreamIterator.__iter__c C s0 | j j}|jtkr"| j j t t| j |S )N)r currentr\ rX close
StopIterationnext)rw r^ r? r? rC __next__ s
z TokenStreamIterator.__next__N)r} r~ r r ry r r r? r? r? rC r s r c @ s~ e Zd ZdZdd Zdd Zdd ZeZedd d
d Z d d
Z
dd Z d ddZ dd Z
dd Zdd Zdd Zdd Zd S ) TokenStreamzA token stream is an iterable that yields :class:`Token`\s. The
parser however does not iterate over it but calls :meth:`next` to go
one token ahead. The current active token is stored as :attr:`current`.
c C s> t || _t | _|| _|| _d| _tdtd| _ t
| d S )NFr` ) iter_iterr _pushedr r{ closedr
TOKEN_INITIALr r )rw generatorr r{ r? r? rC ry / s
zTokenStream.__init__c C s t | S )N)r )rw r? r? rC r 8 s zTokenStream.__iter__c C s t | jp| jjtk S )N)boolr r r\ rX )rw r? r? rC __bool__; s zTokenStream.__bool__c C s | S )Nr? )rG r? r? rC rJ ? s zTokenStream.z Are we at the end of the stream?)docc C s | j j| dS )z Push a token back to the stream.N)r rm )rw r^ r? r? rC pushA s zTokenStream.pushc C s" t | }| j}| j| || _|S )zLook at the next token.)r r r )rw Z old_tokenresultr? r? rC lookE s
zTokenStream.lookr` c C s xt |D ] }t| q
W dS )zGot n tokens ahead.N)r r )rw nrG r? r? rC skipM s zTokenStream.skipc C s | j j|rt| S dS )zqPerform the token test and return the token if it matched.
Otherwise the return value is `None`.
N)r r r )rw rb r? r? rC next_ifR s zTokenStream.next_ifc C s | j |dk S )z8Like :meth:`next_if` but only returns `True` or `False`.N)r )rw rb r? r? rC skip_ifY s zTokenStream.skip_ifc C sX | j }| jr| jj | _ n:| j jtk rTyt| j| _ W n tk
rR | j Y nX |S )z|Go one token ahead and return the old one.
Use the built-in :func:`next` instead of calling this directly.
) r r popleftr\ rX r r r r )rw rvr? r? rC r ] s zTokenStream.__next__c C s" t | jjtd| _d| _d| _dS )zClose the stream.r NT)r r rz rX r r )rw r? r? rC r l s zTokenStream.closec C st | j j|s^t|}| j jtkr:td| | j j| j| jtd|t | j f | j j| j| jz| j S t
| X dS )z}Expect a given token type and return it. This accepts the same
argument as :meth:`jinja2.lexer.Token.test`.
z(unexpected end of template, expected %r.zexpected token %r, got %rN) r r rc r\ rX r rz r r{ r_ r )rw rb r? r? rC expectr s zTokenStream.expectN)r` )r} r~ r r ry r r Z __nonzero__r Zeosr r r r r r r r r? r? r? rC r ( s
r c C sZ | j | j| j| j| j| j| j| j| j| j | j
| j f }t j
|}|dkrVt| }|t |< |S )z(Return a lexer which is probably cached.N)rj block_end_stringrk variable_end_stringri comment_end_stringrl rn trim_blocks
lstrip_blocksnewline_sequencekeep_trailing_newline _lexer_cacherY Lexer)rp rK Zlexerr? r? rC get_lexer s"
r c @ s> e Zd ZdZdd Zdd Zd
ddZdd d
Zdd d ZdS )r a
Class that implements a lexer for a given environment. Automatically
created by the environment class, usually you don't have to do that.
Note that the lexer is not automatically bound to an environment.
Multiple environments can share the same lexer.
c s dd }t j}ttd fttd fttd ftt d ft
t d ft t
d fg}t|}|jrTdpVd}i |jr\|d}|d||j }|j|j} || rd|| jd pd7 }|j|j} || rd|| jd pd7 }|d||j }
|
j|j} | r
d || jd p d} d
} d | ||j|||jf }
d | ||j| ||jf }|
d
< | d< nd||j }
|j| _|j| _d|ddjd||j|
||j||jf g fdd|D tdfdf|dtd fgt|d||j ||j |f t t fdf|dt dfd fgt |d ||j||j|f t!dfg| t"|d ||j#||j#f t$dfg| t%|d ||j|
||j||j|f tt&fdf|dt d fd fgt'|d t(dfg| t)|d!t*t+fdfgi| _,d S )"Nc S s t j| t jt jB S )N)rE compileMS)rG r? r? rC rJ s z Lexer.__init__..z\n?r r+ z^%s(.*)z|%sr` z(?!%s)z^[ \t]*z%s%s(?!%s)|%s\+?z %s%s%s|%s\+?rg r$ z%srootz (.*?)(?:%s)r< z4(?P(?:\s*%s\-|%s)\s*raw\s*(?:\-%s\s*|%s))c s& g | ] \}}d || j ||f qS )z(?P<%s_begin>\s*%s\-|%s))rY )r@ r r) prefix_rer? rC rD s z"Lexer.__init__..z#bygroupz.+z(.*?)((?:\-%s\s*|%s)%s)z#popz(.)zMissing end of comment tagz(?:\-%s\s*|%s)%sz
\-%s\s*|%sz1(.*?)((?:\s*%s\-|%s)\s*endraw\s*(?:\-%s\s*|%s%s))z Missing end of raw directivez \s*(\n|$)z(.*?)()(?=\n|$))-rE rF
whitespace_reTOKEN_WHITESPACEfloat_re TOKEN_FLOAT
integer_re
TOKEN_INTEGERname_re
TOKEN_NAME string_re TOKEN_STRING operator_reTOKEN_OPERATORrs r r rj matchri grouprk r r joinr rW rM r rO rN rt rQ rR rS r rT TOKEN_RAW_BEGIN
TOKEN_RAW_ENDrU rV TOKEN_LINECOMMENT_BEGINrP TOKEN_LINECOMMENT_ENDrr )rw rp crq Z tag_rulesZroot_tag_rulesZblock_suffix_reZ no_lstrip_reZ
block_diffmZ comment_diffZno_variable_reZ lstrip_reZblock_prefix_reZcomment_prefix_rer? )r rC ry s
zLexer.__init__c C s t j| j|S )z@Called for strings and template data to normalize it to unicode.)rd r r )rw r] r? r? rC _normalize_newlines$ s zLexer._normalize_newlinesNc C s& | j ||||}t| j|||||S )zCCalls tokeniter + tokenize and wraps it in a token stream.
) tokeniterr wrap)rw sourcer r{ stater r? r? rC tokenize( s zLexer.tokenizec c sj xb|D ]X\}}}|t kr"qn2|dkr2d}n"|dkrBd}n|dkrPqn|dkrd| j|}n|dkrr|}n|d krt|}tr|j rtd
|||n|d kry$| j|d d jd
djd}W nH tk
r } z*t|j dd j
}t||||W Y dd}~X nX n:|dkr.t |}n&|dkrBt |}n|dkrTt
| }t|||V qW dS )zThis is called with the stream as returned by `tokenize` and wraps
every token in a :class:`Token` and converts the value.
r% r r& r r" r# r( keywordr z Invalid character in identifierr r` asciibackslashreplacezunicode-escaper; Nr r r )r" r# r )ignored_tokensr r check_ident isidentifierr encodedecode Exceptionra stripintr operatorsr ) rw r r r{ rz r^ r] rq msgr? r? rC r . sD
"
z
Lexer.wrapc c s@ t |}|j }| jr>|r>x"d D ]}|j|r |jd P q W dj|}d}d}dg} |dk rv|dkrv| j|d nd}| j| d }
t|} g } xx|
D ]j\}
}}|
j||}|dkrq| r|d krqt |t
rxt |D ]\}}|j t
kr|||q|d
kr`xt|j D ]0\}}|dk r |||fV ||jd7 }P q W td|
q|j|d }|s~|tkr|||fV ||jd7 }qW n|j }|dkr<|dkr| jd nv|dkr| jd n`|dkr| jd nJ|d kr<| std| |||| j }||kr s
+ ^