R`i: X d Z ddlZddlZddlZddlmZmZ ddlmZ ddl m
Z
m Z m Z m
Z
mZ ddlmZmZmZmZmZmZ ddlmZ g dZ ej2 d Zg d
Z e d Z G d d
e Z G d de Z G d de Z! G d de" Z# G d d Z$ e$ Z% G d de& Z' G d d Z(d Z) G d d Z* e* Z+d Z, G d d Z- G d! d"e Z. G d# d$e Z/ G d% d&e e/ Z0 G d' d( Z1 G d) d*e0 Z2d+ Z3 G d, d-e/ Z4 G d. d/e0e4 Z5y)0z
pygments.lexer
~~~~~~~~~~~~~~
Base lexer classes.
:copyright: Copyright 2006-2023 by the Pygments team, see AUTHORS.
:license: BSD, see LICENSE for details.
N)
apply_filtersFilter)get_filter_by_name)ErrorTextOther
Whitespace
_TokenType) get_bool_opt get_int_opt get_list_optmake_analysatorFuture guess_decode) regex_opt)
Lexer
RegexLexerExtendedRegexLexerDelegatingLexer LexerContextincludeinheritbygroupsusingthisdefaultwordsline_rez.*?
))s utf-8)s zutf-32)s zutf-32be)s zutf-16)s zutf-16bec y)N )xs /builddir/build/BUILDROOT/alt-python312-pip-23.3.1-3.el8.x86_64/opt/alt/python312/lib/python3.12/site-packages/pip/_vendor/pygments/lexer.pyr% " s # c e Zd ZdZd Zy) LexerMetaz
This metaclass automagically converts ``analyse_text`` methods into
static methods which always return float values.
c \ d|v rt |d |d< t j | ||| S )N analyse_text)r type__new__)mcsnamebasesds r$ r, zLexerMeta.__new__+ s3 Q /.0A BAn ||Cua00r& N)__name__
__module__ __qualname____doc__r, r" r& r$ r( r( % s
1r& r( c R e Zd ZdZdZg Zg Zg Zg ZdZ dZ
d Z d Z d Z
d Zd
dZd Zy) r au
Lexer for a specific language.
See also :doc:`lexerdevelopment`, a high-level guide to writing
lexers.
Lexer classes have attributes used for choosing the most appropriate
lexer based on various criteria.
.. autoattribute:: name
:no-value:
.. autoattribute:: aliases
:no-value:
.. autoattribute:: filenames
:no-value:
.. autoattribute:: alias_filenames
.. autoattribute:: mimetypes
:no-value:
.. autoattribute:: priority
Lexers included in Pygments should have an additional attribute:
.. autoattribute:: url
:no-value:
You can pass options to the constructor. The basic options recognized
by all lexers and processed by the base `Lexer` class are:
``stripnl``
Strip leading and trailing newlines from the input (default: True).
``stripall``
Strip all leading and trailing whitespace from the input
(default: False).
``ensurenl``
Make sure that the input ends with a newline (default: True). This
is required for some lexers that consume input linewise.
.. versionadded:: 1.3
``tabsize``
If given and greater than 0, expand tabs in the input (default: 0).
``encoding``
If given, must be an encoding name. This encoding will be used to
convert the input string to Unicode, if it is not already a Unicode
string (default: ``'guess'``, which uses a simple UTF-8 / Locale /
Latin1 detection. Can also be ``'chardet'`` to use the chardet
library, if it is installed.
``inencoding``
Overrides the ``encoding`` if given.
Nr c l || _ t |dd | _ t |dd | _ t |dd | _ t |dd | _ |j dd | _ |j d
xs | j | _ g | _ t |d d D ] }| j | y
)a
This constructor takes arbitrary options as keyword arguments.
Every subclass must first process its own options and then call
the `Lexer` constructor, since it processes the basic
options like `stripnl`.
An example looks like this:
.. sourcecode:: python
def __init__(self, **options):
self.compress = options.get('compress', '')
Lexer.__init__(self, **options)
As these options must all be specifiable as strings (due to the
command line usage), there are various utility functions
available to help with that, see `Utilities`_.
stripnlTstripallFensurenltabsizer encodingguess
inencodingfiltersr" N) optionsr r7 r8 r9 r r: getr; r> r
add_filter)selfr? filter_s r$ __init__zLexer.__init__ s & #GY= $Wj%@
$Wj$?
"7Iq9 J8
L1BT]]
#GY;G OOG $ z)r? __class__r1 rB s r$ __repr__zLexer.__repr__ s@ <<59^^5L5L59\\C
C *DNN,C,CC Cr& c r t |t s t |fi |}| j j | y)z8
Add a new stream filter to this lexer.
N)
isinstancer r r> append)rB rC r? s r$ rA zLexer.add_filter s/ '6*(.streamer s+ 66t<1ad
=s "%)rK strr; r pip._vendorrP ImportError
_encoding_map
startswithlendecodedetectr@ rQ r8 stripr7 r:
expandtabsr9 endswithr r> ) rB rN
unfilteredrW rP edecodedbomr; encrZ streams `` r$
get_tokenszLexer.get_tokens s $$}}'&t,a )+T3 %2MCs+"&s3xy/"8"89"M &3
?!..et5C "kk#''**=*H*3 5G {{4==1??8, H
/Dx(CMN+ ||FD)||D$' ==::F
U # T% 'L MRSTTs G2 2 H ; HH c t )aS
This method should process the text and return an iterable of
``(index, tokentype, value)`` tuples where ``index`` is the starting
position of the token within the input text.
It must be overridden by subclasses. It is recommended to
implement it as a generator to maximize effectiveness.
)NotImplementedError)rB rN s r$ rV z Lexer.get_tokens_unprocessed s
"!r& )F)r1 r2 r3 r4 r. aliases filenamesalias_filenames mimetypespriorityurlrD rI rA r* rl rV r" r& r$ r r 1 sY 1h D G
I O I H C %<D% ">@ "r& r ) metaclassc e Zd ZdZefdZd Zy)r a
This lexer takes two lexer as arguments. A root lexer and
a language lexer. First everything is scanned using the language
lexer, afterwards all ``Other`` tokens are lexed using the root
lexer.
The lexers from the ``template`` lexer package use this base lexer.
c r |di || _ |di || _ || _ t j | fi | y Nr" )
root_lexerlanguage_lexerneedler rD )rB _root_lexer_language_lexer_needler? s r$ rD zDelegatingLexer.__init__ s9 %00 - 8 8
t'w'r& c l d}g }g }| j j | D ]N \ }}}|| j u r&|r |j t | |f g }||z
};|j |||f P |r |j t | |f t || j j | S )N )rz rV r{ rL r`
do_insertionsry )rB rN buffered
insertions
lng_bufferirX rY s r$ rV z&DelegatingLexer.get_tokens_unprocessed s
**AA$GGAq!DKK %%s8}j&AB!#JA
!!1a), H s8}j 9 : Z !__ C CH MO Or& N)r1 r2 r3 r4 r rD rV r" r& r$ r r
s >C ( Or& r c e Zd ZdZy)r zI
Indicates that a state should include rules from another state.
Nr1 r2 r3 r4 r" r& r$ r r 4 s r& r c e Zd ZdZd Zy)_inheritzC
Indicates the a state should inherit from its superclass.
c y)Nr r" rH s r$ rI z_inherit.__repr__? s r& N)r1 r2 r3 r4 rI r" r& r$ r r ; s r& r c e Zd ZdZd Zd Zy)combinedz:
Indicates a state combined from multiple states.
c . t j | | S rU )tupler, )clsargss r$ r, zcombined.__new__J s }}S$''r& c y rU r" )rB r s r$ rD zcombined.__init__M s r& N)r1 r2 r3 r4 r, rD r" r& r$ r r E s (
r& r c : e Zd ZdZd Zd dZd dZd dZd Zd Z y)
_PseudoMatchz:
A pseudo match object constructed from a string.
c || _ || _ y rU )_text_start)rB startrN s r$ rD z_PseudoMatch.__init__W s
r& Nc | j S rU )r rB args r$ r z_PseudoMatch.start[ s {{r& c F | j t | j z S rU )r r` r r s r$ endz_PseudoMatch.end^ s {{S_,,r& c 4 |r t d | j S )Nz
No such group)
IndexErrorr r s r$ groupz_PseudoMatch.groupa s _- -zzr& c | j fS rU )r rH s r$ groupsz_PseudoMatch.groupsf s
} r& c i S rU r" rH s r$ groupdictz_PseudoMatch.groupdicti s r& rU )
r1 r2 r3 r4 rD r r r r r r" r& r$ r r R s% -
r& r c d fd }|S )zL
Callback that yields multiple actions for each group in the match.
c
3 K t D ] \ }}| t | t u r1|j |dz }|s1|j |dz ||f K|j |dz }|b|r|j |dz |_ || t
|j |dz | | D ] }|s| |r|j |_ y y w)N ) enumerater+ r
r r posr r )lexermatchctxr actiondataitemr s r$ callbackzbygroups..callbackq s "4 IAv~f + {{1q5) ++a!e,fd:: {{1q5)#"'++a!e"4 &u'3EKKA4F'Ms!T "&J!T ) iikCG s .callback s
emm,$U__.v.
A4244U[[]PiP1a!eQk! Q ))+ s BBc 3 K
j | j di
}|j } |j |j fi D ] \ }}}||z ||f |r|j |_ y y wrx )r r? r rV r r r ) r r r r r r rX rY _otherr r s r$ r zusing..callback s} MM%-- (!&!B
A4244U[[]PiP1a!eQk! Q ))+ s BBrU )poprK listr r )r r r r r s `` @r$ r r se I& JJw a$ '!"Ig "(!Ig
~
&2 O & Or& c e Zd ZdZd Zy)r z
Indicates a state or state action (e.g. #pop) to apply.
For example default('#pop') is equivalent to ('', Token, '#pop')
Note that state tuples may be used as well.
.. versionadded:: 2.0
c || _ y rU )r )rB r s r$ rD zdefault.__init__ s
r& N)r1 r2 r3 r4 rD r" r& r$ r r s r& r c e Zd ZdZddZd Zy)r z
Indicates a list of literal words that is transformed into an optimized
regex that matches any of the words.
.. versionadded:: 2.0
c . || _ || _ || _ y rU )r prefixsuffix)rB r r r s r$ rD zwords.__init__ s
r& c Z t | j | j | j S )Nr r )r r r r rH s r$ r@ z words.get s DKK LLr& N)r r )r1 r2 r3 r4 rD r@ r" r& r$ r r s
Mr& r c < e Zd ZdZd Zd Zd Zd Zd
dZd Z d Z
y) RegexLexerMetazw
Metaclass for RegexLexer, creates the self._tokens attribute from
self.tokens on the first instantiation.
c t |t r|j }t j || j
S )zBPreprocess the regular expression component of a token definition.)rK r r@ recompiler )r regexrflagsr s r$ _process_regexz RegexLexerMeta._process_regex s. eV $IIKEzz%(...r& c R t | t u st | s
J d| |S )z5Preprocess the token component of a token definition.z0token type must be simple type or callable, not )r+ r
callable)r tokens r$ _process_tokenz RegexLexerMeta._process_token s. E{j(HUO LDI K L; r& c t |t r5|dk( ry||v r|fS |dk( r|S |dd dk( rt |dd S J d|z t |t rfd| j z }| xj d z
c_ g }|D ]3 }||k7 s
J d
|z |j | j
||| 5 |||< |fS t |t r |D ] }||v r|d v r
J d |z |S J d
|z )z=Preprocess the state transition action of a token definition.#pop#pushN z#pop:zunknown new state %rz_tmp_%dr zcircular state ref %r)r r zunknown new state zunknown new state def %r)rK r[ intr _tmpnameextend_process_stater )r new_state unprocessed processed tmp_stateitokensistates r$ _process_new_statez!RegexLexerMeta._process_new_state s? i %F"k)!|#g% 2A ')IabM***@ 4y @@u
8
,!CLL0I LLA LG# *L,Cf,LL*s 1 1+2;V E F $ $+Ii <
5
)# +- "332(612 4 $ @4y@ @5r& c t | t u s
J d|z |d dk7 s
J d|z ||v r|| S g x}||< | j }|| D ]? }t |t r;||k7 s
J d|z |j | j
||t | Ot |t r`t |t rO| j |j || }|j t j d j d|f t | t u s
J d|z | j! |d || }| j' |d
}
t) | dk( rd}n| j |d || }|j ||
|f B |S # t" $ r } t% d |d d
|d | d | | d} ~ ww xY w)z%Preprocess a single state definition.zwrong state name %rr #zinvalid state name %rzcircular state reference %rr Nzwrong rule def %rzuncompilable regex z
in state z of z: r )r+ r[ flagsrK r r r r r r r rL r r r r r Exception
ValueErrorr r` ) r r r r tokensr tdefr rexerrr s r$ r z RegexLexerMeta._process_state s E{c!@#85#@@!Qx3? 7% ?? I U# #$&&5! &D$(u}K&Ce&KK}
c 0 0i14T < =$) $( 2 24::{I V
rzz"~ 3 3T9 EF:& B(;d(B B&
F((a&%@
&&tAw/E4yA~ 2 2473> K
MM3y1 2C 'D
F "&q'5#s"< =BEF
Fs *F F= F88F=Nc i x}| j |< |xs | j | }t | D ] }| j ||| |S )z-Preprocess a dictionary of token definitions.) _all_tokensr r r )r r. tokendefsr r s r$ process_tokendefz RegexLexerMeta.process_tokendef? sN ,.. COOD) 1D!1 )_E y)U ; %r& c i }i }| j D ] }|j j di }|j D ]t \ }}|j | }|!|||< |j t
}|||< :|j |d }|O||||dz |j t
} || z ||< v |S # t $ r Y w xY w# t $ r Y w xY w)a
Merge tokens from superclasses in MRO order, returning a single tokendef
dictionary.
Any state that is not defined by a subclass will be inherited
automatically. States that *are* defined by subclasses will, by
default, override that state in the superclass. If a subclass wishes to
inherit definitions from a superclass, it can use the special value
"inherit", which will cause the superclass' state definition to be
included at that point in the state.
r Nr )__mro____dict__r@ itemsindexr r r )
r r inheritablectoksr r curitems inherit_ndx new_inh_ndxs
r$
get_tokendefsz RegexLexerMeta.get_tokendefsG s A::>>(B/D $
u!::e,#
%*F5M!&+kk'&: *5K& )ooeT : & 7<[]3C #(++g"6K *5{)BK&9 !- B
) & ! ! " s$ B;C
; CC
CCc d| j vrLi | _ d| _ t | d r
| j rn%| j d| j
| _ t j | g|i |S )z:Instantiate cls after preprocessing its token definitions._tokensr token_variantsr )
r r r hasattrr r r r r+ __call__)r r kwdss r$ r zRegexLexerMeta.__call__x sh CLL ( CO CLs ,-#2D2D ! 2 22s7H7H7J K }}S040400r& rU ) r1 r2 r3 r4 r r r r r r r r" r& r$ r r s.
/ !AF*X/b 1r& r c 4 e Zd ZdZej
Zi ZddZy)r z
Base for simple stateful regular expression-based lexers.
Simplifies the lexing process so that you need only
provide a list of states and regular expressions.
c # : K d}| j }t | }||d } |D ]& \ }}} ||| }
|
s|8t | t u r|||
j f n || |
E d{ |
j }| t
| t rX| D ]R } | dk( r t | dkD s|j (| dk( r|j |d B|j | T nWt
| t r#t | t | k\ r|dd= n*|| d= n$| dk( r|j |d n
J d| z ||d } n7 || dk( rd g}|d }|t df |dz
}P|t || f |dz
}d7 # t $ r Y yw xY ww)
z~
Split ``text`` into (tokentype, text) pairs.
``stack`` is the initial stack (default: ``['root']``)
r r r Nr r wrong state def: %rrR r )r r r+ r
r r rK r r` r rL r absr r r ) rB rN r r r
statestack statetokensrexmatchr r mr s r$ rV z!RegexLexer.get_tokens_unprocessed s LL %[
2/ /:+&) T3' ) <:5"%vqwwy"8 8'-dA 6 6%%'C ,%i7)2#(F?'*:':(2(8%*g%5$.$5$5jn$E$.$5$5e$< *3 ( 3 7 #9~Z @$.qrN$.yz$: &' 1 & - -jn = K*?)*K K5&/
2&? C 0;J CyD(&,X
&/&7 !:t 33q ud3i//1HC_ 7P " sM 8F5F0F 1>F0B!F F 2F4F F FFFFN)r ) r1 r2 r3 r4 r MULTILINEr r rV r" r& r$ r r s
LLE0 F;r& r c e Zd ZdZddZd Zy)r z9
A helper object that holds lexer position data.
Nc ` || _ || _ |xs t | | _ |xs dg| _ y )Nr )rN r r` r r )rB rN r r r s r$ rD zLexerContext.__init__ s. ##d)&vh
r& c V d| j d| j d| j dS )Nz
LexerContext(z, ))rN r r rH s r$ rI zLexerContext.__repr__ s IItxx- -r& NN)r1 r2 r3 r4 rD rI r" r& r$ r r s ' -r& r c e Zd ZdZddZy)r zE
A RegexLexer that uses a context object to store its state.
Nc # K | j }|st |d }|d }n |}||j d }|j } |D ] \ }}} |||j |j
} | s)|lt
| t u r5|j || j f | j |_ n& || | | E d{ |s||j d }|5t |t r|D ] }
|
dk( r4t |j dkD s!|j j <|
dk( r)|j j |j d j|j j |
nt |t rAt | t |j k\ r|j dd= nH|j |d= n8|dk( r)|j j |j d n
J d|z ||j d } n |j |j
k\ ry||j d k( r9dg|_ |d }|j t d f |xj dz
c_ ;|j t" ||j f |xj dz
c_ s7 # t$ $ r Y yw xY ww)
z
Split ``text`` into (tokentype, text) pairs.
If ``context`` is given, use this lexer context instead.
r r r r Nr r r rR )r r r rN r r r+ r
r rK r r` r rL r r r r r ) rB rN contextr r r r
r r r r s r$ rV z)ExtendedRegexLexer.get_tokens_unprocessed sk
LL tQ'C#F+KC#CIIbM2K88D/:+&) T377CGG4 ) <:5"%''61779"< <&'eegCG'-dAs'; ; ;#,.7 "
.F ,%i7)2#(F?'*399~'9(+
%*g%5$'II$4$4SYYr]$C$'II$4$4U$; *3 ( 3 7 "9~SYY ?$'IIabM$'IIij$9 &' 1 II , ,SYYr] ; K*?)*K K5&/ "
&> G 0;J
ww#'') CGG},%+H &/&7 !ggtT 111 ''5$sww-77GGqLGc 7J: 5K : KK KK r )r1 r2 r3 r4 rV r" r& r$ r r s
@r& r c # K t | } t | \ }}d}d}|D ] \ }}}||}d} |rx|t | z |k\ rg|| ||z
}
|
r|||
f |t |
z
}|D ] \ } } }
|| |
f |t |
z
} ||z
} t | \ }}|r|t | z |k\ rg| t | k s|||| d f |t | | z
z
} |r9|xs d}|D ] \ }}}|||f |t | z
} t | \ }}|r8yy# t $ r |E d{ 7 Y yw xY w# t $ r d}Y w xY w# t $ r d}Y yw xY ww)ag
Helper for lexers which must combine the results of several
sublexers.
``insertions`` is a list of ``(index, itokens)`` pairs.
Each ``itokens`` iterable should be inserted at position
``index`` into the token stream given by the ``tokens``
argument.
The result is a combined token stream.
TODO: clean up the code here.
NTr F)iternext
StopIterationr` )r r r r realposinsleftr rX rY olditmpvalit_indexit_tokenit_valueps r$ r r ? s j!J j)w GG 1a ?G!c!f*-tEAI&F q&((3v;&07,(H x113x=( 18 19D
!%j!1w !c!f*- #a&=1ah& & s1v} $G+ 0 ,Q GAq!1a- s1v G !* -NE7 E 4 !
G s ED A*ED,E*E9AE?D=
EED) D#!D)&E(D))E, D:7E9D::E= E E
E Ec e Zd ZdZd Zy)ProfilingRegexLexerMetaz>Metaclass for ProfilingRegexLexer, collects regex timing info.c t |t r-t |j |j |j n|t j | t j f fd }|S )Nr c j d j
fddg }t j }j | || }t j }|dxx dz
cc< |dxx ||z
z
cc< |S )Nr r r! r )
_prof_data
setdefaulttimer ) rN r endposinfot0rest1r compiledr r s r$
match_funcz:ProfilingRegexLexerMeta._process_regex..match_func sr >>"%00% 3xHDB..sF3CB GqLG GrBw GJr& ) rK r r r r r r sysmaxsize)r r r r r/ r. r s ` ` @@r$ r z&ProfilingRegexLexerMeta._process_regex sZ eU #EKK #(<<1C C::c6*), r& N)r1 r2 r3 r4 r r" r& r$ r# r# s
Hr& r# c e Zd ZdZg ZdZddZy)ProfilingRegexLexerzFDrop-in replacement for RegexLexer that does profiling of its regexes. c # J K j j j i t j || E d { j j j }t
d |j D fdd }t d |D }t t d j j t | |fz t d t dd z t d
|D ] }t d |z t d y 7 ݭw) Nc 3 K | ]H \ \ }}\ }}|t | j d j dd dd |d|z d|z |z f J yw)zu'z\\\NA i )reprrc rQ ).0r rnrX s r$ z=ProfilingRegexLexer.get_tokens_unprocessed.. sa @/>+FQFQ 47==/77EcrJ4!8TAX\3/>s AAc " | j S rU )_prof_sort_index)r# rB s r$ r% z. s Ad&;&;$. s +d!ds z2Profiling result for %s lexing %d chars in %.3f mszn==============================================================================================================z$%-20s %-64s ncalls tottime percall)r r zn--------------------------------------------------------------------------------------------------------------z%-20s %-65s %5d %8.4f %8.4f) rG r& rL r rV r sortedr sumprintr1 r` )rB rN r rawdatar sum_totalr0 s ` r$ rV z*ProfilingRegexLexer.get_tokens_unprocessed s !!((, 44T4GGG..++//1 @/6}}@ = " $
+d++
B~~&&D 9=> ?
i
47IIJ
iA /!3 4
i# Hs A D#D!C D#Nr )r1 r2 r3 r4 r& r? rV r" r& r$ r3 r3 s PJr& r3 )6r4 r r0 r( pip._vendor.pygments.filterr r pip._vendor.pygments.filtersr pip._vendor.pygments.tokenr r r r r
pip._vendor.pygments.utilr r r
r r r pip._vendor.pygments.regexoptr __all__r r r^ staticmethod_default_analyser+ r( r r r[ r r r r r r r r r r r r r r r r r r# r3 r" r& r$ rQ sJ
= ; Q Q* * 3 * "**W
,
. 1 1Y"i Y"x Oe ON c *
u
64
w/d
MF
M e1Y e1P^. ^B
-
- E EP=@n ,*0G r&