U
ʗRe} @ s d Z ddlZddlZddlZddlmZmZ ddlmZ ddl m
Z
m Z m Z m
Z
ddlmZmZmZmZmZmZ ddlmZ dd d
d d d
ddddddg ZdddddgZedd ZG dd d eZ G d d de d Z G d d d e Z G d d
d
e Z G d!d" d"Z!e! Z"G d#d$ d$e#Z$G d%d& d&Z%d'd Z&G d(d) d)Z'e' Z(d*d Z)G d+d dZ*G d,d deZ+G d-d. d.e Z,G d/d d e e,d Z-G d0d d Z.G d1d
d
e-Z/d2d3 Z0G d4d5 d5e,Z1G d6d7 d7e-e1d Z2dS )8z
pygments.lexer
~~~~~~~~~~~~~~
Base lexer classes.
:copyright: Copyright 2006-2022 by the Pygments team, see AUTHORS.
:license: BSD, see LICENSE for details.
N)
apply_filtersFilter)get_filter_by_name)ErrorTextOther
_TokenType) get_bool_opt get_int_opt get_list_optmake_analysatorFuture guess_decode) regex_optLexer
RegexLexerExtendedRegexLexerDelegatingLexer LexerContextincludeinheritbygroupsusingthisdefaultwords)s utf-8)s zutf-32)s zutf-32be)s zutf-16)s zutf-16bec C s dS )N xr r /builddir/build/BUILDROOT/alt-python38-pip-22.2.1-2.el8.x86_64/opt/alt/python38/lib/python3.8/site-packages/pip/_vendor/pygments/lexer.py! r" c @ s e Zd ZdZdd ZdS ) LexerMetaz
This metaclass automagically converts ``analyse_text`` methods into
static methods which always return float values.
c C s( d|krt |d |d< t| |||S )N analyse_text)r type__new__)mcsnamebasesdr r r! r' * s zLexerMeta.__new__N)__name__
__module__ __qualname____doc__r' r r r r! r$ $ s r$ c @ s^ e Zd ZdZdZdZg Zg Zg Zg Z dZ
dd Z dd Z dd Z
d
d Zdd
dZdd ZdS )r a
Lexer for a specific language.
Basic options recognized:
``stripnl``
Strip leading and trailing newlines from the input (default: True).
``stripall``
Strip all leading and trailing whitespace from the input
(default: False).
``ensurenl``
Make sure that the input ends with a newline (default: True). This
is required for some lexers that consume input linewise.
.. versionadded:: 1.3
``tabsize``
If given and greater than 0, expand tabs in the input (default: 0).
``encoding``
If given, must be an encoding name. This encoding will be used to
convert the input string to Unicode, if it is not already a Unicode
string (default: ``'guess'``, which uses a simple UTF-8 / Locale /
Latin1 detection. Can also be ``'chardet'`` to use the chardet
library, if it is installed.
``inencoding``
Overrides the ``encoding`` if given.
Nr c K s || _ t|dd| _t|dd| _t|dd| _t|dd| _|dd | _|d
pZ| j| _g | _ t
|d d D ]}| | qpd S )
NstripnlTstripallFensurenltabsizer encodingguess
inencodingfiltersr ) optionsr r0 r1 r2 r
r3 getr4 r7 r
add_filter)selfr8 filter_r r r! __init__a s zLexer.__init__c C s( | j rd| jj| j f S d| jj S d S )Nz z)r8 __class__r, r; r r r! __repr__m s
zLexer.__repr__c K s& t |tst|f|}| j| dS )z8
Add a new stream filter to this lexer.
N)
isinstancer r r7 append)r; r< r8 r r r! r: t s
zLexer.add_filterc C s dS )a~
Has to return a float between ``0`` and ``1`` that indicates
if a lexer wants to highlight this text. Used by ``guess_lexer``.
If this method returns ``0`` it won't highlight it in any case, if
it returns ``1`` highlighting with this lexer is guaranteed.
The `LexerMeta` metaclass automatically wraps this function so
that it works like a static method (no ``self`` or ``cls``
parameter) and the return value is automatically converted to
`float`. If the return value is an object that is boolean `False`
it's the same as if the return values was ``0.0``.
Nr )textr r r! r% | s zLexer.analyse_textFc
s t ts jdkr$t\}nވ jdkrzddlm} W n, tk
rj } ztd|W 5 d}~X Y nX d}tD ].\}}|rtt |d
|d} qqt|dkr| dd }
| d pd
d}|n(
jd r t d d n d r t d d
d d
dd
jrJ n jr\d
jdkrt j jrd
sd
7 fdd}
|
} |st| j } | S )a=
Return an iterable of (tokentype, value) pairs generated from
`text`. If `unfiltered` is set to `True`, the filtering mechanism
is bypassed even if filters are defined.
Also preprocess the text, i.e. expand tabs and strip it if
wanted and applies registered filters.
r5 chardetr )rD zkTo enable chardet encoding guessing, please install the chardet library from http://chardet.feedparser.org/Nreplacei r4 r u z
c 3 s$ D ]\} }}||fV q
d S N)get_tokens_unprocessed)_tvr; rC r r! streamer s z"Lexer.get_tokens..streamer)rA strr4 r pip._vendorrD ImportError
_encoding_map
startswithlendecodedetectr9 rE r1 stripr0 r3
expandtabsr2 endswithr r7 ) r; rC
unfilteredrJ rD edecodedbomr4 encrN streamr rM r!
get_tokens sR
zLexer.get_tokensc C s t dS )z
Return an iterable of (index, tokentype, value) pairs where "index"
is the starting position of the token within the input text.
In subclasses, implement this method as a generator to
maximize effectiveness.
N)NotImplementedErrorrM r r r! rI s z Lexer.get_tokens_unprocessed)F)r, r- r. r/ r) urlaliases filenamesalias_filenames mimetypespriorityr= r@ r: r% r` rI r r r r! r 0 s
;) metaclassc @ s$ e Zd ZdZefddZdd ZdS )r a
This lexer takes two lexer as arguments. A root lexer and
a language lexer. First everything is scanned using the language
lexer, afterwards all ``Other`` tokens are lexed using the root
lexer.
The lexers from the ``template`` lexer package use this base lexer.
c K s0 |f || _ |f || _|| _tj| f| d S rH )
root_lexerlanguage_lexerneedler r= )r; _root_lexer_language_lexer_needler8 r r r! r= s zDelegatingLexer.__init__c C s d}g }g }| j |D ]H\}}}|| jkrP|rF|t||f g }||7 }q||||f q|rx|t||f t|| j|S )N )rj rI rk rB rT
do_insertionsri )r; rC buffered
insertions
lng_bufferirK rL r r r! rI s
z&DelegatingLexer.get_tokens_unprocessedN)r, r- r. r/ r r= rI r r r r! r s c @ s e Zd ZdZdS )r zI
Indicates that a state should include rules from another state.
Nr, r- r. r/ r r r r! r s c @ s e Zd ZdZdd ZdS )_inheritzC
Indicates the a state should inherit from its superclass.
c C s dS )Nr r r? r r r! r@ s z_inherit.__repr__N)r, r- r. r/ r@ r r r r! rv s rv c @ s e Zd ZdZdd Zdd ZdS )combinedz:
Indicates a state combined from multiple states.
c G s t | |S rH )tupler' )clsargsr r r! r'
s zcombined.__new__c G s d S rH r )r; rz r r r! r= s zcombined.__init__N)r, r- r. r/ r' r= r r r r! rw s rw c @ sF e Zd ZdZdd ZdddZdddZdd d
Zd d Zd
d Z dS ) _PseudoMatchz:
A pseudo match object constructed from a string.
c C s || _ || _d S rH )_text_start)r; startrC r r r! r= s z_PseudoMatch.__init__Nc C s | j S rH )r} r; argr r r! r~ s z_PseudoMatch.startc C s | j t| j S rH )r} rT r| r r r r! end! s z_PseudoMatch.endc C s |r t d| jS )Nz
No such group)
IndexErrorr| r r r r! group$ s z_PseudoMatch.groupc C s | j fS rH )r| r? r r r! groups) s z_PseudoMatch.groupsc C s i S rH r r? r r r! groupdict, s z_PseudoMatch.groupdict)N)N)N)
r, r- r. r/ r= r~ r r r r r r r r! r{ s
r{ c s d fdd }|S )zL
Callback that yields multiple actions for each group in the match.
Nc 3 s t D ]\}}|d kr qqt|tkrR||d }|r||d ||fV q||d }|d k r|r|||d |_|| t||d ||D ]}|r|V qq|r| |_d S )N ) enumerater& r r r~ posr{ r )lexermatchctxrt actiondataitemrz r r! callback4 s&
zbygroups..callback)Nr )rz r r r r! r 0 s c @ s e Zd ZdZdS )_ThiszX
Special singleton used for indicating the caller class.
Used by ``using``.
Nru r r r r! r J s r c sj i dkr: d}t|ttfr.|d< n d|fd< tkrTdfdd }nd fdd }|S )
a
Callback that processes the match with a different lexer.
The keyword arguments are forwarded to the lexer, except `state` which
is handled separately.
`state` specifies the state that the new lexer will start in, and can
be an enumerable such as ('root', 'inline', 'string') or a simple
string which is assumed to be on top of the root state.
Note: For that to work, `_other` must not be an `ExtendedRegexLexer`.
statestackrootNc 3 sj r | j | jf }n| }| }|j| f D ]\}}}|| ||fV q<|rf| |_d S rH )updater8 r> r~ rI r r r r r r lxsrt rK rL ) gt_kwargskwargsr r! r i s zusing..callbackc 3 s^ | j f }| }|j| fD ]\}}}|| ||fV q0|rZ| |_d S rH )r r8 r~ rI r r r r _otherr r r r! r x s
)N)N)poprA listrx r )r r r r r r r! r S s
c @ s e Zd ZdZdd ZdS )r z
Indicates a state or state action (e.g. #pop) to apply.
For example default('#pop') is equivalent to ('', Token, '#pop')
Note that state tuples may be used as well.
.. versionadded:: 2.0
c C s
|| _ d S rH )r )r; r r r r! r= s zdefault.__init__N)r, r- r. r/ r= r r r r! r s c @ s" e Zd ZdZdddZdd ZdS ) r z
Indicates a list of literal words that is transformed into an optimized
regex that matches any of the words.
.. versionadded:: 2.0
ro c C s || _ || _|| _d S rH )r prefixsuffix)r; r r r r r r! r= s zwords.__init__c C s t | j| j| jdS )Nr r )r r r r r? r r r! r9 s z words.getN)ro ro )r, r- r. r/ r= r9 r r r r! r s
c @ sJ e Zd ZdZdd Zdd Zdd Zdd Zdd d Zd
d Z dd Z
d
S )RegexLexerMetazw
Metaclass for RegexLexer, creates the self._tokens attribute from
self.tokens on the first instantiation.
c C s t |tr| }t||jS )zBPreprocess the regular expression component of a token definition.)rA r
r9 recompiler )ry regexrflagsr r r r! _process_regex s
z RegexLexerMeta._process_regexc C s& t |tks"t|s"td|f |S )z5Preprocess the token component of a token definition.z2token type must be simple type or callable, not %r)r& r callableAssertionError)ry tokenr r r! _process_token s z RegexLexerMeta._process_tokenc C s t |trd|dkrdS ||kr$|fS |dkr0|S |dd dkrRt|dd S dsbtd| nt |trd | j }| jd
7 _g }|D ],}||kstd | || ||| q|||< |fS t |tr |D ] }||ks|d kstd
| q|S dstd| dS )z=Preprocess the state transition action of a token definition.#pop#pushN z#pop:Fzunknown new state %rz_tmp_%dr zcircular state ref %r)r r zunknown new state zunknown new state def %r) rA rO intr rw _tmpnameextend_process_staterx )ry new_state unprocessed processed tmp_stateitokensistater r r! _process_new_state s>
z!RegexLexerMeta._process_new_statec C s t |tkstd| |d dks0td| ||kr@|| S g }||< | j}|| D ].}t|tr||ks~td| || ||t| qZt|trqZt|t r|
|j ||}| t
djd|f qZt |tkstd| z| |d ||}W n> tk
rD } z td |d || | f | W 5 d} ~ X Y nX | |d
}
t|d krhd}n|
|d ||}| ||
|f qZ|S ) z%Preprocess a single state definition.zwrong state name %rr #zinvalid state name %rzcircular state reference %rro Nzwrong rule def %rz+uncompilable regex %r in state %r of %r: %sr )r& rO r flagsrA r r r rv r r r rB r r r rx r Exception
ValueErrorr rT ) ry r r r tokensr tdefr rexerrr r r r! r sJ
z RegexLexerMeta._process_stateNc C s< i }| j |< |p| j| }t|D ]}| ||| q$|S )z-Preprocess a dictionary of token definitions.) _all_tokensr r r )ry r) tokendefsr r r r r! process_tokendef s
z RegexLexerMeta.process_tokendefc
C s i }i }| j D ]}|jdi }| D ]\}}||}|dkr||||< z|t}W n tk
rp Y q(Y nX |||< q(||d}|dkrq(||||d < z|t} W n tk
r Y q(X || ||< q(q|S )a
Merge tokens from superclasses in MRO order, returning a single tokendef
dictionary.
Any state that is not defined by a subclass will be inherited
automatically. States that *are* defined by subclasses will, by
default, override that state in the superclass. If a subclass wishes to
inherit definitions from a superclass, it can use the special value
"inherit", which will cause the superclass' state definition to be
included at that point in the state.
r Nr )__mro____dict__r9 itemsindexr r r )
ry r inheritablectoksr r curitems inherit_ndx new_inh_ndxr r r!
get_tokendefs
s0
z RegexLexerMeta.get_tokendefsc O sL d| j kr:i | _d| _t| dr(| jr(n| d| | _tj | f||S )z:Instantiate cls after preprocessing its token definitions._tokensr token_variantsro )
r r r hasattrr r r r r& __call__)ry rz kwdsr r r! r ; s
zRegexLexerMeta.__call__)N) r, r- r. r/ r r r r r r r r r r r! r s #,
1r c @ s$ e Zd ZdZejZi ZdddZdS )r z
Base for simple stateful regular expression-based lexers.
Simplifies the lexing process so that you need only
provide a list of states and regular expressions.
r c c s d}| j }t|}||d }|D ](\}}} |||}
|
r"|dk rrt|tkrb|||
fV n|| |
E dH |
}| dk rHt| tr| D ]D} | dkrt|dkr| q| dkr|
|d q|
| qnbt| t rt | t|kr|dd= n
|| d= n,| dkr*|
|d ndsMetaclass for ProfilingRegexLexer, collects regex timing info.c sL t |tr t|j|j|jdn|t|tjf fdd }|S )Nr c s` j d fddg}t }| ||}t }|d d7 < |d || 7 < |S )Nr r r r )
_prof_data
setdefaulttimer )rC r endposinfot0rest1ry compiledr r r r!
match_funcM s z:ProfilingRegexLexerMeta._process_regex..match_func) rA r r r r r r sysmaxsize)ry r r r r r r r! r E s
z&ProfilingRegexLexerMeta._process_regexN)r, r- r. r/ r r r r r! r B s r c @ s" e Zd ZdZg ZdZdddZdS )ProfilingRegexLexerzFDrop-in replacement for RegexLexer that does profiling of its regexes. r c # s j ji t ||E d H j j }tdd | D fdddd}tdd |D }t t d j j
t ||f t d t d
d t d |D ]}t d
| qt d d S )Nc s sN | ]F\\}}\}}|t |d dddd |d| d| | fV qdS )zu'z\\\NA i )reprrW rE ).0r rnrK r r r! c s
z=ProfilingRegexLexer.get_tokens_unprocessed..c s
| j S rH )_prof_sort_indexr r? r r! r" f r# z.T)keyreversec s s | ]}|d V qdS ) Nr )r r r r r! r h s z2Profiling result for %s lexing %d chars in %.3f mszn==============================================================================================================z$%-20s %-64s ncalls tottime percall)r r zn--------------------------------------------------------------------------------------------------------------z%-20s %-65s %5d %8.4f %8.4f) r> r rB r rI r sortedr sumprintr, rT )r; rC r rawdatar sum_totalr+ r r? r! rI ^ s(
z*ProfilingRegexLexer.get_tokens_unprocessedN)r )r, r- r. r/ r r rI r r r r! r X s r )3r/ r r r pip._vendor.pygments.filterr r pip._vendor.pygments.filtersr pip._vendor.pygments.tokenr r r r pip._vendor.pygments.utilr r
r r r
r pip._vendor.pygments.regexoptr __all__rR staticmethod_default_analyser& r$ r r rO r rv r rx rw r{ r r r r r r r r r r rp r r r r r r! s`
!'
2 )aH@