R`i : d Z ddlZddlZddlZddlmZmZmZ ddlm Z ddl
m Z ddl m
Z
mZmZ ddlmZ dd lmZ dd
lmZ dd lmZ dd lmZ dd
lmZ ddl m Z G d d Z y)a
Module containing the UniversalDetector detector class, which is the primary
class a user of ``chardet`` should use.
:author: Mark Pilgrim (initial port to Python)
:author: Shy Shalom (original C code)
:author: Dan Blanchard (major refactoring for 3.0)
:author: Ian Cordasco
N)ListOptionalUnion )CharSetGroupProber)
CharSetProber)
InputStateLanguageFilter ProbingState)EscCharSetProber) Latin1Prober)MacRomanProber)MBCSGroupProber)
ResultDict)SBCSGroupProber)
UTF1632Proberc N e Zd ZdZdZ ej d Z ej d Z ej d Z dddd d
d d d
dZ
dddd ddddZ e j dfde deddfdZedefd Zedefd Zedee fd Zd!d Zd eeef ddfd Zde fd Z y)"UniversalDetectoraq
The ``UniversalDetector`` class underlies the ``chardet.detect`` function
and coordinates all of the different charset probers.
To get a ``dict`` containing an encoding and its confidence, you can simply
run:
.. code::
u = UniversalDetector()
u.feed(some_bytes)
u.close()
detected = u.result
g?s [-]s (|~{)s [-]z Windows-1252z Windows-1250z Windows-1251z Windows-1256z Windows-1253z Windows-1255z Windows-1254z Windows-1257)
iso-8859-1z
iso-8859-2z
iso-8859-5z
iso-8859-6z
iso-8859-7z
iso-8859-8
iso-8859-9z iso-8859-13z ISO-8859-11GB18030CP949UTF-16)asciir ztis-620r gb2312zeuc-krzutf-16leF lang_filtershould_rename_legacyreturnNc d | _ d | _ g | _ d dd d| _ d| _ d| _ t j | _ d| _ || _
t j t | _ d| _ || _ | j# y )N encoding
confidencelanguageF )_esc_charset_prober_utf1632_prober_charset_probersresultdone _got_datar
PURE_ASCII _input_state
_last_charr logging getLogger__name__logger_has_win_bytesr reset)selfr r s /builddir/build/BUILDROOT/alt-python312-pip-23.3.1-3.el8.x86_64/opt/alt/python312/lib/python3.12/site-packages/pip/_vendor/chardet/universaldetector.py__init__zUniversalDetector.__init__d s
@D 8< 57 #
& 1 1 & ''1 #$8!
r% c | j S N)r- r5 s r6 input_statez UniversalDetector.input_state{ s r% c | j S r9 )r3 r: s r6
has_win_bytesz UniversalDetector.has_win_bytes s """r% c | j S r9 )r( r: s r6 charset_probersz!UniversalDetector.charset_probers s $$$r% c V dddd| _ d| _ d| _ d| _ t j
| _ d| _ | j r| j j | j r| j j | j D ] }|j y)z
Reset the UniversalDetector and all of its probers back to their
initial states. This is called by ``__init__``, so you only need to
call this directly in between analyses of different documents.
Nr r! Fr% ) r) r* r+ r3 r r, r- r. r& r4 r' r( )r5 probers r6 r4 zUniversalDetector.reset s $(sM # & 1 1 # # $ $ * * , & & (++F LLN ,r% byte_strc V | j ry|syt |t s t | }| j s|j t
j r dddd| _ n|j t
j t
j f r dddd| _ nt|j d r dddd| _ nW|j d r d
ddd| _ n:|j t
j t
j f r d ddd| _ d | _ | j d
d | _ y| j t j k( r| j j! | rt j" | _ nZ| j t j k( r=| j$ j! | j& |z rt j( | _ |dd | _ | j* st- | _ | j* j. t0 j2 k( rk| j* j5 | t0 j6 k( r?| j* j8 | j* j; dd| _ d | _ y| j t j( k( r| j< st? | j@ | _ | j< j5 | t0 j6 k( rS| j< j8 | j< j; | j< jB d| _ d | _ yy| j t j" k( r:| jD stG | j@ g| _" | j@ tH jJ z r#| jD jM tO | jD jM tQ | jD jM tS | jD D ]Z }|j5 | t0 j6 k( s&|j8 |j; |jB d| _ d | _ n | jT j! | rd | _+ yyy)a
Takes a chunk of a document and feeds it through all of the relevant
charset probers.
After calling ``feed``, you can check the value of the ``done``
attribute to see if you need to continue feeding the
``UniversalDetector`` more data, or if it has made a prediction
(in the ``result`` attribute).
.. note::
You should always call ``close`` when you're done feeding in your
document if ``done`` is not already ``True``.
Nz UTF-8-SIG ? r! zUTF-32s zX-ISO-10646-UCS-4-3412s zX-ISO-10646-UCS-4-2143r Tr" ),r*
isinstance bytearrayr+
startswithcodecsBOM_UTF8r) BOM_UTF32_LE BOM_UTF32_BEBOM_LEBOM_BEr- r r, HIGH_BYTE_DETECTORsearch HIGH_BYTE ESC_DETECTORr. ESC_ASCIIr' r stater DETECTINGfeedFOUND_IT charset_nameget_confidencer& r r r$ r( r r
NON_CJKappendr r
r WIN_BYTE_DETECTORr3 )r5 rB rA s r6 rW zUniversalDetector.feed s 99 (I. *H ~~""6??3 !,"% "
$$f&9&96;N;N%OP ,43TV W $$%89 !9"% " $$%89 !9"% " $$fmmV]]%CD ,43TV W !DN{{:&2
5 5 5&&--h7$.$8$8!!!Z%:%::%%,,T__x-GH$.$8$8!"23- ###0?D % % )?)? ?##((2l6K6KK $ 4 4 A A"&"6"6"E"E"G "
!
4 4 4+++;D //;;x(L,A,AA$*$7$7&,&;&;&=$*OO#DK
!%DI 0 %%,,X6&*# 7# 7r% c H | j r | j S d| _ | j s | j j d nD| j
t j k( r
dddd| _ n| j
t j k( rd}d}d}| j D ] }|s|j }||kD s |}|}! |r|| j kD r|j }|J |j }|j }|j d r(| j r | j j# || }| j$ r.| j& j# |xs dj | }|||j( d| _ | j j+ t, j. k r| j d
| j j d | j D ] }|st1 |t2 rR|j4 D ]B }| j j d |j |j( |j D h| j j d |j |j( |j | j S )
z
Stop analyzing the current document and come up with a final
prediction.
:returns: The ``result`` attribute, a ``dict`` with the keys
`encoding`, `confidence`, and `language`.
Tzno data received!r rD rE r! Nr ziso-8859r" z no probers hit minimum thresholdz%s %s confidence = %s)r* r) r+ r2 debugr- r r, rR r( rZ MINIMUM_THRESHOLDrY lowerrI r3 ISO_WIN_MAPgetr
LEGACY_MAPr$ getEffectiveLevelr/ DEBUGrG r probers) r5 prober_confidencemax_prober_confidence
max_proberrA rY lower_charset_namer# group_probers r6 closezUniversalDetector.close sa 99;; ~~ KK 1 2
*"7"7
7'.crRDK
*"6"6
6 $ $' ! J // $*$9$9$;!$'<<,=)!'J
0 4t7M7M M ) 6 6 #///%1%7%7%9" ' 6 6 8
&00<**'+'7'7';'; . ( ,,#'??#6#6%+224l$L !-", * 3 3 ;; ( ( *gmm ;{{:&. !!"DE$($9$9L' !,0BC&2&:&:F KK - - 7 & 3 3 & & 5 5 7 '; )) 3 ( 5 5 ( 1 1 ( 7 7 9 %:$ {{r% )r N) r1
__module__ __qualname____doc__r` recompilerP rS r] rb rd r
ALLboolr7 propertyintr; r= r r r? r4 r bytesrH rW r rm r% r6 r r 8 s7 #N3 2::l+L"
>2$$$$$$$% K $ $ J '5&8&8%*# #
. !S ! ! #t # # %m!4 % %&A+U5)#3 4 A+ A+FMz Mr% r ) rp rJ r/ rq typingr r r charsetgroupproberr
charsetproberr enumsr r
r escproberr latin1proberr
macromanproberr mbcsgroupproberr
resultdictr sbcsgroupproberr
utf1632proberr r rx r% r6 r sH 8 ( ( 2 ( ; ; ' & * , " , (r rr%