2bi : S r SSKrSSKrSSKrSSKJrJrJr SSKJ r SSK
J r SSK J
r
JrJr SSKJr SS KJr SS
KJr SS KJr SS KJr SS
KJr SSK J r " S S5 r g)a
Module containing the UniversalDetector detector class, which is the primary
class a user of ``chardet`` should use.
:author: Mark Pilgrim (initial port to Python)
:author: Shy Shalom (original C code)
:author: Dan Blanchard (major refactoring for 3.0)
:author: Ian Cordasco
N)ListOptionalUnion )CharSetGroupProber)
CharSetProber)
InputStateLanguageFilter ProbingState)EscCharSetProber) Latin1Prober)MacRomanProber)MBCSGroupProber)
ResultDict)SBCSGroupProber)
UTF1632Proberc f \ rS rSrSrSr\R " S5 r\R " S5 r \R " S5 r
SSS S
S S S
SS.r SSSS
SSSS.r \
R S4S\
S\SS4S jjr\S\4S j5 r\S\4S j5 r\S\\ 4S j5 rS#S jrS \\\4 SS4S jr S\ 4S! jr S"r g)$UniversalDetector8 aI
The ``UniversalDetector`` class underlies the ``chardet.detect`` function
and coordinates all of the different charset probers.
To get a ``dict`` containing an encoding and its confidence, you can simply
run:
.. code::
u = UniversalDetector()
u.feed(some_bytes)
u.close()
detected = u.result
g?s [-]s (|~{)s [-]z Windows-1252z Windows-1250z Windows-1251z Windows-1256z Windows-1253z Windows-1255z Windows-1254z Windows-1257)
iso-8859-1z
iso-8859-2z
iso-8859-5z
iso-8859-6z
iso-8859-7z
iso-8859-8
iso-8859-9z iso-8859-13z ISO-8859-11GB18030CP949UTF-16)asciir ztis-620r gb2312zeuc-krzutf-16leF lang_filtershould_rename_legacyreturnNc S U l S U l / U l S SS S.U l SU l SU l [ R U l SU l Xl
[ R " [ 5 U l SU l X l U R# 5 g )N encoding
confidencelanguageF )_esc_charset_prober_utf1632_prober_charset_probersresultdone _got_datar
PURE_ASCII _input_state
_last_charr logging getLogger__name__logger_has_win_bytesr reset)selfr r s ڗ/builddir/build/BUILDROOT/alt-python313-pip-23.3.1-3.el8.x86_64/opt/alt/python313/lib/python3.13/site-packages/pip/_vendor/chardet/universaldetector.py__init__UniversalDetector.__init__d s
@D 8< 57 #
& 1 1 & ''1 #$8!
r& c U R $ N)r. r6 s r7 input_state UniversalDetector.input_state{ s r& c U R $ r; )r4 r< s r7
has_win_bytes UniversalDetector.has_win_bytes s """r& c U R $ r; )r) r< s r7 charset_probers!UniversalDetector.charset_probers s $$$r& c n SSSS.U l SU l SU l SU l [ R
U l SU l U R ( a U R R 5 U R ( a U R R 5 U R H nUR 5 M g)z
Reset the UniversalDetector and all of its probers back to their
initial states. This is called by ``__init__``, so you only need to
call this directly in between analyses of different documents.
Nr! r" Fr& ) r* r+ r, r4 r r- r. r/ r' r5 r( r) )r6 probers r7 r5 UniversalDetector.reset s $(sM # & 1 1 # # $ $ * * , & & (++F LLN ,r& byte_strc
U R ( a gU( d g[ U[ 5 ( d [ U5 nU R ( Gd UR [
R 5 ( a SSSS.U l OUR [
R [
R 45 ( a SSSS.U l OUR S5 ( a SSSS.U l OaUR S 5 ( a S
SSS.U l O?UR [
R [
R 45 ( a S SSS.U l S U l U R S
b S U l gU R [ R :X a U R R! U5 ( a [ R" U l O`U R [ R :X aB U R$ R! U R& U- 5 ( a [ R( U l USS U l U R* ( d [- 5 U l U R* R. [0 R2 :X al U R* R5 U5 [0 R6 :X a? U R* R8 U R* R; 5 SS.U l S U l gU R [ R( :X a U R< ( d [? U R@ 5 U l U R< R5 U5 [0 R6 :X aS U R< R8 U R< R; 5 U R< RB S.U l S U l ggU R [ R" :X GaL U RD ( d [G U R@ 5 /U l" U R@ [H RJ - ( a# U RD RM [O 5 5 U RD RM [Q 5 5 U RD RM [S 5 5 U RD H\ nUR5 U5 [0 R6 :X d M( UR8 UR; 5 URB S.U l S U l O U RT R! U5 ( a S U l+ ggg)a
Takes a chunk of a document and feeds it through all of the relevant
charset probers.
After calling ``feed``, you can check the value of the ``done``
attribute to see if you need to continue feeding the
``UniversalDetector`` more data, or if it has made a prediction
(in the ``result`` attribute).
.. note::
You should always call ``close`` when you're done feeding in your
document if ``done`` is not already ``True``.
Nz UTF-8-SIG ? r" zUTF-32s zX-ISO-10646-UCS-4-3412s zX-ISO-10646-UCS-4-2143r Tr# ),r+
isinstance bytearrayr,
startswithcodecsBOM_UTF8r* BOM_UTF32_LE BOM_UTF32_BEBOM_LEBOM_BEr. r r- HIGH_BYTE_DETECTORsearch HIGH_BYTE ESC_DETECTORr/ ESC_ASCIIr( r stater DETECTINGfeedFOUND_IT charset_nameget_confidencer' r r r% r) r r
NON_CJKappendr r
r WIN_BYTE_DETECTORr4 )r6 rH rF s r7 r] UniversalDetector.feed s 99 (I.. *H ~~~""6??33 !,"% "
$$f&9&96;N;N%OPP ,43TV W $$%899 !9"% " $$%899 !9"% " $$fmmV]]%CDD ,43TV W !DN{{:&2
5 5 5&&--h77$.$8$8!!!Z%:%::%%,,T__x-GHH$.$8$8!"23- ###0?D % % )?)? ?##((2l6K6KK $ 4 4 A A"&"6"6"E"E"G "
!
4 4 4+++;D //;;x(L,A,AA$*$7$7&,&;&;&=$*OO#DK
!%DI 0 %%,,X66&*# 7# 7r& c U R ( a U R $ SU l U R ( d U R R S5 GOiU R
[ R :X a
SSSS.U l GO>U R
[ R :X Ga SnSnSnU R H' nU( d M UR 5 nX: d M# UnUnM) U( a X R : a UR nUc eUR 5 nUR 5 nUR S 5 ( a, U R ( a U R R# Xe5 nU R$ ( a3 U R& R# U=( d SR 5 U5 nUUUR( S.U l U R R+ 5 [, R. :: a U R S
c U R R S 5 U R H nU( d M [1 U[2 5 ( aU UR4 HC nU R R S UR UR( UR 5 5 ME Mv U R R S UR UR( UR 5 5 M U R $ )
z
Stop analyzing the current document and come up with a final
prediction.
:returns: The ``result`` attribute, a ``dict`` with the keys
`encoding`, `confidence`, and `language`.
Tzno data received!r rJ rK r" Nr! ziso-8859r# z no probers hit minimum thresholdz%s %s confidence = %s)r+ r* r, r3 debugr. r r- rX r) r` MINIMUM_THRESHOLDr_ lowerrO r4 ISO_WIN_MAPgetr
LEGACY_MAPr% getEffectiveLevelr0 DEBUGrM r probers) r6 prober_confidencemax_prober_confidence
max_proberrF r_ lower_charset_namer$ group_probers r7 closeUniversalDetector.close s` 99;; ~~ KK 1 2
*"7"7
7'.crRDK
*"6"6
6 $ $' ! J // $*$9$9$;!$<,=)!'J
0 47M7M M ) 6 6 #///%1%7%7%9" ' 6 6 8
&00<<**'+'7'7';'; .( ,,#'??#6#6%+224l$L !-", * 3 3 ;; ( ( *gmm ;{{:&. !!"DE$($9$9L' !,0BCC&2&:&:F KK - - 7 & 3 3 & & 5 5 7 '; )) 3 ( 5 5 ( 1 1 ( 7 7 9 %:$ {{r& ) r) r' r, r4 r. r/ r( r+ r r3 r* r )r N) r2
__module__ __qualname____firstlineno____doc__rg recompilerV rY rc ri rk r
ALLboolr8 propertyintr= r@ r r rC r5 r bytesrN r] r rt __static_attributes__ r& r7 r r 8 s1 N3::l+L
>2$$$$$$$% K $ $ J '5&8&8%*# #
. !S ! ! #t # # %m!4 % %&A+U5)#3 4 A+ A+FMz Mr& r ) ry rP r0 rz typingr r r charsetgroupproberr
charsetproberr enumsr r
r escproberr latin1proberr
macromanproberr mbcsgroupproberr
resultdictr sbcsgroupproberr
utf1632proberr r r r& r7 r sH 8 ( ( 2 ( ; ; ' & * , " , (r rr&