2bi9! B S SK JrJr SSKJr SSKJr " S S\5 rg) )ListUnion )
CharSetProber) ProbingStatec > ^ \ rS rSrSrSrSrSU 4S jjrSU 4S jjr\ S\
4S j5 r \ S\
4S
j5 r S\
4S jrS\
4S jrS\4S
jrS\4S jrS\4S jrS\4S jrS\\ SS4S jrS\\ SS4S jrS\\\4 S\ 4S jr \ S\ 4S j5 r S\
4S jr Sr U =r!$ )
UTF1632Prober aL
This class simply looks for occurrences of zero bytes, and infers
whether the file is UTF16 or UTF32 (low-endian or big-endian)
For instance, files looking like ( [nonzero] )+
have a good probability to be UTF32BE. Files looking like ( [nonzero] )+
may be guessed to be UTF16BE, and inversely for little-endian varieties.
gGz?returnNc > [ TU ] 5 SU l S/S- U l S/S- U l [
R U l / SQU l SU l SU l
SU l SU l SU l
SU l U R 5 g )Nr r r r r F)super__init__position zeros_at_modnonzeros_at_modr DETECTING_statequadinvalid_utf16beinvalid_utf16leinvalid_utf32beinvalid_utf32le'first_half_surrogate_pair_detected_16be'first_half_surrogate_pair_detected_16leresetself __class__s ړ/builddir/build/BUILDROOT/alt-python313-pip-23.3.1-3.el8.x86_64/opt/alt/python313/lib/python3.13/site-packages/pip/_vendor/chardet/utf1632prober.pyr UTF1632Prober.__init__) s~
C!G !sQw ",, $ $ $ $ 7<47<4
c > [ TU ] 5 SU l S/S- U l S/S- U l [
R U l SU l SU l SU l
SU l SU l SU l
/ SQU l g )Nr r Fr )r r r r r r r r r r r r r r r r s r" r UTF1632Prober.reset8 ss
C!G !sQw ",, $ $ $ $ 7<47<4 r$ c U R 5 ( a gU R 5 ( a gU R 5 ( a gU R 5 ( a gg)Nzutf-32bezutf-32lezutf-16bezutf-16lezutf-16)is_likely_utf32beis_likely_utf32leis_likely_utf16beis_likely_utf16ler s r" charset_nameUTF1632Prober.charset_nameF sM ! ! # # ! ! # # ! ! # # ! ! # # r$ c g)N r, s r" languageUTF1632Prober.languageS s r$ c 4 [ SU R S- 5 $ )N ?g @maxr r, s r" approx_32bit_chars UTF1632Prober.approx_32bit_charsW 3
+,,r$ c 4 [ SU R S- 5 $ )Nr5 g @r6 r, s r" approx_16bit_chars UTF1632Prober.approx_16bit_charsZ r: r$ c U R 5 nXR : =( a U R S U- U R : =( a U R S U- U R : =( a] U R S U- U R : =( a7 U R S U- U R : =( a U R
( + $ Nr r )r8 MIN_CHARS_FOR_DETECTIONr EXPECTED_RATIOr r r approx_charss r" r( UTF1632Prober.is_likely_utf32be] s ..0 ; ;;
a < /$2E2E E
)!!!$|3d6I6II
)!!!$|3d6I6II
) $$Q',69L9LL
) (((
r$ c U R 5 nXR : =( a U R S U- U R : =( a U R S U- U R : =( a] U R S U- U R : =( a7 U R S U- U R : =( a U R
( + $ r? )r8 rB r rC r r rD s r" r) UTF1632Prober.is_likely_utf32leg s ..0 ; ;;
#l 2T5H5H H
)!!!$|3d6I6II
)!!!$|3d6I6II
) !!!$|3d6I6II
) (((
r$ c F U R 5 nXR : =( a} U R S U R S - U- U R : =( aG U R S U R S - U- U R : =( a U R
( + $ )Nr rA r r@ )r< rB r rC r r rD s r" r* UTF1632Prober.is_likely_utf16beq ..0 ; ;;
!
!!
$t';';A'>
>, N!!
"
)""1%(9(9!(<< L!!"
) (((
r$ c F U R 5 nXR : =( a} U R S U R S - U- U R : =( aG U R S U R S - U- U R : =( a U R
( + $ )Nr r@ r rA )r< rB r rC r r rD s r" r+ UTF1632Prober.is_likely_utf16le{ rK r$ r c US S:w d. US S: d% US S:X a# US S:X a SUS s=:: a S:: a
O OSU l US S:w d. US S: d% US S:X a& US S:X a SUS s=:: a S:: a O g SU l g g g g )
z
Validate if the quad of bytes is valid UTF-32.
UTF-32 is valid in the range 0x00000000 - 0x0010FFFF
excluding 0x0000D800 - 0x0000DFFF
https://en.wikipedia.org/wiki/UTF-32
r r r@ TrA N)r r )r r s r" validate_utf32_characters'UTF1632Prober.validate_utf32_characters s
GqLAw~Q1 aA $$q'2IT2I#'D GqLAw~Q1 aA $$q'2IT2I#'D 3J r$ pairc U R ( d6 SUS s=:: a S:: a O OSU l O=SUS s=:: a S:: a
O O*SU l O"SUS s=:: a S:: a O OSU l OSU l U R ( d7 SUS s=:: a S:: a O OSU l g SUS s=:: a S:: a O g SU l g g SUS s=:: a S:: a O OSU l g SU l g )
a
Validate if the pair of bytes is valid UTF-16.
UTF-16 is valid in the range 0x0000 - 0xFFFF excluding 0xD800 - 0xFFFF
with an exception for surrogate pairs, which must be in the range
0xD800-0xDBFF followed by 0xDC00-0xDFFF
https://en.wikipedia.org/wiki/UTF-16
rP r T rQ Fr N)r r r r )r rT s r" validate_utf16_characters'UTF1632Prober.validate_utf16_characters s ;;tAw&$&?C<a(D('+$tAw&$&?D<'+$;;tAw&$&?C<a(D('+$ ) tAw&$&?D<'+$r$ byte_strc U H nU R S- nX R U' US:X aW U R U R 5 U R U R SS 5 U R U R SS 5 US:X a U R U== S-
ss' OU R
U== S-
ss' U =R S-
sl M U R $ )Nr rA r r@ r )r r rR rX r r state)r rZ cmod4s r" feedUTF1632Prober.feed s A==1$D IIdOqy..tyy9..tyy1~>..tyy1~>Av!!$'1,'$$T*a/* MMQ M zzr$ c B U R [ R [ R 1; a U R $ U R 5 S: a! [ R U l U R $ U R
S: a [ R U l U R $ )Ng?i )r r NOT_MEFOUND_ITget_confidencer r, s r" r\ UTF1632Prober.state sz ;;<.. 0E0EF F;; 4 '&//DK
{{ ]]X
% '--DK{{r$ c U R 5 ( d? U R 5 ( d* U R 5 ( d U R 5 ( a S$ S$ )Ng333333?g )r+ r* r) r( r, s r" rd UTF1632Prober.get_confidence sT &&(())++))++))++
r$ ) r r r r r r r r r r r )r N)"__name__
__module__ __qualname____firstlineno____doc__rB rC r r propertystrr- r2 floatr8 r< boolr( r) r* r+ r intrR rX r bytes bytearrayr r_ r\ rd __static_attributes__
__classcell__)r! s @r" r r s !N
!
c
# -E --E -
4
4
4
4
(d3i (D (, ,d3i ,D ,@
U5)#3 4
|
r$ r N)typingr r
charsetproberr enumsr r r1 r$ r" ry s * ( F
M F
r$