U
ʗRea @ s, d dl mZ d dlmZ G dd deZdS ) )
CharSetProber) ProbingStatec s e Zd ZdZdZdZ fddZ fddZedd Z ed
d Z
d d
Z dd Z dd Z
dd Zdd Zdd Zdd Zdd Zd d Zed d Zd d! Z ZS )"
UTF1632Proberad
This class simply looks for occurrences of zero bytes, and infers
whether the file is UTF16 or UTF32 (low-endian or big-endian)
For instance, files looking like ( [nonzero] )+
have a good probability to be UTF32BE. Files looking like ( [nonzero] )+
may be guessed to be UTF16BE, and inversely for little-endian varieties.
gGz?c sn t d| _dgd | _dgd | _tj| _ddddg| _d| _ d| _
d| _ d| _ d| _
d| _| d S N F)super__init__position zeros_at_modnonzeros_at_modr DETECTING_statequadinvalid_utf16beinvalid_utf16leinvalid_utf32beinvalid_utf32le'first_half_surrogate_pair_detected_16be'first_half_surrogate_pair_detected_16leresetself __class__ /builddir/build/BUILDROOT/alt-python38-pip-22.2.1-2.el8.x86_64/opt/alt/python38/lib/python3.8/site-packages/pip/_vendor/chardet/utf1632prober.pyr
' s
zUTF1632Prober.__init__c sf t d| _dgd | _dgd | _tj| _d| _d| _ d| _
d| _ d| _ d| _
ddddg| _d S r )r r r r r
r r r r r r r r r r r r r r r 6 s
zUTF1632Prober.resetc C s4 | r dS | rdS | r$dS | r0dS dS )Nzutf-32bezutf-32lezutf-16bezutf-16lezutf-16)is_likely_utf32beis_likely_utf32leis_likely_utf16beis_likely_utf16ler r r r charset_nameD s zUTF1632Prober.charset_namec C s dS )N r r r r r languageQ s zUTF1632Prober.languagec C s t d| jd S )N ?g @maxr r r r r approx_32bit_charsU s z UTF1632Prober.approx_32bit_charsc C s t d| jd S )Nr% g @r& r r r r approx_16bit_charsX s z UTF1632Prober.approx_16bit_charsc C sj | }|| jkoh| jd | | jkoh| jd | | jkoh| jd | | jkoh| jd | | jkoh| j S Nr r )r( MIN_CHARS_FOR_DETECTIONr EXPECTED_RATIOr
r r approx_charsr r r r [ s
z UTF1632Prober.is_likely_utf32bec C sj | }|| jkoh| jd | | jkoh| jd | | jkoh| jd | | jkoh| jd | | jkoh| j S r* )r( r- r
r. r r r/ r r r r e s
z UTF1632Prober.is_likely_utf32lec C sV | }|| jkoT| jd | jd | | jkoT| jd | jd | | jkoT| j S )Nr r, r r+ )r) r- r
r. r r r/ r r r r o s
z UTF1632Prober.is_likely_utf16bec C sV | }|| jkoT| jd | jd | | jkoT| jd | jd | | jkoT| j S )Nr r+ r r, )r) r- r
r. r r r/ r r r r! y s
z UTF1632Prober.is_likely_utf16lec C s |d dksL|d dksL|d dkrR|d dkrRd|d krHdkrRn nd| _ |d dks|d dks|d dkr|d dkrd|d krdkrn nd| _d S )
z
Validate if the quad of bytes is valid UTF-32.
UTF-32 is valid in the range 0x00000000 - 0x0010FFFF
excluding 0x0000D800 - 0x0000DFFF
https://en.wikipedia.org/wiki/UTF-32
r r r+ Tr, N)r r )r r r r r validate_utf32_characters s8
z'UTF1632Prober.validate_utf32_charactersc C s | j sNd|d kr dkr*n nd| _ qxd|d krBdkrxn qxd| _n*d|d krfdkrrn nd| _ nd| _| jsd|d krdkrn nd| _qd|d krdkrn qd| _n*d|d krdkrn nd| _nd| _d S )
a9
Validate if the pair of bytes is valid UTF-16.
UTF-16 is valid in the range 0x0000 - 0xFFFF excluding 0xD800 - 0xFFFF
with an exception for surrogate pairs, which must be in the range
0xD800-0xDBFF followed by 0xDC00-0xDFFF
https://en.wikipedia.org/wiki/UTF-16
r2 r T r3 Fr N)r r r r )r pairr r r validate_utf16_characters s
z'UTF1632Prober.validate_utf16_charactersc C s |D ]}| j d }|| j|< |dkrX| | j | | jdd | | jdd |dkrt| j| d7 < n| j| d7 < | j d7 _ q| jS )Nr r, r r+ r )r r r4 r8 r r
state)r byte_strcmod4r r r feed s
zUTF1632Prober.feedc C sF | j tjtjhkr| j S | dkr.tj| _ n| jdkr@tj| _ | j S )Ng?i )r r NOT_MEFOUND_ITget_confidencer r r r r r9 s
zUTF1632Prober.statec C s( | s | s | s | r$dS dS )Ng333333?g )r! r r r r r r r r@ s z UTF1632Prober.get_confidence)__name__
__module__ __qualname____doc__r- r. r
r propertyr" r$ r( r) r r r r! r4 r8 r= r9 r@
__classcell__r r r r r s*
r N)
charsetproberr enumsr r r r r r s