U
ʗRe3 @ s d Z ddlZddlZddlZddlmZ ddlmZmZm Z ddl
m Z ddl m
Z
ddlmZ dd lmZ dd
lmZ G d d d ZdS )
a
Module containing the UniversalDetector detector class, which is the primary
class a user of ``chardet`` should use.
:author: Mark Pilgrim (initial port to Python)
:author: Shy Shalom (original C code)
:author: Dan Blanchard (major refactoring for 3.0)
:author: Ian Cordasco
N )CharSetGroupProber)
InputStateLanguageFilter ProbingState)EscCharSetProber) Latin1Prober)MBCSGroupProber)SBCSGroupProber)
UTF1632Proberc @ s e Zd ZdZdZedZedZedZ dddd d
d d d
dZ
e j fddZ
edd Zedd Zedd Zdd Zdd Zdd Zd S ) UniversalDetectoraq
The ``UniversalDetector`` class underlies the ``chardet.detect`` function
and coordinates all of the different charset probers.
To get a ``dict`` containing an encoding and its confidence, you can simply
run:
.. code::
u = UniversalDetector()
u.feed(some_bytes)
u.close()
detected = u.result
g?s [-]s (|~{)s [-]z Windows-1252z Windows-1250z Windows-1251z Windows-1256z Windows-1253z Windows-1255z Windows-1254z Windows-1257)z
iso-8859-1z
iso-8859-2z
iso-8859-5z
iso-8859-6z
iso-8859-7z
iso-8859-8z
iso-8859-9z iso-8859-13c C sT d | _ d | _g | _d | _d | _d | _d | _d | _|| _t
t | _ d | _
| d S N)_esc_charset_prober_utf1632_prober_charset_probersresultdone _got_data _input_state
_last_char lang_filterlogging getLogger__name__logger_has_win_bytesreset)selfr r /builddir/build/BUILDROOT/alt-python38-pip-22.2.1-2.el8.x86_64/opt/alt/python38/lib/python3.8/site-packages/pip/_vendor/chardet/universaldetector.py__init__T s zUniversalDetector.__init__c C s | j S r
)r r r r r input_stateb s z UniversalDetector.input_statec C s | j S r
)r r! r r r
has_win_bytesf s z UniversalDetector.has_win_bytesc C s | j S r
)r r! r r r charset_probersj s z!UniversalDetector.charset_probersc C sf dddd| _ d| _d| _d| _tj| _d| _| jr>| j | j
rN| j
| j D ] }| qTdS )z
Reset the UniversalDetector and all of its probers back to their
initial states. This is called by ``__init__``, so you only need to
call this directly in between analyses of different documents.
N encoding
confidencelanguageF ) r r r r r
PURE_ASCIIr r r r r r )r proberr r r r n s
zUniversalDetector.resetc C s | j r
dS |sdS t|ts$t|}| js|tjrFdddd| _nv|tjtj frhdddd| _nT|drdddd| _n:|d rd
ddd| _n |tj
tj frd ddd| _d | _| jd
dk rd | _ dS | j t
jkr(| j|rt
j| _ n*| j t
jkr(| j| j| r(t
j| _ |dd | _| jsFt | _| jjtjkr| j|tjkr| jj | j dd| _d | _ dS | j t
jkr| j st | j | _ | j |tjkr| j j | j | j j!d| _d | _ n| j t
jkr| j"s4t#| j g| _"| j t$j%@ r&| j"&t' | j"&t( | j"D ]:}||tjkr:|j | |j!d| _d | _ qvq:| j)|rd | _*dS )a
Takes a chunk of a document and feeds it through all of the relevant
charset probers.
After calling ``feed``, you can check the value of the ``done``
attribute to see if you need to continue feeding the
``UniversalDetector`` more data, or if it has made a prediction
(in the ``result`` attribute).
.. note::
You should always call ``close`` when you're done feeding in your
document if ``done`` is not already ``True``.
Nz UTF-8-SIG ? r& zUTF-32s zX-ISO-10646-UCS-4-3412s zX-ISO-10646-UCS-4-2143zUTF-16Tr' )+r
isinstance bytearrayr
startswithcodecsBOM_UTF8r BOM_UTF32_LE BOM_UTF32_BEBOM_LEBOM_BEr r r+ HIGH_BYTE_DETECTORsearch HIGH_BYTE ESC_DETECTORr ESC_ASCIIr r stater DETECTINGfeedFOUND_IT charset_nameget_confidencer r r r) r r r NON_CJKappendr
r WIN_BYTE_DETECTORr )r byte_strr, r r r r@ s
zUniversalDetector.feedc C st | j r | jS d| _ | js&| jd n| jtjkrBdddd| _n| jtjkrd}d}d}| j D ]"}|sjq`|
}||kr`|}|}q`|r|| j kr|j }|j
}|
}|d r| jr| j||}|||jd| _| j tjkrn| jd
dkrn| jd | j D ]`}|sq t|trP|jD ] }| jd |j |j|
q,n| jd |j |j|
q | jS )
z
Stop analyzing the current document and come up with a final
prediction.
:returns: The ``result`` attribute, a ``dict`` with the keys
`encoding`, `confidence`, and `language`.
Tzno data received!asciir- r. r& Nr% ziso-8859r' z no probers hit minimum thresholdz%s %s confidence = %s)r r r r debugr r r+ r; r rC MINIMUM_THRESHOLDrB lowerr2 r ISO_WIN_MAPgetr) getEffectiveLevelr DEBUGr0 r probers) r prober_confidencemax_prober_confidence
max_proberr, rB lower_charset_namer( group_proberr r r close sj
zUniversalDetector.closeN)r
__module__ __qualname____doc__rJ recompiler9 r< rF rL r ALLr propertyr" r# r$ r r@ rV r r r r r 4 s2
r )rY r3 r rZ charsetgroupproberr enumsr r r escproberr latin1proberr mbcsgroupproberr sbcsgroupproberr
utf1632proberr r r r r r s