U
f1 @ s d dl Z d dlmZ d dlmZ d dlmZ d dlmZm Z m
Z
m Z m Z ddl
mZmZmZmZmZ ddlmZ dd lmZ dd
lmZmZmZmZm Z e e
e d d d
Z e e
e dddZ e e e
e d ddZ e e e
e d ddZ!eede e e"e"f dddZ#d-e
e e"e
e ddd Z$e e
e e%d d d Z&e e
e d d!d"Z'e
e ed#d$d%Z(eed#d&d'Z)ed(dd.e e%e e ed*d+d,Z*dS )/ N)IncrementalDecoder)Counter) lru_cache)r DictListOptionalTuple ) FREQUENCIESKO_NAMESLANGUAGE_SUPPORTED_COUNTTOO_SMALL_SEQUENCEZH_NAMES) is_suspiciously_successive_range)CoherenceMatches)is_accentuatedis_latinis_multi_byte_encodingis_unicode_range_secondary
unicode_range) iana_namereturnc s t | rtdtd| j}|dd}i d tddD ]^}|t|g}|r>t |}|dkrhq>t
|d kr>|krd|< | d
7 < d
7 q>t fd d D S )
zF
Return associated unicode ranges in a single byte code page.
z.Function not supported on multi-byte code pagez encodings.{}ignore)errorsr @ NFr c s g | ]}| d kr|qS )g333333? ).0character_rangecharacter_countZ seen_rangesr H/opt/alt/python38/lib64/python3.8/site-packages/charset_normalizer/cd.py
8 s z*encoding_unicode_range..) r IOError importlib
import_moduleformatr rangedecodebytesr r sorted)r decoderpichunkr r r r! encoding_unicode_range s0
r/ )
primary_ranger c C s> g }t D ],\}}|D ] }t|| kr|| q qq |S )z>
Return inferred languages used with a unicode range.
)r
itemsr append)r0 languageslanguage
characters characterr r r! unicode_range_languages@ s
r7 c C s< t | }d}|D ]}d|kr|} q&q|dkr4dgS t|S )z
Single-byte encoding language association. Some code page are heavily linked to particular language(s).
This function does the correspondence.
NZLatin Latin Based)r/ r7 )r Zunicode_rangesr0 Zspecified_ranger r r! encoding_languagesO s r9 c C s` | ds&| ds&| ds&| dkr,dgS | ds>| tkrDdgS | dsV| tkr\d gS g S )
z
Multi-byte encoding language association. Some code page are heavily linked to particular language(s).
This function does the correspondence.
Zshift_
iso2022_jpZeuc_jcp932JapanesegbChinese
iso2022_krKorean)
startswithr r )r r r r! mb_encoding_languagesc s rB )maxsize)r4 r c C sB d}d}t | D ](}|s$t|r$d}|rt|dkrd}q||fS )zg
Determine main aspects from a supported language if it contains accents and if is pure Latin.
FT)r
r r )r4 target_have_accentstarget_pure_latinr6 r r r! get_target_featuresx s rF F)r5 ignore_non_latinr c s g }t dd D }t D ]l\}}t|\}}|r@|dkr@q |dkrN|rNq t|}t fdd|D } | | }
|
dkr |||
f q t|dd d d
}d d |D S ) zE
Return associated languages associated to given characters.
c s s | ]}t |V qd S N)r )r r6 r r r! s z%alphabet_languages..Fc s g | ]}| kr|qS r r )r cr5 r r! r" s z&alphabet_languages..g?c S s | d S Nr r xr r r! z$alphabet_languages..Tkeyreversec S s g | ] }|d qS )r r )r Zcompatible_languager r r! r" s )anyr
r1 rF lenr2 r* ) r5 rG r3 Zsource_have_accentsr4 Zlanguage_charactersrD rE r Zcharacter_match_countratior rK r! alphabet_languages s" rW )r4 ordered_charactersr c C s | t krtd| d}tt | }t|}tt | }|dk}t|td|D ]D\}}||krfqRt | |} || }
t||
} |dkrt | | dkrqR|dkrt | | |d k r|d7 }qRt | d| } t | | d }
|d| }||d }tt|t| @ }tt|t|
@ }t| dkrJ|dkrJ|d7 }qRt|
dkrl|dkrl|d7 }qR|t| d
ks|t|
d
krR|d7 }qRqR|t| S ) aN
Determine if a ordered characters list (by occurrence from most appearance to rarest) match a particular language.
The result is a ratio between 0. (absolutely no correspondence) and 1. (near perfect fit).
Beware that is function is not strict on the match in order to ease the detection. (Meaning close match is 1.)
z{} not availabler F T r Ng?)
r
ValueErrorr& setrU zipr' indexintabs)r4 rX Zcharacter_approved_countZFREQUENCIES_language_setZordered_characters_countZ target_language_characters_countZlarge_alphabetr6 Zcharacter_rankZcharacter_rank_in_languageZexpected_projection_ratioZcharacter_rank_projectionZcharacters_before_sourceZcharacters_after_sourceZcharacters_beforeZcharacters_afterZbefore_match_countZafter_match_countr r r! characters_popularity_compare st
rb )decoded_sequencer c C s i }| D ]~}| dkrqt|}|dkr,qd}|D ]}t||dkr4|} qPq4|dkr\|}||krr| ||< q|| | 7 < qt| S )a
Given a decoded text sequence, return a list of str. Unicode range / alphabet separation.
Ex. a text containing English/Latin with a bit a Hebrew will return two items in the resulting list;
One containing the latin letters and the other hebrew.
FN)isalphar r lowerlistvalues)rc Zlayersr6 r Zlayer_target_rangeZdiscovered_ranger r r! alpha_unicode_split s, rh )resultsr c sf i | D ]8}|D ].}|\}}| kr0|g |< q | | qq fdd D }t|dd ddS )z
This function merge results previously given by the function coherence_ratio.
The return type is the same as coherence_ratio.
c s. g | ]&}|t t | t | d fqS )rZ )roundsumrU )r r4 Zper_language_ratiosr r! r" 1 s z*merge_coherence_ratios..c S s | d S rL r rM r r r! rO < rP z(merge_coherence_ratios..TrQ )r2 r* )ri resultZ
sub_resultr4 rV merger rl r! merge_coherence_ratios# s
ro c s t | D ]6}|\}}|dd}| kr2g |< | | q
t fdd D rg } D ]}||t | f q`|S | S )u
We shall NOT return "English—" in CoherenceMatches because it is an alternative
of "English". This function only keeps the best match and remove the em-dash in it.
u — c 3 s | ]}t | d kV qdS )r N)rU )r eZ
index_resultsr r! rI O s z/filter_alt_coherence_matches..)dictreplacer2 rT max)ri rm r4 rV Z
no_em_nameZfiltered_resultsr rr r! filter_alt_coherence_matches? s rv i 皙?)rc threshold lg_inclusionr c C s g }d}d}|dk r | dng }d|kr8d}|d t| D ]}t|}| } tdd | D }
|
tkrpq@d d
| D } |pt| |D ]J} t| | }
|
|k rqn|
d kr|d 7 }| | t
|
d
f |dkr q@qq@t t |dd ddS )z
Detect ANY language that can be identified in given sequence. The sequence will be analysed by layers.
A layer = Character extraction by alphabets/ranges.
Fr N,r8 Tc s s | ]\}}|V qd S rH r r rJ or r r! rI q s z"coherence_ratio..c S s g | ] \}}|qS r r r{ r r r! r" v s z#coherence_ratio..g?r rZ r[ c S s | d S rL r rM r r r! rO rP z!coherence_ratio..rQ )
splitremoverh r most_commonrk r
rW rb r2 rj r* rv )rc rx ry ri rG Zsufficient_match_countZlg_inclusion_listlayerZsequence_frequenciesr r Zpopular_character_orderedr4 rV r r r! coherence_ratioZ sD
r )F)rw N)+r$ codecsr collectionsr functoolsr typingZ TypeCounterr r r r Zconstantr
r r r
r mdr modelsr utilsr r r r r strr/ r7 r9 rB boolrF rW floatrb rh ro rv r r r r r! sL ' $ P'