U
fv. @ sl d dl Z d dlZd dlZd dlmZ d dlmZ d dlmZ d dl m
Z
d dl m Z m
Z
mZmZmZmZ d dlmZ dd lmZmZmZmZmZmZ eed
ee d d d
Z eed
eed ddZ eed
eee d ddZ eed
ee d ddZ eed
ee d ddZ!eed
ee d ddZ"eed
ee d ddZ#eed
ee d ddZ$eed
ee d d d Z%eed
ee d d d Z&eed
ee d d d!Z'eed
ee d d"d#Z(eed
ee d d$d%Z)eed
ee d d&d'Z*eed
ee d d(d)Z+eed
ee d d*d+Z,ee-ed
ee d,d-d.Z.eed
ee d d/d0Z/dSe0e1ee d2d3d4Z2ed5d
ee d6d7d8Z3e0eee e0f d9d:d;Z4ee d<d=d>Z5dTee ed@dAdBZ6ee
e dCdDdEZ7eee8dFdGdHZ9eee dFdIdJZ:dKej;dLfee1eddMdNdOZdS )V N)IncrementalDecoder)aliases) lru_cache)findall) GeneratorListOptionalSetTupleUnion)MultibyteIncrementalDecoder )ENCODING_MARKSIANA_SUPPORTED_SIMILAR RE_POSSIBLE_ENCODING_INDICATIONUNICODE_RANGES_COMBINED UNICODE_SECONDARY_RANGE_KEYWORDUTF8_MAXIMAL_ALLOCATION)maxsize) characterreturnc C sf zt | }W n tk
r$ Y dS X d|kpdd|kpdd|kpdd|kpdd|kpdd|kpdd|kpdd |kS )
NFz
WITH GRAVEz
WITH ACUTEz WITH CEDILLAzWITH DIAERESISzWITH CIRCUMFLEXz
WITH TILDEz WITH MACRONzWITH RING ABOVE unicodedataname
ValueErrorr description r K/opt/alt/python38/lib64/python3.8/site-packages/charset_normalizer/utils.pyis_accentuated s& r c C s. t | }|s| S |d}tt|d dS )N r )r
decompositionsplitchrint)r Z
decomposedcodesr r r
remove_accent( s
r' c C s. t | }t D ]\}}||kr| S qdS )zK
Retrieve the Unicode range official name from a single character.
N)ordr items)r Z
character_ord
range_nameZ ord_ranger r r
unicode_range3 s
r+ c C s. zt | }W n tk
r$ Y dS X d|kS )NFZLATINr r r r r is_latinA s
r, c C s2 t | }d|krdS t| }|d kr*dS d|kS )NPTF Punctuationr categoryr+ r character_categorycharacter_ranger r r is_punctuationJ s
r4 c C sB t | }d|ksd|kr dS t| }|d kr2dS d|ko@|dkS )NSNTFZFormsZLor/ r1 r r r is_symbolY s
r7 c C s$ t | }|d krdS d|kp"d|kS )NFZ EmoticonsZ Pictographs)r+ )r r3 r r r is_emoticonh s r8 c C s. | s| dkrdS t| }d|kp,|dkS )N> > ｜<+TZ> PoPdPc)isspacer r0 )r r2 r r r is_separatorr s
rB c C s | | kS N)islowerisupperr r r r is_case_variable| s rG c C s. zt | }W n tk
r$ Y dS X d|kS )NFCJKr r Zcharacter_namer r r is_cjk s
rJ c C s. zt | }W n tk
r$ Y dS X d|kS )NFZHIRAGANAr rI r r r is_hiragana s
rK c C s. zt | }W n tk
r$ Y dS X d|kS )NFZKATAKANAr rI r r r is_katakana s
rL c C s. zt | }W n tk
r$ Y dS X d|kS )NFZHANGULr rI r r r is_hangul s
rM c C s. zt | }W n tk
r$ Y dS X d|kS )NFZTHAIr rI r r r is_thai s
rN c C s. zt | }W n tk
r$ Y dS X d|kS )NFARABICr rI r r r is_arabic s
rP c C s6 zt | }W n tk
r$ Y dS X d|ko4d|kS )NFrO z
ISOLATED FORMr rI r r r is_arabic_isolated_form s
rQ )r* r c s t fddtD S )Nc 3 s | ]}| kV qd S rC r ).0keywordr* r r s z-is_unicode_range_secondary..)anyr rT r rT r is_unicode_range_secondary s rW c C s( | dko&| dko&| dko&| dkS )NFu )rA isprintablerF r r r is_unprintable s
rZ )sequence search_zoner c C s t | tstt| }tt| dt|| jddd}t|dkrHdS |D ]N}| dd}t
D ]0\}}||kr| S ||krh| S qhqLdS )zW
Extract using ASCII-only decoder any specified encoding in the first n-bytes.
Nasciiignoreerrorsr -_)
isinstancebytes TypeErrorlenr r mindecodelowerreplacer r) )r\ r] Zseq_lenresultsZspecified_encodingencoding_alias
encoding_ianar r r any_specified_encoding s"
ro )r r c C s | dkp t td| jtS )zQ
Verify is a specific encoding is a multi byte one based on it IANA name
> utf_16_be utf_32_beutf_16utf_8 utf_32_leutf_7 utf_8_sigutf_32 utf_16_le encodings.{})
issubclass importlib
import_moduleformatr r )r r r r is_multi_byte_encoding s
r )r\ r c C sJ t D ]@}t | }t|tr |g}|D ] }| |r$||f S q$qdS )z9
Identify and extract SIG/BOM in given sequence.
)N )r rd re
startswith)r\
iana_encodingZmarksmarkr r r identify_sig_or_bom s
r )r r c C s | dkS )N> rx rs r )r r r r should_strip_sig_or_bom s r T)cp_namestrictr c C sL | dd} t D ] \}}| ||fkr| S q|rHtd| | S )Nrb rc z Unable to retrieve IANA for '{}')rj rk r r) r r~ )r r rm rn r r r iana_name" s
r )decoded_sequencer c C s4 t }| D ] }t|}|d kr q
|| q
t|S rC )setr+ addlist)r rangesr r3 r r r
range_scan2 s r ) iana_name_a iana_name_br c C s t | st |rdS td| j}td|j}|dd}|dd}d}tdD ]*}t|g}||||krX|d7 }qX|d S ) Ng rz r_ r` r r
)r r| r} r~ r rangere ri ) r r Z decoder_aZ decoder_bZid_aZid_bZcharacter_match_countiZ
to_be_decodedr r r
cp_similarity@ s
r c C s | t ko|t | kS )z
Determine if two code page are at least 80% similar. IANA_SUPPORTED_SIMILAR dict was generated using
the function cp_similarity.
)r )r r r r r
is_cp_similarX s
r Zcharset_normalizerz)%(asctime)s | %(levelname)s | %(message)s)r level
format_stringr c C s: t | }|| t }|t | || d S rC )logging getLoggersetLevel
StreamHandler setFormatter Formatter
addHandler)r r r loggerhandlerr r r set_logging_handlerc s
r )
sequencesrn offsets
chunk_sizebom_or_sig_availablestrip_sig_or_bom sig_payloadis_multi_byte_decoderdecoded_payloadr c c s* |r6|dkr6|D ]"} || | | }
|
s, q4|
V qn|D ]} | | } | t | d krXq:| | | | } |r||dkr||| } | j||rdndd}
|r | dkr t|d}
|r |
d |
|kr t| | d d D ]H}| || } |r|dkr|| } | j|dd}
|
d |
|kr q q|
V q:d S )
NF r_ r r` r r! )rg ri rh r )r rn r r r r r r r r chunkZ chunk_endZ cut_sequenceZchunk_partial_size_chkjr r r cut_sequence_chunksp s>
r )r[ )T)N)?r| r r codecsr Zencodings.aliasesr functoolsr rer typingr r r r r
r Z_multibytecodecr Zconstantr r r r r r strboolr r' r+ r, r4 r7 r8 rB rG rJ rK rL rM rN rP rQ rg rW rZ re r% ro r r r r r floatr r INFOr r r r r r r s