U
fiR
@ s d dl Z d dlmZ d dlmZmZmZmZmZ ddl m
Z
m Z m Z m
Z
ddlmZmZmZmZ ddlmZ ddlmZmZ dd lmZmZmZm Z m Z m Z m Z e d
Z!e " Z#e#$e %d d ee&e'f e(e(e)eee* eee* e+e+e)e+ed ddZ,d ee(e(e)eee* eee* e+e+e)e+ed ddZ-d ee*e&ef e(e(e)eee* eee* e+e+e)e+ed ddZ.d!eee*ee&f e(e(e)eee* eee* e+e+e)e+e+d d d Z/dS )" N)PathLike)BinaryIOListOptionalSetUnion )coherence_ratioencoding_languagesmb_encoding_languagesmerge_coherence_ratios)IANA_SUPPORTEDTOO_BIG_SEQUENCETOO_SMALL_SEQUENCETRACE)
mess_ratio) CharsetMatchCharsetMatches)any_specified_encodingcut_sequence_chunks iana_nameidentify_sig_or_bom
is_cp_similaris_multi_byte_encodingshould_strip_sig_or_bomZcharset_normalizerz)%(asctime)s | %(levelname)s | %(message)s 皙?TF皙?) sequencessteps
chunk_size threshold cp_isolation cp_exclusionpreemptive_behaviourexplainlanguage_thresholdenable_fallbackreturnc
/ C s t | ttfs tdt| |r>tj}
tt t
t t | } | dkrt
d |rvtt t
|
prtj tt| dddg dgS |dk rtt d d
| d d |D }ng }|dk rtt d
d
| dd |D }ng }| || krtt d||| d}| }|dkr:| | |k r:t| | }t | tk } t | tk}
| rltt d| n|
rtt d| g }|rt| nd}|dk r|| tt d| t }g }g }d}d}d}t }t| \}}|dk r|| tt dt || |d d|kr.|d |t D ]}|rP||krPq6|rd||krdq6||krrq6| | d}||k}|ot |} |dkr|stt d| q6|dkr|stt d| q6z t |} W n. t t!fk
r tt d| Y q6Y nX zr|
r`| dkr`t"| dkrD| dtd n| t |td |d n&t"| dkrp| n| t |d |d }W n\ t#t$fk
r } z8t | t$stt d |t"| || W Y q6W 5 d} ~ X Y nX d} |D ] } t%|| rd } q q| r&tt d || q6t&|s2dnt || t| | }!| ob|dk obt || k }"|"rxtt d | tt |!d! }#t'|#d"}#d}$d}%g }&g }'zt(| ||!||| || | D ]|}(|&|( |'t)|(||d kodt | kod"kn |'d# |kr|$d7 }$|$|#ks0|r| dkr q:qW n@ t#k
r| } z tt d$|t"| |#}$d }%W 5 d} ~ X Y nX |%s|
r| sz | td%d j*|d&d' W nL t#k
r } z,tt d(|t"| || W Y q6W 5 d} ~ X Y nX |'rt+|'t |' nd})|)|ks*|$|#kr|| tt d)||$t,|)d* d+d, | r6|dd|fkr6|%s6t| ||dg |}*||kr|*}n|dkr|*}n|*}q6tt d-|t,|)d* d+d, | st-|}+nt.|}+|+rtt d.|t"|+ g },|dkr:|&D ],}(t/|(||+r&d/|+nd}-|,|- q t0|,}.|.r\tt d0|.| |t| ||)||.| ||ddfkr|)d1k rt
d2| |rtt t
|
t|| g S ||kr6t
d3| |rtt t
|
t|| g S q6t |dkr|s,|s,|r8tt d4 |rXt
d5|j1 || nd|rh|dks|r|r|j2|j2ks|dk rt
d6 || n|rt
d7 || |rt
d8|3 j1t |d n
t
d9 | rtt t
|
|S ):af
Given a raw bytes sequence, return the best possibles charset usable to render str objects.
If there is no results, it is a strong indicator that the source is binary/not text.
By default, the process will extract 5 blocks of 512o each to assess the mess and coherence of a given sequence.
And will give up a particular code page after 20% of measured mess. Those criteria are customizable at will.
The preemptive behavior DOES NOT replace the traditional detection workflow, it prioritize a particular code page
but never take it for granted. Can improve the performance.
You may want to focus your attention to some code page or/and not others, use cp_isolation and cp_exclusion for that
purpose.
This function will strip the SIG in the payload/sequence every time except on UTF-16, UTF-32.
By default the library does not setup any handler other than the NullHandler, if you choose to set the 'explain'
toggle to True it will alter the logger configuration to add a StreamHandler that is suitable for debugging.
Custom logging format and handler can be set manually.
z4Expected object of type bytes or bytearray, got: {0}r z[ s z from_bytes..zacp_exclusion is set. use this flag for debugging purpose. limited list of encoding excluded : %s.c S s g | ]}t |d qS r, r- r. r1 r1 r2 r3 f s z^override steps (%i) and chunk_size (%i) as content does not fit (%i byte(s) given) parameters.r z>Trying to detect encoding from a tiny portion of ({}) byte(s).zIUsing lazy str decoding because the payload is quite large, ({}) byte(s).z@Detected declarative mark in sequence. Priority +1 given for %s.zIDetected a SIG or BOM mark on first %i byte(s). Priority +1 given for %s.ascii> utf_32utf_16z\Encoding %s won't be tested as-is because it require a BOM. Will try some sub-encoder LE/BE.> utf_7zREncoding %s won't be tested as-is because detection is unreliable without BOM/SIG.z2Encoding %s does not provide an IncrementalDecoderg A)encodingz9Code page %s does not fit given bytes sequence at ALL. %sTzW%s is deemed too similar to code page %s and was consider unsuited already. Continuing!zpCode page %s is a multi byte encoding table and it appear that at least one character was encoded using n-bytes. zaLazyStr Loading: After MD chunk decode, code page %s does not fit given bytes sequence at ALL. %sg j@strict)errorsz^LazyStr Loading: After final lookup, code page %s does not fit given bytes sequence at ALL. %szc%s was excluded because of initial chaos probing. Gave up %i time(s). Computed mean chaos is %f %%.d )ndigitsz=%s passed initial chaos probing. Mean measured chaos is %f %%z&{} should target any language(s) of {},z We detected language {} using {}r z.Encoding detection: %s is most likely the one.zoEncoding detection: %s is most likely the one as we detected a BOM or SIG within the beginning of the sequence.zONothing got out of the detection process. Using ASCII/UTF-8/Specified fallback.z7Encoding detection: %s will be used as a fallback matchz:Encoding detection: utf_8 will be used as a fallback matchz:Encoding detection: ascii will be used as a fallback matchz]Encoding detection: Found %s as plausible (best-candidate) for content. With %i alternatives.z=Encoding detection: Unable to determine any suitable charset.)4
isinstance bytearraybytes TypeErrorformattypeloggerlevel
addHandlerexplain_handlersetLevelr lendebug
removeHandlerloggingWARNINGr r logjoinintr r r appendsetr r
addr r ModuleNotFoundError ImportErrorstrUnicodeDecodeError LookupErrorr rangemaxr r decodesumroundr
r r r r8 fingerprintbest)/r r r! r" r# r$ r% r& r' r( Zprevious_logger_levellengthZis_too_small_sequenceZis_too_large_sequenceZprioritized_encodingsZspecified_encodingZtestedZtested_but_hard_failureZtested_but_soft_failureZfallback_asciiZ fallback_u8Zfallback_specifiedresultsZ sig_encodingZ sig_payloadZ
encoding_ianaZdecoded_payloadZbom_or_sig_availableZstrip_sig_or_bomZis_multi_byte_decodereZsimilar_soft_failure_testZencoding_soft_failedZr_Zmulti_byte_bonusZmax_chunk_gave_upZearly_stop_countZlazy_str_hard_failureZ md_chunksZ md_ratioschunkZmean_mess_ratioZfallback_entryZtarget_languagesZ cd_ratiosZchunk_languagesZcd_ratios_mergedr1 r1 r2
from_bytes! s
&
rh ) fpr r! r" r# r$ r% r& r' r( r) c
C s t | |||||||||
S )z
Same thing than the function from_bytes but using a file pointer that is already ready.
Will not close the file pointer.
)rh read)
ri r r! r" r# r$ r% r& r' r( r1 r1 r2 from_fp s rk ) pathr r! r" r# r$ r% r& r' r( r) c
C s< t | d(}
t|
|||||||||
W 5 Q R S Q R X dS )z
Same thing than the function from_bytes but with one extra step. Opening and reading given file path in binary mode.
Can raise IOError.
rbN)openrk ) rl r r! r" r# r$ r% r& r' r( ri r1 r1 r2 from_path s ro ) fp_or_path_or_payloadr r! r" r# r$ r% r& r' r( r) c
C sz t | ttfr,t| ||||||||| d
}
nHt | ttfrXt| ||||||||| d
}
n t| ||||||||| d
}
|
S )a)
Detect if the given input (file, bytes, or path) points to a binary file. aka. not a string.
Based on the same main heuristic algorithms and default kwargs at the sole exception that fallbacks match
are disabled to be stricter around ASCII-compatible but unlikely to be a string.
) r r! r" r# r$ r% r& r' r( )rB rZ r ro rD rC rh rk ) rp r r! r" r# r$ r% r& r' r( Zguessesr1 r1 r2 is_binary3 sX
rq ) r r r NNTFr T) r r r NNTFr T) r r r NNTFr T) r r r NNTFr F)0rP osr typingr r r r r cdr r
r r Zconstantr
r r r mdr modelsr r utilsr r r r r r r getLoggerrH
StreamHandlerrK setFormatter FormatterrD rC rT floatrZ boolrh rk ro rq r1 r1 r1 r2 s $
Z
!