R`i, l d dl Z d dlZd dlmZmZ ddlmZmZ ej d Z G d d Z
y) N)OptionalUnion )LanguageFilter ProbingStates% [a-zA-Z]*[-]+[a-zA-Z]*[^a-zA-Z-]?c e Zd ZdZej
fdeddfdZddZede e
fd Z ede e
fd Z d e
eef defd
Zedefd Zdefd Zed
e
eef defd Zed
e
eef defd Zed
e
eef defd Zy)
CharSetProbergffffff? lang_filterreturnNc t j | _ d| _ || _ t j t | _ y )NT) r DETECTING_stateactiver
logging getLogger__name__logger)selfr
s /builddir/build/BUILDROOT/alt-python312-pip-23.3.1-3.el8.x86_64/opt/alt/python312/lib/python3.12/site-packages/pip/_vendor/chardet/charsetprober.py__init__zCharSetProber.__init__, s0 ",, & ''1 c . t j | _ y N)r r
r r s r resetzCharSetProber.reset2 s ",, r c y r r s r charset_namezCharSetProber.charset_name5 s r c t r NotImplementedErrorr s r languagezCharSetProber.language9 s !!r byte_strc t r r )r r# s r feedzCharSetProber.feed= s !!r c | j S r )r r s r statezCharSetProber.state@ s {{r c y)Ng r r s r get_confidencez CharSetProber.get_confidenceD s r bufc 4 t j dd| } | S )Ns ([ -])+ )resub)r* s r filter_high_byte_onlyz#CharSetProber.filter_high_byte_onlyG s ff&c2
r c t }t j | }|D ]C }|j |dd |dd }|j s|dk rd}|j | E |S )u7
We define three types of bytes:
alphabet: english alphabets [a-zA-Z]
international: international characters [-ÿ]
marker: everything else [^a-zA-Z-ÿ]
The input buffer can be thought to contain a series of words delimited
by markers. This function works to filter all words that contain at
least one international character. All contiguous sequences of markers
are replaced by a single space ascii character.
This filter applies to all scripts which do not use English characters.
N r, ) bytearrayINTERNATIONAL_WORDS_PATTERNfindallextendisalpha)r* filteredwordsword last_chars r filter_international_wordsz(CharSetProber.filter_international_wordsL sq ;
,33C8D OOD"I & RS I $$&9w+> OOI & r c * t }d}d}t | j d } t | D ]F \ }}|dk( r|dz }d}|dk( s||kD r'|s%|j | || |j d d}H |s|j | |d |S )
a[
Returns a copy of ``buf`` that retains only the sequences of English
alphabet and high byte characters that are not between <> characters.
This filter can be applied to all scripts which contain both English
characters and extended ASCII characters, but is currently only used by
``Latin1Prober``.
Fr c >r (? I B $U5)#3 4 $ $ $r r ) r r- typingr r enumsr r compiler4 r r r r rU s3 : " / (bjj8
k kr