|
Server : Apache System : Linux iad1-shared-b8-43 6.6.49-grsec-jammy+ #10 SMP Thu Sep 12 23:23:08 UTC 2024 x86_64 User : dh_edsupp ( 6597262) PHP Version : 8.2.26 Disable Function : NONE Directory : /lib/python3/dist-packages/chardet/__pycache__/ |
Upload File : |
o
-��_� � @ s0 d dl Z d dlZddlmZ G dd� de�ZdS )� N� )� ProbingStatec @ sn e Zd ZdZddd�Zdd� Zedd� �Zd d
� Zed d � �Z d
d� Z
e dd� �Z e dd� �Z
e dd� �ZdS )�
CharSetProbergffffff�?Nc C s d | _ || _t�t�| _d S �N)�_state� lang_filter�logging� getLogger�__name__�logger)�selfr � r
�7/usr/lib/python3/dist-packages/chardet/charsetprober.py�__init__' s zCharSetProber.__init__c C s t j| _d S r )r � DETECTINGr �r r
r
r �reset, s zCharSetProber.resetc C � d S r r
r r
r
r � charset_name/ s zCharSetProber.charset_namec C r r r
)r �bufr
r
r �feed3 � zCharSetProber.feedc C s | j S r )r r r
r
r �state6 s zCharSetProber.statec C s dS )Ng r
r r
r
r �get_confidence: r z CharSetProber.get_confidencec C s t �dd| �} | S )Ns ([ -])+� )�re�sub)r r
r
r �filter_high_byte_only= s z#CharSetProber.filter_high_byte_onlyc C s\ t � }t�d| �}|D ] }|�|dd� � |dd� }|�� s&|dk r&d}|�|� q |S )u9
We define three types of bytes:
alphabet: english alphabets [a-zA-Z]
international: international characters [-ÿ]
marker: everything else [^a-zA-Z-ÿ]
The input buffer can be thought to contain a series of words delimited
by markers. This function works to filter all words that contain at
least one international character. All contiguous sequences of markers
are replaced by a single space ascii character.
This filter applies to all scripts which do not use English characters.
s% [a-zA-Z]*[�-�]+[a-zA-Z]*[^a-zA-Z�-�]?N���� �r )� bytearrayr �findall�extend�isalpha)r �filtered�words�word� last_charr
r
r �filter_international_wordsB s � z(CharSetProber.filter_international_wordsc C s� t � }d}d}tt| ��D ]7}| ||d � }|dkr d}n|dkr$d}|dk rD|�� sD||kr@|s@|�| ||� � |�d� |d }q
|sP|�| |d � � |S )
a�
Returns a copy of ``buf`` that retains only the sequences of English
alphabet and high byte characters that are not between <> characters.
Also retains English alphabet and high byte characters immediately
before occurrences of >.
This filter can be applied to all scripts which contain both English
characters and extended ASCII characters, but is currently only used by
``Latin1Prober``.
Fr r � >� <Tr r N)r �range�lenr# r" )r r$ �in_tag�prev�curr�buf_charr
r
r �filter_with_english_lettersg s$
�z)CharSetProber.filter_with_english_lettersr )r
�
__module__� __qualname__�SHORTCUT_THRESHOLDr r �propertyr r r r � staticmethodr r( r1 r
r
r
r r # s
$r )r r �enumsr �objectr r
r
r
r �<module> s