D7net
Home
Console
Upload
information
Create File
Create Folder
About
Tools
:
/
opt
/
alt
/
python37
/
lib
/
python3.7
/
site-packages
/
charset_normalizer
/
__pycache__
/
Filename :
md.cpython-37.pyc
back
Copy
B �B�h�N � @ sf d dl mZ d dlmZ d dlmZ ddlmZmZm Z ddl m Z m Z m Z mZmZmZmZmZmZmZmZmZmZmZmZmZmZm Z G dd� d�Z G d d � d e �Z G d d � d e �Z G d d� de �Z G dd� de �Z!G dd� de �Z"G dd� de �Z#G dd� de �Z$G dd� de �Z%G dd� de �Z&edd �d d d d �d d!��Z'ed"d �d+d%d&d d&d'�d(d)��Z(d*S ),� )� annotations)� lru_cache)� getLogger� )� COMMON_SAFE_ASCII_CHARACTERS�TRACE� UNICODE_SECONDARY_RANGE_KEYWORD)�is_accentuated� is_arabic�is_arabic_isolated_form�is_case_variable�is_cjk� is_emoticon� is_hangul� is_hiragana� is_katakana�is_latin�is_punctuation� is_separator� is_symbol�is_thai�is_unprintable� remove_accent� unicode_range�is_cjk_uncommonc @ sP e Zd ZdZddd�dd�Zddd�dd �Zdd �d d �Zed d �dd��ZdS )�MessDetectorPluginzy Base abstract class used for mess detection plugins. All detectors MUST extend and implement given methods. �str�bool)� character�returnc C s t �dS )z@ Determine if given character should be fed in. N)�NotImplementedError)�selfr � r" �F/opt/alt/python37/lib/python3.7/site-packages/charset_normalizer/md.py�eligible' s zMessDetectorPlugin.eligible�Nonec C s t �dS )z� The main routine to be executed upon character. Insert the logic in witch the text would be considered chaotic. N)r )r! r r" r" r# �feed- s zMessDetectorPlugin.feed)r c C s t �dS )zB Permit to reset the plugin to the initial state. N)r )r! r" r" r# �reset4 s zMessDetectorPlugin.reset�floatc C s t �dS )z� Compute the chaos ratio based on what your feed() has seen. Must NOT be lower than 0.; No restriction gt 0. N)r )r! r" r" r# �ratio: s zMessDetectorPlugin.ratioN) �__name__� __module__� __qualname__�__doc__r$ r& r' �propertyr) r" r" r" r# r ! s r c @ sZ e Zd Zdd�dd�Zddd�dd �Zddd�d d �Zdd�d d �Zedd�dd��ZdS )� TooManySymbolOrPunctuationPluginr% )r c C s" d| _ d| _d| _d | _d| _d S )Nr F)�_punctuation_count� _symbol_count�_character_count�_last_printable_charZ_frenzy_symbol_in_word)r! r" r" r# �__init__D s z)TooManySymbolOrPunctuationPlugin.__init__r r )r r c C s |� � S )N)� isprintable)r! r r" r" r# r$ L s z)TooManySymbolOrPunctuationPlugin.eligiblec C sp | j d7 _ || jkrf|tkrft|�r8| jd7 _n.|�� dkrft|�rft|�dkrf| jd7 _|| _d S )Nr F� ) r2 r3 r r r0 �isdigitr r r1 )r! r r" r" r# r& O s z%TooManySymbolOrPunctuationPlugin.feedc C s d| _ d| _d| _d S )Nr )r0 r2 r1 )r! r" r" r# r' a s z&TooManySymbolOrPunctuationPlugin.resetr( c C s0 | j dkrdS | j| j | j }|dkr,|S dS )Nr g g333333�?)r2 r0 r1 )r! Zratio_of_punctuationr" r" r# r) f s z&TooManySymbolOrPunctuationPlugin.ratioN) r* r+ r, r4 r$ r& r' r. r) r" r" r" r# r/ C s r/ c @ sZ e Zd Zdd�dd�Zddd�dd �Zddd�d d �Zdd�d d �Zedd�dd��ZdS )�TooManyAccentuatedPluginr% )r c C s d| _ d| _d S )Nr )r2 �_accentuated_count)r! r" r" r# r4 s s z!TooManyAccentuatedPlugin.__init__r r )r r c C s |� � S )N)�isalpha)r! r r" r" r# r$ w s z!TooManyAccentuatedPlugin.eligiblec C s( | j d7 _ t|�r$| jd7 _d S )Nr )r2 r r9 )r! r r" r" r# r& z s z TooManyAccentuatedPlugin.feedc C s d| _ d| _d S )Nr )r2 r9 )r! r" r" r# r' � s z TooManyAccentuatedPlugin.resetr( c C s* | j dk rdS | j| j }|dkr&|S dS )N� g gffffff�?)r2 r9 )r! Zratio_of_accentuationr" r" r# r) � s z TooManyAccentuatedPlugin.ratioN) r* r+ r, r4 r$ r& r' r. r) r" r" r" r# r8 r s r8 c @ sZ e Zd Zdd�dd�Zddd�dd �Zddd�d d �Zdd�d d �Zedd�dd��ZdS )�UnprintablePluginr% )r c C s d| _ d| _d S )Nr )�_unprintable_countr2 )r! r" r" r# r4 � s zUnprintablePlugin.__init__r r )r r c C s dS )NTr" )r! r r" r" r# r$ � s zUnprintablePlugin.eligiblec C s( t |�r| jd7 _| jd7 _d S )Nr )r r= r2 )r! r r" r" r# r& � s zUnprintablePlugin.feedc C s d| _ d S )Nr )r= )r! r" r" r# r' � s zUnprintablePlugin.resetr( c C s | j dkrdS | jd | j S )Nr g r; )r2 r= )r! r" r" r# r) � s zUnprintablePlugin.ratioN) r* r+ r, r4 r$ r& r' r. r) r" r" r" r# r< � s r< c @ sZ e Zd Zdd�dd�Zddd�dd �Zddd�d d �Zdd�d d �Zedd�dd��ZdS )� SuspiciousDuplicateAccentPluginr% )r c C s d| _ d| _d | _d S )Nr )�_successive_countr2 �_last_latin_character)r! r" r" r# r4 � s z(SuspiciousDuplicateAccentPlugin.__init__r r )r r c C s |� � ot|�S )N)r: r )r! r r" r" r# r$ � s z(SuspiciousDuplicateAccentPlugin.eligiblec C st | j d7 _ | jd k rjt|�rjt| j�rj|�� rJ| j�� rJ| jd7 _t|�t| j�krj| jd7 _|| _d S )Nr )r2 r@ r �isupperr? r )r! r r" r" r# r& � s z$SuspiciousDuplicateAccentPlugin.feedc C s d| _ d| _d | _d S )Nr )r? r2 r@ )r! r" r" r# r' � s z%SuspiciousDuplicateAccentPlugin.resetr( c C s | j dkrdS | jd | j S )Nr g r6 )r2 r? )r! r" r" r# r) � s z%SuspiciousDuplicateAccentPlugin.ratioN) r* r+ r, r4 r$ r& r' r. r) r" r" r" r# r> � s r> c @ sZ e Zd Zdd�dd�Zddd�dd �Zddd�d d �Zdd�d d �Zedd�dd��ZdS )�SuspiciousRanger% )r c C s d| _ d| _d | _d S )Nr )�"_suspicious_successive_range_countr2 �_last_printable_seen)r! r" r" r# r4 � s zSuspiciousRange.__init__r r )r r c C s |� � S )N)r5 )r! r r" r" r# r$ � s zSuspiciousRange.eligiblec C sx | j d7 _ |�� s&t|�s&|tkr0d | _d S | jd krD|| _d S t| j�}t|�}t||�rn| jd7 _|| _d S )Nr )r2 �isspacer r rD r � is_suspiciously_successive_rangerC )r! r �unicode_range_a�unicode_range_br" r" r# r& � s zSuspiciousRange.feedc C s d| _ d| _d | _d S )Nr )r2 rC rD )r! r" r" r# r' � s zSuspiciousRange.resetr( c C s" | j dkrdS | jd | j }|S )N� g r6 )r2 rC )r! Z ratio_of_suspicious_range_usager" r" r# r) � s zSuspiciousRange.ratioN) r* r+ r, r4 r$ r& r' r. r) r" r" r" r# rB � s rB c @ sZ e Zd Zdd�dd�Zddd�dd �Zddd�d d �Zdd�d d �Zedd�dd��ZdS )�SuperWeirdWordPluginr% )r c C s@ d| _ d| _d| _d| _d| _d| _d| _d| _d| _d| _ d S )Nr F� ) � _word_count�_bad_word_count�_foreign_long_count�_is_current_word_bad�_foreign_long_watchr2 �_bad_character_count�_buffer�_buffer_accent_count�_buffer_glyph_count)r! r" r" r# r4 � s z SuperWeirdWordPlugin.__init__r r )r r c C s dS )NTr" )r! r r" r" r# r$ s z SuperWeirdWordPlugin.eligiblec C s� |� � r�| j|7 _t|�r,| jd7 _| jdkr�t|�dksJt|�r�t|�dkr�t|�dkr�t|�dkr�t |�dkr�t |�dkr�d| _t|�s�t|�s�t|�s�t |�s�t |�r�| j d7 _ d S | js�d S |� � s�t |�s�t|��r\| j�r\| jd7 _t| j�}| j|7 _|dk�r�| j| dk�r:d| _npt| jd ��r�| jd �� �r�tdd� | jD ��dk�r�| jd7 _d| _n | j dk�r�d| _| jd7 _|d k�r| j�rd d � t| jtd |��D �}d}|�r�t|�| d k�r�d}|�s| jd7 _d| _| j�rB| jd7 _| jt| j�7 _d| _d| _d| _d | _d | _ n6|dk�r�|�� dk�r�t|��r�d| _| j|7 _d S )Nr FT� g �?���c s s | ]}|� � V qd S )N)rA )�.0�_r" r" r# � <genexpr>8 s z,SuperWeirdWordPlugin.feed.<locals>.<genexpr>� c S s g | ]\}}|� � r|�qS r" )rA )rW �c�ir" r" r# � <listcomp>A s z-SuperWeirdWordPlugin.feed.<locals>.<listcomp>r g333333�?rK > rX �<�~�>�=�|�-) r: rR r rS rP r r r r r r rT rE r r rL �lenr2 rO rA �allrN �zip�rangerM rQ r7 r )r! r Z buffer_lengthZcamel_case_dstZprobable_camel_casedr" r" r# r& sv zSuperWeirdWordPlugin.feedc C s4 d| _ d| _d| _d| _d| _d| _d| _d| _d S )NrK Fr )rR rO rP rM rL r2 rQ rN )r! r" r" r# r' _ s zSuperWeirdWordPlugin.resetr( c C s$ | j dkr| jdkrdS | j| j S )N� r g )rL rN rQ r2 )r! r" r" r# r) i s zSuperWeirdWordPlugin.ratioN) r* r+ r, r4 r$ r& r' r. r) r" r" r" r# rJ � s Q rJ c @ s^ e Zd ZdZdd�dd�Zddd�d d �Zddd�d d �Zdd�d d�Zedd�dd��Z dS )�CjkUncommonPluginz< Detect messy CJK text that probably means nothing. r% )r c C s d| _ d| _d S )Nr )r2 �_uncommon_count)r! r" r" r# r4 v s zCjkUncommonPlugin.__init__r r )r r c C s t |�S )N)r )r! r r" r" r# r$ z s zCjkUncommonPlugin.eligiblec C s, | j d7 _ t|�r(| jd7 _d S d S )Nr )r2 r rj )r! r r" r" r# r& } s zCjkUncommonPlugin.feedc C s d| _ d| _d S )Nr )r2 rj )r! r" r" r# r' � s zCjkUncommonPlugin.resetr( c C s. | j dk rdS | j| j }|dkr*|d S dS )Nr; g g �?rh )r2 rj )r! Zuncommon_form_usager" r" r# r) � s zCjkUncommonPlugin.ratioN) r* r+ r, r- r4 r$ r& r' r. r) r" r" r" r# ri q s ri c @ sZ e Zd Zdd�dd�Zddd�dd �Zddd�d d �Zdd�d d �Zedd�dd��ZdS )�ArchaicUpperLowerPluginr% )r c C s. d| _ d| _d| _d| _d| _d | _d| _d S )NFr T)�_buf� _character_count_since_last_sep� _successive_upper_lower_count�#_successive_upper_lower_count_finalr2 �_last_alpha_seen�_current_ascii_only)r! r" r" r# r4 � s z ArchaicUpperLowerPlugin.__init__r r )r r c C s dS )NTr" )r! r r" r" r# r$ � s z ArchaicUpperLowerPlugin.eligiblec C s$ |� � ot|�}|dk}|r�| jdkr�| jdkrV|�� dkrV| jdkrV| j| j7 _d| _d| _d | _d| _| j d7 _ d| _d S | jdkr�|� � dkr�d| _| jd k r�|� � r�| j� � s�|� � r�| j� � r�| jdkr�| jd7 _d| _q�d| _nd| _| j d7 _ | jd7 _|| _d S )NFr �@ r Tr6 ) r: r rm r7 rq ro rn rp rl r2 �isasciirA �islower)r! r Z is_concernedZ chunk_sepr" r" r# r& � s8 z ArchaicUpperLowerPlugin.feedc C s. d| _ d| _d| _d| _d | _d| _d| _d S )Nr FT)r2 rm rn ro rp rl rq )r! r" r" r# r' � s z ArchaicUpperLowerPlugin.resetr( c C s | j dkrdS | j| j S )Nr g )r2 ro )r! r" r" r# r) � s z ArchaicUpperLowerPlugin.ratioN) r* r+ r, r4 r$ r& r' r. r) r" r" r" r# rk � s * rk c @ sZ e Zd Zdd�dd�Zdd�dd�Zddd �d d �Zddd �d d �Zedd�dd��ZdS )�ArabicIsolatedFormPluginr% )r c C s d| _ d| _d S )Nr )r2 �_isolated_form_count)r! r" r" r# r4 � s z!ArabicIsolatedFormPlugin.__init__c C s d| _ d| _d S )Nr )r2 rv )r! r" r" r# r' � s z ArabicIsolatedFormPlugin.resetr r )r r c C s t |�S )N)r )r! r r" r" r# r$ � s z!ArabicIsolatedFormPlugin.eligiblec C s( | j d7 _ t|�r$| jd7 _d S )Nr )r2 r rv )r! r r" r" r# r& � s z ArabicIsolatedFormPlugin.feedr( c C s | j dk rdS | j| j }|S )Nr; g )r2 rv )r! Zisolated_form_usager" r" r# r) � s z ArabicIsolatedFormPlugin.ratioN) r* r+ r, r4 r' r$ r& r. r) r" r" r" r# ru � s ru i )�maxsizez str | Noner )rG rH r c C s� | dks|dkrdS | |kr dS d| kr4d|kr4dS d| ksDd|krHdS d| ksXd|krld| kshd|krldS | � d�|� d� }}x"|D ]}|tkr�q�||kr�dS q�W | dk|dk }}|s�|r�d | ks�d |kr�dS |r�|r�dS d | ks�d |k�r"d | k�sd |k�r dS | d k�s |d k�r"dS d | k�sJd |k�sJ| d k�r�|d k�r�d | k�s^d |k�rbdS d| k�svd|k�rzdS | d k�s�|d k�r�dS dS )za Determine if two Unicode range seen next to each other can be considered as suspicious. NTFZLatinZ EmoticonsZ Combining� )�Hiragana�Katakana�CJKZHangulz Basic Latin)rz ry Z PunctuationZForms)�splitr )rG rH Zkeywords_range_aZkeywords_range_b�elZrange_a_jp_charsZrange_b_jp_charsr" r" r# rF � sR rF i 皙�����?Fr r( )�decoded_sequence�maximum_threshold�debugr c C sb dd� t �� D �}t| �d }d}|dk r0d}n|dkr>d}nd }x|t| d t|��D ]f\}}x |D ]} | �|�rd| �|� qdW |d kr�|| d ks�||d krVtd d � |D ��}||krVP qVW |�rXtd�} | � t d|� d|� d|� �� t| �dk�r.| � t d| dd� � �� | � t d| dd� � �� x(|D ] } | � t | j � d| j � �� �q4W t |d�S )zw Compute a mess ratio given a decoded bytes sequence. The maximum threshold does stop the computation earlier. c S s g | ] }|� �qS r" r" )rW Zmd_classr" r" r# r] O s z mess_ratio.<locals>.<listcomp>r g i � i rr � � r c s s | ] }|j V qd S )N)r) )rW �dtr" r" r# rY e s z mess_ratio.<locals>.<genexpr>Zcharset_normalizerzIMess-detector extended-analysis start. intermediary_mean_mess_ratio_calc=z mean_mess_ratio=z maximum_threshold=� zStarting with: Nz Ending with: i�z: � )r �__subclasses__rd rf rg r$ r& �sumr �logr � __class__r) �round) r r� r� Z detectors�lengthZmean_mess_ratioZ!intermediary_mean_mess_ratio_calcr �index�detector�loggerr� r" r" r# � mess_ratioF s8 r� N)r~ F))� __future__r � functoolsr �loggingr Zconstantr r r �utilsr r r r r r r r r r r r r r r r r r r r/ r8 r< r> rB rJ ri rk ru rF r� r" r" r" r# �<module> s" P"/%1v#L J