D7net
Home
Console
Upload
information
Create File
Create Folder
About
Tools
:
/
opt
/
alt
/
python37
/
lib
/
python3.7
/
site-packages
/
charset_normalizer
/
__pycache__
/
Filename :
utils.cpython-37.pyc
back
Copy
B �B�h�/ � @ sL d dl mZ d dlZd dlZd dlZd dlmZ d dlmZ d dl m Z d dl m Z d dl mZ d dlmZ d d lmZmZmZmZmZmZmZ e ed �d d d�dd��Ze ed �d d d�dd��Ze ed �d dd�dd��Ze ed �d d d�dd��Z e ed �d d d�dd��Z e ed �d d d�dd��Z e ed �d d d�d d ��Z e ed �d d d�d d ��Z e ed �d d d�d d!��Z!e ed �d d d�d"d#��Z"e ed �d d d�d$d%��Z#e ed �d d d�d&d'��Z$e ed �d d d�d(d)��Z%e ed �d d d�d*d+��Z&e ed �d d d�d,d-��Z'e ed �d d d�d.d/��Z(e ed �d d d�d0d1��Z)e e*e�d �d d d2�d3d4��Z+e ed �d d d�d5d6��Z,d]d8d9dd:�d;d<�Z-e d=d �d d d>�d?d@��Z.d8dAdB�dCdD�Z/d d dE�dFdG�Z0d^d d d dI�dJdK�Z1d d dLdM�dNdO�Z2d d d dM�dPdQ�Z3dRej4dSfd d9d dTdU�dVdW�Z5d_d8d dXd9d d d8d ddYdZ� d[d\�Z6dS )`� )� annotationsN)�IncrementalDecoder)�aliases)� lru_cache)�findall)� Generator)�MultibyteIncrementalDecoder� )�ENCODING_MARKS�IANA_SUPPORTED_SIMILAR� RE_POSSIBLE_ENCODING_INDICATION�UNICODE_RANGES_COMBINED� UNICODE_SECONDARY_RANGE_KEYWORD�UTF8_MAXIMAL_ALLOCATION�COMMON_CJK_CHARACTERS)�maxsize�str�bool)� character�returnc C sd yt �| �}W n tk r" dS X d|kpbd|kpbd|kpbd|kpbd|kpbd|kpbd|kpbd |kS ) NFz WITH GRAVEz WITH ACUTEz WITH CEDILLAzWITH DIAERESISzWITH CIRCUMFLEXz WITH TILDEz WITH MACRONzWITH RING ABOVE)� unicodedata�name� ValueError)r � description� r �I/opt/alt/python37/lib/python3.7/site-packages/charset_normalizer/utils.py�is_accentuated s r c C s. t �| �}|s| S |�d�}tt|d d��S )N� r � )r � decomposition�split�chr�int)r Z decomposed�codesr r r � remove_accent- s r$ z str | Nonec C s. t | �}x t�� D ]\}}||kr|S qW dS )zK Retrieve the Unicode range official name from a single character. N)�ordr �items)r Z character_ord� range_nameZ ord_ranger r r � unicode_range8 s r( c C s, yt �| �}W n tk r" dS X d|kS )NFZLATIN)r r r )r r r r r �is_latinF s r) c C s2 t �| �}d|krdS t| �}|d kr*dS d|kS )N�PTFZ Punctuation)r �categoryr( )r �character_category�character_ranger r r �is_punctuationO s r. c C sB t �| �}d|ksd|kr dS t| �}|d kr2dS d|ko@|dkS )N�S�NTFZFormsZLo)r r+ r( )r r, r- r r r � is_symbol^ s r1 c C s$ t | �}|d krdS d|kp"d|kS )NFZ EmoticonsZ Pictographs)r( )r r- r r r � is_emoticonm s r2 c C s. | � � s| dkrdS t�| �}d|kp,|dkS )N> �>�<� ｜�+T�Z> �Pc�Po�Pd)�isspacer r+ )r r, r r r � is_separatorw s r< c C s | � � | �� kS )N)�islower�isupper)r r r r �is_case_variable� s r? c C s, yt �| �}W n tk r" dS X d|kS )NF�CJK)r r r )r �character_namer r r �is_cjk� s rB c C s, yt �| �}W n tk r" dS X d|kS )NFZHIRAGANA)r r r )r rA r r r � is_hiragana� s rC c C s, yt �| �}W n tk r" dS X d|kS )NFZKATAKANA)r r r )r rA r r r � is_katakana� s rD c C s, yt �| �}W n tk r" dS X d|kS )NFZHANGUL)r r r )r rA r r r � is_hangul� s rE c C s, yt �| �}W n tk r" dS X d|kS )NFZTHAI)r r r )r rA r r r �is_thai� s rF c C s, yt �| �}W n tk r" dS X d|kS )NF�ARABIC)r r r )r rA r r r � is_arabic� s rH c C s4 yt �| �}W n tk r" dS X d|ko2d|kS )NFrG z ISOLATED FORM)r r r )r rA r r r �is_arabic_isolated_form� s rI c C s | t kS )N)r )r r r r �is_cjk_uncommon� s rJ )r' r c s t � fdd�tD ��S )Nc 3 s | ]}|� kV qd S )Nr )�.0�keyword)r' r r � <genexpr>� s z-is_unicode_range_secondary.<locals>.<genexpr>)�anyr )r' r )r' r �is_unicode_range_secondary� s rO c C s( | � � dko&| �� dko&| dko&| dkS )NF�u )r; � isprintable)r r r r �is_unprintable� s rR � �bytesr" )�sequence� search_zoner c C s� t | t�st�t| �}tt| dt||�� jddd��}t|�dkrHdS xJ|D ]B}|�� � dd�}x,t � � D ] \}}||kr�|S ||krl|S qlW qNW dS )zW Extract using ASCII-only decoder any specified encoding in the first n-bytes. N�ascii�ignore)�errorsr �-�_) � isinstancerT � TypeError�lenr r �min�decode�lower�replacer r& )rU rV Zseq_len�resultsZspecified_encoding�encoding_alias� encoding_ianar r r �any_specified_encoding� s rf � )r r c C s | dkp t t�d| � ��jt�S )zQ Verify is a specific encoding is a multi byte one based on it IANA name > �utf_7� utf_8_sig� utf_16_be� utf_16_le� utf_32_be�utf_8� utf_32_le�utf_16�utf_32z encodings.)� issubclass� importlib� import_moduler r )r r r r �is_multi_byte_encoding s rt ztuple[str | None, bytes])rU r c C sJ xDt D ]<}t | }t|t�r"|g}x |D ]}| �|�r(||fS q(W qW dS )z9 Identify and extract SIG/BOM in given sequence. )N� )r r\ rT � startswith)rU � iana_encodingZmarks�markr r r �identify_sig_or_bom s ry )rw r c C s | dkS )N> ro rp r )rw r r r �should_strip_sig_or_bom( s rz T)�cp_name�strictr c C sN | � � �dd�} x$t�� D ]\}}| ||gkr|S qW |rJtd| � d���| S )zIReturns the Python normalized encoding name (Not the IANA official name).rZ r[ z Unable to retrieve IANA for '�')ra rb r r& r )r{ r| rd re r r r � iana_name, s r~ �float)� iana_name_a� iana_name_br c C s� t | �st |�rdS t�d| � ��j}t�d|� ��j}|dd�}|dd�}d}x6td�D ]*}t|g�}|�|�|�|�krZ|d7 }qZW |d S ) Ng z encodings.rX )rY r � r � )rt rr rs r �rangerT r` ) r� r� Z decoder_aZ decoder_bZid_aZid_bZcharacter_match_count�iZ to_be_decodedr r r � cp_similarity= s r� c C s | t ko|t | kS )z� Determine if two code page are at least 80% similar. IANA_SUPPORTED_SIMILAR dict was generated using the function cp_similarity. )r )r� r� r r r � is_cp_similarQ s r� Zcharset_normalizerz)%(asctime)s | %(levelname)s | %(message)s�None)r �level� format_stringr c C s: t �| �}|�|� t �� }|�t �|�� |�|� d S )N)�logging� getLogger�setLevel� StreamHandler� setFormatter� Formatter� addHandler)r r� r� �logger�handlerr r r �set_logging_handler\ s r� r� zGenerator[str, None, None]) � sequencesre �offsets� chunk_size�bom_or_sig_available�strip_sig_or_bom� sig_payload�is_multi_byte_decoder�decoded_payloadr c c s0 |r8|dkr8x(|D ] } || | | � } | s,P | V qW n�x�|D ]�} | | } | t | �d kr\q>| | | | � } |r�|dkr�|| } | j||r�dndd�} |�r"| dk�r"t|d�} |�r"| d | � |k�r"xXt| | d d �D ]D}| || � } |r�|dkr�|| } | j|dd�} | d | � |kr�P q�W | V q>W d S ) NF� rX r| )rY r r � ���)r^ r` r_ r� )r� re r� r� r� r� r� r� r� r� �chunkZ chunk_endZ cut_sequenceZchunk_partial_size_chk�jr r r �cut_sequence_chunksi s8 r� )rS )T)N)7� __future__r rr r� r �codecsr Zencodings.aliasesr � functoolsr �rer �typingr Z_multibytecodecr Zconstantr r r r r r r r r$ r( r) r. r1 r2 r< r? rB rC rD rE rF rH rI rJ r^ rO rR rf rt ry rz r~ r� r� �INFOr� r� r r r r �<module> sP $