D7net
Home
Console
Upload
information
Create File
Create Folder
About
Tools
:
/
opt
/
alt
/
python37
/
lib
/
python3.7
/
site-packages
/
charset_normalizer
/
__pycache__
/
Filename :
api.cpython-37.pyc
back
Copy
B �B�h�X � @ sR d dl mZ d dlZd dlmZ d dlmZ ddlmZm Z m Z m Z ddl m Z mZmZmZ ddlmZ dd lmZmZ dd lmZmZmZmZmZm Z m Z e� d �Z e� � Z!e!�"e�#d �� d(dddddddddddd� dd�Z$d)d ddddddddddd � d d �Z%d*d ddddddddddd!� d"d#�Z&d+d$ddddddddddd%� d&d'�Z'dS ),� )� annotationsN)�PathLike)�BinaryIO� )�coherence_ratio�encoding_languages�mb_encoding_languages�merge_coherence_ratios)�IANA_SUPPORTED�TOO_BIG_SEQUENCE�TOO_SMALL_SEQUENCE�TRACE)� mess_ratio)� CharsetMatch�CharsetMatches)�any_specified_encoding�cut_sequence_chunks� iana_name�identify_sig_or_bom� is_cp_similar�is_multi_byte_encoding�should_strip_sig_or_bomZcharset_normalizerz)%(asctime)s | %(levelname)s | %(message)s� � 皙�����?TF皙�����?zbytes | bytearray�int�floatzlist[str] | None�boolr ) � sequences�steps� chunk_size� threshold� cp_isolation� cp_exclusion�preemptive_behaviour�explain�language_threshold�enable_fallback�returnc 2 C s� t | ttf�s td�t| ����|r>tj} t�t � t� t � t | �} | dkr�t� d� |rvt�t � t� | prtj� tt| dddg d�g�S |dk r�t�t d d �|�� d d � |D �}ng }|dk r�t�t d d �|�� dd � |D �}ng }| || k�rt�t d||| � d}| }|dk�r:| | |k �r:t| | �}t | �tk } t | �tk} | �rlt�t d�| �� n| �r�t�t d�| �� g }|�r�t| �nd}|dk �r�|�|� t�t d|� t� }g }g }d}d}d}t� }t� }t| �\}}|dk �r|�|� t�t dt |�|� |�d� d|k�r4|�d� �xz|t D �]l}|�rZ||k�rZ�q@|�rn||k�rn�q@||k�r|�q@|� |� d}||k} | �o�t |�} |dk�r�| �s�t�t d|� �q@|dk�r�| �s�t�t d|� �q@y t |�} W n, t t!fk �r t�t d|� �w@Y nX yr| �rh| dk�rht"| dk�rL| dtd�� n| t |�td�� |d � n&t"| dk�rx| n| t |�d� |d �}W nV t#t$fk �r� } z2t | t$��s�t�t d |t"| �� |�|� �w@W dd} ~ X Y nX d} x |D ]}!t%||!��r�d } P �q�W | �r(t�t d ||!� �q@t&| �s4dnt |�| t| | ��}"| �od|dk �odt |�| k }#|#�rzt�t d |� tt |"�d! �}$t'|$d"�}$d}%d}&g }'g }(y�x�t(| ||"|| | || |� D ]x})|'�|)� |(�t)|)||d k�o�dt |� k�o�d"kn �� |(d# |k�r|%d7 }%|%|$k�s4| �r�| dk�r�P �q�W W n@ t#k �r~ } z t�t d$|t"| �� |$}%d }&W dd} ~ X Y nX |&�s�| �r�| �s�y | td%�d� j*|d&d'� W nF t#k �r� } z&t�t d(|t"| �� |�|� �w@W dd} ~ X Y nX |(�rt+|(�t |(� nd}*|*|k�s&|%|$k�r�|�|� t�t d)||%t,|*d* d+d,�� | �r@|dd|d-d.gk�r@|&�s@t| ||| g ||d/�}+||k�r�|+}n|dk�r�|+}n|+}�q@t�t d0|t,|*d* d+d,�� | �s�t-|�},nt.|�},|,�r�t�t d1�|t"|,��� g }-|dk�rBx4|'D ],})t/|)||,�r,d2�|,�nd�}.|-�|.� �qW t0|-�}/|/�rdt�t d3�|/|�� t| ||*| |/| dk�s�||ddgk�r�|nd|d/�}0|�|0� ||ddgk�r|*d4k �r|*dk�r�t� d5|0j1� |�r�t�t � t� | � t|0g�S |�|0� t |��rn|dk�s ||k�rnd|k�rnd|k�rn|�2� }1t� d5|1j1� |�rdt�t � t� | � t|1g�S ||k�r@t� d6|� |�r�t�t � t� | � t|| g�S �q@W t |�dk� rb|�s�|�s�|�r�t�t d7� |�r�t� d8|j1� |�|� nd|� r|dk� s2|� r(|� r(|j3|j3k� s2|dk � rHt� d9� |�|� n|� rbt� d:� |�|� |� r�t� d;|�2� j1t |�d � n t� d<� |� r�t�t � t� | � |S )=af Given a raw bytes sequence, return the best possibles charset usable to render str objects. If there is no results, it is a strong indicator that the source is binary/not text. By default, the process will extract 5 blocks of 512o each to assess the mess and coherence of a given sequence. And will give up a particular code page after 20% of measured mess. Those criteria are customizable at will. The preemptive behavior DOES NOT replace the traditional detection workflow, it prioritize a particular code page but never take it for granted. Can improve the performance. You may want to focus your attention to some code page or/and not others, use cp_isolation and cp_exclusion for that purpose. This function will strip the SIG in the payload/sequence every time except on UTF-16, UTF-32. By default the library does not setup any handler other than the NullHandler, if you choose to set the 'explain' toggle to True it will alter the logger configuration to add a StreamHandler that is suitable for debugging. Custom logging format and handler can be set manually. z3Expected object of type bytes or bytearray, got: {}r z<Encoding detection on empty bytes, assuming utf_8 intention.�utf_8g F� Nz`cp_isolation is set. use this flag for debugging purpose. limited list of encoding allowed : %s.z, c S s g | ]}t |d ��qS )F)r )�.0�cp� r. �G/opt/alt/python37/lib/python3.7/site-packages/charset_normalizer/api.py� <listcomp>[ s z from_bytes.<locals>.<listcomp>zacp_exclusion is set. use this flag for debugging purpose. limited list of encoding excluded : %s.c S s g | ]}t |d ��qS )F)r )r, r- r. r. r/ r0 f s z^override steps (%i) and chunk_size (%i) as content does not fit (%i byte(s) given) parameters.r z>Trying to detect encoding from a tiny portion of ({}) byte(s).zIUsing lazy str decoding because the payload is quite large, ({}) byte(s).z@Detected declarative mark in sequence. Priority +1 given for %s.zIDetected a SIG or BOM mark on first %i byte(s). Priority +1 given for %s.�ascii> �utf_16�utf_32z\Encoding %s won't be tested as-is because it require a BOM. Will try some sub-encoder LE/BE.> �utf_7zREncoding %s won't be tested as-is because detection is unreliable without BOM/SIG.z2Encoding %s does not provide an IncrementalDecoderg �� A)�encodingz9Code page %s does not fit given bytes sequence at ALL. %sTzW%s is deemed too similar to code page %s and was consider unsuited already. Continuing!zpCode page %s is a multi byte encoding table and it appear that at least one character was encoded using n-bytes.� � ���zaLazyStr Loading: After MD chunk decode, code page %s does not fit given bytes sequence at ALL. %sg j�@�strict)�errorsz^LazyStr Loading: After final lookup, code page %s does not fit given bytes sequence at ALL. %szc%s was excluded because of initial chaos probing. Gave up %i time(s). Computed mean chaos is %f %%.�d � )�ndigitsr2 r3 )Zpreemptive_declarationz=%s passed initial chaos probing. Mean measured chaos is %f %%z&{} should target any language(s) of {}�,z We detected language {} using {}g�������?z.Encoding detection: %s is most likely the one.zoEncoding detection: %s is most likely the one as we detected a BOM or SIG within the beginning of the sequence.zONothing got out of the detection process. Using ASCII/UTF-8/Specified fallback.z7Encoding detection: %s will be used as a fallback matchz:Encoding detection: utf_8 will be used as a fallback matchz:Encoding detection: ascii will be used as a fallback matchz]Encoding detection: Found %s as plausible (best-candidate) for content. With %i alternatives.z=Encoding detection: Unable to determine any suitable charset.)4� isinstance� bytearray�bytes� TypeError�format�type�logger�level� addHandler�explain_handler�setLevelr �len�debug� removeHandler�logging�WARNINGr r �log�joinr r r r �append�setr r �addr r �ModuleNotFoundError� ImportError�str�UnicodeDecodeError� LookupErrorr �range�maxr r �decode�sum�roundr r r r r5 �best� fingerprint)2r r r! r" r# r$ r% r& r' r( Zprevious_logger_level�lengthZis_too_small_sequenceZis_too_large_sequenceZprioritized_encodingsZspecified_encodingZtestedZtested_but_hard_failureZtested_but_soft_failureZfallback_asciiZ fallback_u8Zfallback_specified�resultsZearly_stop_resultsZ sig_encodingZ sig_payloadZ encoding_ianaZdecoded_payloadZbom_or_sig_availableZstrip_sig_or_bomZis_multi_byte_decoder�eZsimilar_soft_failure_testZencoding_soft_failedZr_Zmulti_byte_bonusZmax_chunk_gave_upZearly_stop_countZlazy_str_hard_failureZ md_chunksZ md_ratios�chunkZmean_mess_ratioZfallback_entryZtarget_languagesZ cd_ratiosZchunk_languagesZcd_ratios_mergedZ current_matchZprobable_resultr. r. r/ � from_bytes! s� , rd r ) �fpr r! r" r# r$ r% r& r' r( r) c C s t | �� ||||||||| � S )z� Same thing than the function from_bytes but using a file pointer that is already ready. Will not close the file pointer. )rd �read) re r r! r" r# r$ r% r& r' r( r. r. r/ �from_fp! s rg zstr | bytes | PathLike) �pathr r! r" r# r$ r% r& r' r( r) c C s0 t | d�� } t| ||||||||| � S Q R X dS )z� Same thing than the function from_bytes but with one extra step. Opening and reading given file path in binary mode. Can raise IOError. �rbN)�openrg ) rh r r! r" r# r$ r% r& r' r( re r. r. r/ � from_path? s rk z!PathLike | str | BinaryIO | bytes) �fp_or_path_or_payloadr r! r" r# r$ r% r& r' r( r) c C sz t | ttf�r,t| ||||||||| d� } nHt | ttf�rXt| ||||||||| d� } n t| ||||||||| d� } | S )a) Detect if the given input (file, bytes, or path) points to a binary file. aka. not a string. Based on the same main heuristic algorithms and default kwargs at the sole exception that fallbacks match are disabled to be stricter around ASCII-compatible but unlikely to be a string. ) r r! r" r# r$ r% r& r' r( )r? rV r rk rA r@ rd rg ) rl r r! r" r# r$ r% r& r' r( Zguessesr. r. r/ � is_binary^ sN rm ) r r r NNTFr T) r r r NNTFr T) r r r NNTFr T) r r r NNTFr F)(� __future__r rM �osr �typingr Zcdr r r r Zconstantr r r r Zmdr �modelsr r �utilsr r r r r r r � getLoggerrE � StreamHandlerrH � setFormatter� Formatterrd rg rk rm r. r. r. r/ �<module> sf $ $ { $ $