|
Server : Apache System : Linux iad1-shared-b8-43 6.6.49-grsec-jammy+ #10 SMP Thu Sep 12 23:23:08 UTC 2024 x86_64 User : dh_edsupp ( 6597262) PHP Version : 8.2.26 Disable Function : NONE Directory : /lib/python3/dist-packages/S3/__pycache__/ |
Upload File : |
o
�Qa>5 � @ s� d dl mZ d dlZd dlmZmZmZmZ ddlm Z ddl
m Z ddl m
Z
mZmZmZ ddlmZmZ d ZG d
d � d e�ZdS ) � )�absolute_importN)�debug�info�warning�error� )�ParameterError)�S3UriS3)�getTextFromXml�getTreeFromXml�s3_quote�
parseNodes)�
formatSize�calculateChecksumi c @ st e Zd ZdZdZdZdZ ddd�Zdd � Zd
d � Z d d
� Z
ddd�Z ddd�Z d dd�Z
dd� Zdd� ZdS ) �MultiPartUploadz<Supports MultiPartUpload and MultiPartUpload(Copy) operation� i i P Nc C s� || _ d | _d | _|| _|| _i | _|pi | _t|t�r-|| _|s%t d��| j j
j t }n
|| _| j j
j
t }|| _| �� | _d S )Nz8Source size is missing for MultipartUploadCopy operation)�s3� file_stream�src_uri�src_size�dst_uri�parts�headers_baseline�
isinstancer r �config� multipart_copy_chunk_size_mb�SIZE_1MB�multipart_chunk_size_mb�
chunk_size�initiate_multipart_upload� upload_id)�selfr �srcr r r �c_size� r$ �./usr/lib/python3/dist-packages/S3/MultiPart.py�__init__ s
zMultiPartUpload.__init__c C sV | j �||�}t� }|D ] }z|d |d d�|t|d �< W q ty( Y q w |S )N�ETag�Size)�checksum�size�
PartNumber)r �list_multipart�dict�int�KeyError)r! �urir � part_listr �elemr$ r$ r% �get_parts_information2 s � �z%MultiPartUpload.get_parts_informationc C s� d}| j �|�}|D ]5}z*|d }|d }td||�� f � ||�� kr5|r3td|tjd |f ��|}W q
ty? Y q
w |S )N� �UploadId�Keyzmp_path: %s, object: %sz�More than one UploadId for URI %s. Disable multipart upload, or use
%s multipart %s
to list the Ids, then pass a unique --upload-id into the put command.r )r �
get_multipartr �object�
ValueError�sys�argvr/ )r! r0 r �multipart_list�mpupload� mp_upload_id�mp_pathr$ r$ r% �get_unique_upload_idA s( ��� �z$MultiPartUpload.get_unique_upload_idc C s� | j jjr | j jj| _n| j jjr| �| j�| _nd| _| js=| j jd| j| jddid�}| j �|�}|d }t |d�| _| jS )z�
Begin a multipart upload
http://docs.amazonwebservices.com/AmazonS3/latest/API/index.html?mpUploadInitiate.html
r4 � OBJECT_POST�uploadsN�r0 �headers�
uri_params�datar5 )
r r r � put_continuer@ r �create_requestr � send_requestr
)r! �request�responserF r$ r$ r% r V s
� z)MultiPartUpload.initiate_multipart_uploadr4 c C sD | j std��i }| jr| j�� }n| jj}| jjjr#| � | j
| j �}|r)d| }|| j
�� d�}d}| j r�| j }| j | j | j | j oEd }t
d||f � |dkr�| j |d }t| j | | j �} || 8 }d||dt| d d
� |f |d < z | jr�| j||| ||�|�d � n
| j||| ||�|�d � W n td
||tjd | j
| j tjd | j f � � |d7 }|dksSt
d|d � dS t
d| � | j�| j �}
d}t|
�} d|dt| d d
� |f |d < |
s�n0z| j||| ||
|�|�d � W n td||tjd | j
| j tjd | j f � � |d7 }q�t
d|d � dS )z�
Execute a full multipart upload on a file
Returns the seq/etag dict
TODO use num_processes to thread it
zAAttempting to use a multipart upload that has not been initiated.� )�source� destinationr z#MultiPart: Uploading %s in %d partsr z[part %d of %d, %s]%sz%d%sBT)�human_readable�extra)�
remote_statusz�
Upload of '%s' part %d failed. Use
%s abortmp %s %s
to abort the upload, or
%s --upload-id %s put ...
to continue the upload.z$MultiPart: Upload finished: %d partsNz MultiPart: Uploading from %sz[part %d of -, %s]%szw
Upload of '%s' part %d failed. Use
%s abortmp %s %s
to abort, or
%s --upload-id %s put ...
to continue the upload.)r r r r0 r � stream_namer r rG r3 r r r r �minr � upload_part�get� copy_partr r: r; �read�len) r! � extra_label�remote_statuses�filename�labels�seq� size_left�nr_parts�offset�current_chunk_size�bufferr$ r$ r% �upload_all_partsl s�
��
�
�
�
������� �
�
����z MultiPartUpload.upload_all_partsc
C s t d|| j|f � |durZt|d �|krKt|| j||| jjj�}|d �d�}||kr?t d| j
|f � |d | j |< dS t d||| j
|f � nt dt|d �|| j
|f � d t |�i} d
| | jd �}
| jj
d | j
| |
d
�} | jj| | j||||d�} | d �dd��d�| j |< | S )z�
Upload a file chunk
http://docs.amazonwebservices.com/AmazonS3/latest/API/index.html?mpUploadUploadPart.html
z"Uploading part %i of %r (%s bytes)Nr* r) z"'z:MultiPart: size and md5sum match for %s part %d, skipping.zJMultiPart: checksum (%s vs %s) does not match for %s part %d, reuploading.zFMultiPart: size (%d vs %d) does not match for %s part %d, reuploading.�content-length�%s��
partNumber�uploadId�
OBJECT_PUTrC )r` r rD �etagr4 )r r r. r r r r �
send_chunk�stripr r r �strrH � send_filerU )
r! r] r` r r\ rb rQ r) �remote_checksumrD �query_string_paramsrJ rK r$ r$ r% rT � sJ
�
�� �
�� � �� zMultiPartUpload.upload_partc
C s� t d|| j|f � dtd| j�� | j�� f ddd�i}d||| d f |d < d
| | jd �}| jjd | j||d
�}d|d< | j� |||�} t
| d d�pPd| j |< | S )z�
Copy a remote file chunk
http://docs.amazonwebservices.com/AmazonS3/latest/API/index.html?mpUploadUploadPart.html
http://docs.amazonwebservices.com/AmazonS3/latest/API/mpUploadUploadPartCopy.html
z Copying part %i of %r (%s bytes)zx-amz-copy-sourcez/%s/%sFT)�quote_backslashes�unicode_outputz bytes=%d-%dr zx-amz-copy-source-rangere rf ri rC z remote copy�actionrF r' r4 ) r r r r �bucketr8 r rH r �send_request_with_progressr
r )
r! r] r` r r\ rQ rD rp rJ rK r$ r$ r% rV � s2
� ��� �� �
�zMultiPartUpload.copy_partc C s� t d| j � g }d}| j�� D ]
\}}|�|||f � qdd�|� }dtt|��i}| jj d| j
||d| jid�}| j� |�}|S ) z�
Finish a multipart upload
http://docs.amazonwebservices.com/AmazonS3/latest/API/index.html?mpUploadComplete.html
z MultiPart: Completing upload: %sz7<Part><PartNumber>%i</PartNumber><ETag>%s</ETag></Part>z5<CompleteMultipartUpload>%s</CompleteMultipartUpload>r4 rd rA rh )r0 rD �bodyrE ) r r r �items�append�joinrm rX r rH r rI ) r! � parts_xml�part_xmlr] rj rv rD rJ rK r$ r$ r% �complete_multipart_upload# s �
� z)MultiPartUpload.complete_multipart_uploadc C s t d| j � d}|S )z�
Abort multipart upload
http://docs.amazonwebservices.com/AmazonS3/latest/API/index.html?mpUploadAbort.html
z MultiPart: Aborting upload: %sN)r r )r! rK r$ r$ r% � abort_upload9 s z MultiPartUpload.abort_upload)NN)r4 )r4 N)N)�__name__�
__module__� __qualname__�__doc__�MIN_CHUNK_SIZE_MB�MAX_CHUNK_SIZE_MB�
MAX_FILE_SIZEr& r3 r@ r rc rT rV r| r} r$ r$ r$ r% r s$
�
`
�
). r )�
__future__r r: �loggingr r r r �
Exceptionsr �S3Urir � BaseUtilsr
r r r
�Utilsr r r r8 r r$ r$ r$ r% �<module> s