|
Server : Apache System : Linux iad1-shared-b8-43 6.6.49-grsec-jammy+ #10 SMP Thu Sep 12 23:23:08 UTC 2024 x86_64 User : dh_edsupp ( 6597262) PHP Version : 8.2.26 Disable Function : NONE Directory : /lib/python3/dist-packages/s3transfer/__pycache__/ |
Upload File : |
o
�y�`�s � @ s d dl Z d dlmZ d dlmZmZ d dlmZ d dlm Z d dlm
Z
d dlm Z d dlm Z d d l
mZ d d
l
mZ d d l
mZmZ G d d
� d
e�ZG dd� de�ZG dd� de�ZG dd� de�ZG dd� de�ZG dd� de�ZG dd� de
�ZG dd� de �ZG d d � d e �ZdS ) � N)�six)�seekable�readable)�IN_MEMORY_UPLOAD_TAG)�Task)�SubmissionTask)�CreateMultipartUploadTask)�CompleteMultipartUploadTask)�
get_callbacks)�get_filtered_dict)�DeferredOpenFile�ChunksizeAdjusterc @ s. e Zd Zd dd�Zdd� Zdd� Zdd � Zd
S ) �AggregatedProgressCallback� c C s || _ || _d| _dS )a� Aggregates progress updates for every provided progress callback
:type callbacks: A list of functions that accepts bytes_transferred
as a single argument
:param callbacks: The callbacks to invoke when threshold is reached
:type threshold: int
:param threshold: The progress threshold in which to take the
aggregated progress and invoke the progress callback with that
aggregated progress total
r N)�
_callbacks�
_threshold� _bytes_seen)�self� callbacks� threshold� r �3/usr/lib/python3/dist-packages/s3transfer/upload.py�__init__ s
z#AggregatedProgressCallback.__init__c C s* | j |7 _ | j | jkr| �� d S d S �N)r r �_trigger_callbacks)r �bytes_transferredr r r �__call__- s �z#AggregatedProgressCallback.__call__c C s | j dkr | �� dS dS )z@Flushes out any progress that has not been sent to its callbacksr N)r r �r r r r �flush2 s
�z AggregatedProgressCallback.flushc C s" | j D ]}|| jd� qd| _d S )N)r r )r r )r �callbackr r r r 7 s
z-AggregatedProgressCallback._trigger_callbacksN)r )�__name__�
__module__� __qualname__r r r r r r r r r s
r c @ sL e Zd ZdZdd� Zddd�Zddd �Zd
d � Zd d
� Zdd� Z dd� Z
dS )�InterruptReadera� Wrapper that can interrupt reading using an error
It uses a transfer coordinator to propagate an error if it notices
that a read is being made while the file is being read from.
:type fileobj: file-like obj
:param fileobj: The file-like object to read from
:type transfer_coordinator: s3transfer.futures.TransferCoordinator
:param transfer_coordinator: The transfer coordinator to use if the
reader needs to be interrupted.
c C s || _ || _d S r )�_fileobj�_transfer_coordinator)r �fileobj�transfer_coordinatorr r r r J s
zInterruptReader.__init__Nc C s | j jr| j j�| j�|�S r )r% � exceptionr$ �read)r �amountr r r r) N s zInterruptReader.readr c C s | j �||� d S r )r$ �seek)r �where�whencer r r r+ X s zInterruptReader.seekc C s
| j �� S r )r$ �tellr r r r r. [ s
zInterruptReader.tellc C s | j �� d S r )r$ �closer r r r r/ ^ � zInterruptReader.closec C s | S r r r r r r � __enter__a � zInterruptReader.__enter__c O s | � � d S r )r/ )r �args�kwargsr r r �__exit__d � zInterruptReader.__exit__r )r ) r r! r" �__doc__r r) r+ r. r/ r1 r5 r r r r r# = s
r# c @ sf e Zd ZdZddd�Zedd� �Zdd� Zd d
� Zd d � Z d
d� Z
dd� Z dd� Z dd� Z
dd� ZdS )�UploadInputManageraJ Base manager class for handling various types of files for uploads
This class is typically used for the UploadSubmissionTask class to help
determine the following:
* How to determine the size of the file
* How to determine if a multipart upload is required
* How to retrieve the body for a PutObject
* How to retrieve the bodies for a set of UploadParts
The answers/implementations differ for the various types of file inputs
that may be accepted. All implementations must subclass and override
public methods from this class.
Nc C s || _ || _|| _d S r )�_osutilr% �_bandwidth_limiter�r �osutilr' �bandwidth_limiterr r r r w s
zUploadInputManager.__init__c C � t d��)a Determines if the source for the upload is compatible with manager
:param upload_source: The source for which the upload will pull data
from.
:returns: True if the manager can handle the type of source specified
otherwise returns False.
z must implement _is_compatible()��NotImplementedError��cls�
upload_sourcer r r �
is_compatible| s
z UploadInputManager.is_compatiblec C r> )a� Whether the body it provides are stored in-memory
:type operation_name: str
:param operation_name: The name of the client operation that the body
is being used for. Valid operation_names are ``put_object`` and
``upload_part``.
:rtype: boolean
:returns: True if the body returned by the manager will be stored in
memory. False if the manager will not directly store the body in
memory.
z%must implement store_body_in_memory())�NotImplemented�r �operation_namer r r �stores_body_in_memory� �
z(UploadInputManager.stores_body_in_memoryc C r> )z�Provides the transfer size of an upload
:type transfer_future: s3transfer.futures.TransferFuture
:param transfer_future: The future associated with upload request
z&must implement provide_transfer_size()r? �r �transfer_futurer r r �provide_transfer_size� s z(UploadInputManager.provide_transfer_sizec C r> )a� Determines where a multipart upload is required
:type transfer_future: s3transfer.futures.TransferFuture
:param transfer_future: The future associated with upload request
:type config: s3transfer.manager.TransferConfig
:param config: The config associated to the transfer manager
:rtype: boolean
:returns: True, if the upload should be multipart based on
configuartion and size. False, otherwise.
z*must implement requires_multipart_upload()r? �r rK �configr r r �requires_multipart_upload� rI z,UploadInputManager.requires_multipart_uploadc C r> )a� Returns the body to use for PutObject
:type transfer_future: s3transfer.futures.TransferFuture
:param transfer_future: The future associated with upload request
:type config: s3transfer.manager.TransferConfig
:param config: The config associated to the transfer manager
:rtype: s3transfer.utils.ReadFileChunk
:returns: A ReadFileChunk including all progress callbacks
associated with the transfer future.
z$must implement get_put_object_body()r? rJ r r r �get_put_object_body� rI z&UploadInputManager.get_put_object_bodyc C r> )a Yields the part number and body to use for each UploadPart
:type transfer_future: s3transfer.futures.TransferFuture
:param transfer_future: The future associated with upload request
:type chunksize: int
:param chunksize: The chunksize to use for this upload.
:rtype: int, s3transfer.utils.ReadFileChunk
:returns: Yields the part number and the ReadFileChunk including all
progress callbacks associated with the transfer future for that
specific yielded part.
z)must implement yield_upload_part_bodies()r? )r rK � chunksizer r r �yield_upload_part_bodies� s z+UploadInputManager.yield_upload_part_bodiesc C s* t || j�}| jr| jj|| jdd�}|S )NF)�enabled)r# r% r: �get_bandwith_limited_stream)r r& r r r �
_wrap_fileobj� s �z UploadInputManager._wrap_fileobjc C s t |d�}|r t|�gS g S )N�progress)r
r )r rK r r r r �_get_progress_callbacks� s
z*UploadInputManager._get_progress_callbacksc C s dd� |D �S )Nc S s g | ]}|j �qS r )r )�.0r r r r �
<listcomp>� s z;UploadInputManager._get_close_callbacks.<locals>.<listcomp>r )r � aggregated_progress_callbacksr r r �_get_close_callbacks� r0 z'UploadInputManager._get_close_callbacksr )r r! r" r7 r � classmethodrD rH rL rO rP rR rU rW r[ r r r r r8 h s
r8 c @ sd e Zd ZdZedd� �Zdd� Zdd� Zdd � Zd
d � Z d d
� Z
dd� Z dd� Z dd� Z
dd� ZdS )�UploadFilenameInputManagerz Upload utility for filenamesc C s t |tj�S r )�
isinstancer � string_typesrA r r r rD � s z(UploadFilenameInputManager.is_compatiblec C � dS )NFr rF r r r rH � r2 z0UploadFilenameInputManager.stores_body_in_memoryc C s |j �| j�|j jj�� d S r )�metarL r9 �
get_file_size� call_argsr& rJ r r r rL � s
��z0UploadFilenameInputManager.provide_transfer_sizec C s |j j|jkS r )ra �size�multipart_thresholdrM r r r rO � r0 z4UploadFilenameInputManager.requires_multipart_uploadc C sJ | � |�\}}| �|�}| �|�}| �|�}|jj}| jj|||||d�S )N�r& �
chunk_size�full_file_sizer �close_callbacks)�&_get_put_object_fileobj_with_full_sizerU rW r[ ra rd r9 �#open_file_chunk_reader_from_fileobj)r rK r& � full_sizer ri rd r r r rP � s �
�z.UploadFilenameInputManager.get_put_object_bodyc c s� � |j j}| �||�}td|d �D ]5}| �|�}| �|�}||d }| j|j jj|||d�\} }
| � | �} | j
j | ||
||d�} || fV qd S )N� )�
start_byte� part_sizerh rf ) ra rd �_get_num_parts�rangerW r[ �'_get_upload_part_fileobj_with_full_sizerc r& rU r9 rk ) r rK rQ rh � num_parts� part_numberr ri rn r& rl �read_file_chunkr r r rR s& �
�
� �z3UploadFilenameInputManager.yield_upload_part_bodiesc C s t ||| jjd�}|S )N)�
open_function)r r9 �open)r r& rn r r r �_get_deferred_open_file s
�z2UploadFilenameInputManager._get_deferred_open_filec C s" |j jj}|j j}| �|d�|fS )Nr )ra rc r& rd rx �r rK r& rd r r r rj # s
zAUploadFilenameInputManager._get_put_object_fileobj_with_full_sizec K s |d }|d }| � ||�|fS )Nrn rh )rx )r r& r4 rn rl r r r rr ( s zBUploadFilenameInputManager._get_upload_part_fileobj_with_full_sizec C s t t�|jjt|� ��S r )�int�math�ceilra rd �float)r rK ro r r r rp - s �z)UploadFilenameInputManager._get_num_partsN)r r! r" r7 r\ rD rH rL rO rP rR rx rj rr rp r r r r r] � s
r] c @ s< e Zd ZdZedd� �Zdd� Zdd� Zdd � Zd
d � Z d S )
�UploadSeekableInputManagerz&Upload utility for an open file objectc C s t |�ot|�S r )r r rA r r r rD 4 s z(UploadSeekableInputManager.is_compatiblec C s |dkrdS dS )N�
put_objectFTr rF r r r rH 8 s z0UploadSeekableInputManager.stores_body_in_memoryc C sD |j jj}|�� }|�dd� |�� }|�|� |j �|| � d S )Nr � )ra rc r& r. r+ rL )r rK r& �start_position� end_positionr r r rL > s
�z0UploadSeekableInputManager.provide_transfer_sizec K s |� |d �}t�|�t|�fS )Nro )r) r �BytesIO�len)r r& r4 �datar r r rr J s zBUploadSeekableInputManager._get_upload_part_fileobj_with_full_sizec C s" |j jj}|�� |j j }||fS r )ra rc r& r. rd ry r r r rj Y s
zAUploadSeekableInputManager._get_put_object_fileobj_with_full_sizeN)
r r! r" r7 r\ rD rH rL rr rj r r r r r~ 2 s
r~ c sh e Zd ZdZd� fdd� Zedd� �Zdd� Zd d
� Zd d � Z d
d� Z
dd� Z ddd�Z dd� Z
� ZS )� UploadNonSeekableInputManagerz7Upload utility for a file-like object that cannot seek.Nc s t t| ��|||� d| _d S )N� )�superr� r �
_initial_datar; �� __class__r r r c s
�
z&UploadNonSeekableInputManager.__init__c C s t |�S r )r rA r r r rD h s z+UploadNonSeekableInputManager.is_compatiblec C r` )NTr rF r r r rH l r2 z3UploadNonSeekableInputManager.stores_body_in_memoryc C s d S r r rJ r r r rL o s z3UploadNonSeekableInputManager.provide_transfer_sizec C sP |j jd ur
|j j|jkS |j jj}|j}| �||d�| _t| j�|k r&dS dS )NFT)ra rd re rc r& �_readr� r� )r rK rN r& r r r r rO t s
z7UploadNonSeekableInputManager.requires_multipart_uploadc C s@ | � |�}| �|�}|jjj}| �| j|�� ||�}d | _|S r )rW r[ ra rc r& �
_wrap_datar� r) )r rK r ri r& �bodyr r r rP � s
�z1UploadNonSeekableInputManager.get_put_object_bodyc c s` � |j jj}d} | �|�}| �|�}|d7 }| �||�}|s!d S | �|||�}d }||fV q )Nr Trm )ra rc r& rW r[ r� r� ) r rK rQ � file_objectrt r ri � part_content� part_objectr r r rR � s �
�
�z6UploadNonSeekableInputManager.yield_upload_part_bodiesTc C sx t | j�dkr |�|�S |t | j�kr&| jd|� }|r$| j|d� | _|S |t | j� }| j|�|� }|r:d| _|S )a=
Reads a specific amount of data from a stream and returns it. If there
is any data in initial_data, that will be popped out first.
:type fileobj: A file-like object that implements read
:param fileobj: The stream to read from.
:type amount: int
:param amount: The number of bytes to read from the stream.
:type truncate: bool
:param truncate: Whether or not to truncate initial_data after
reading from it.
:return: Generator which generates part bodies from the initial data.
r Nr� )r� r� r) )r r& r* �truncater� �amount_to_readr r r r� � s
z#UploadNonSeekableInputManager._readc C s. | � t�|��}| jj|t|�t|�||d�S )a�
Wraps data with the interrupt reader and the file chunk reader.
:type data: bytes
:param data: The data to wrap.
:type callbacks: list
:param callbacks: The callbacks associated with the transfer future.
:type close_callbacks: list
:param close_callbacks: The callbacks to be called when closing the
wrapper for the data.
:return: Fully wrapped data.
rf )rU r r� r9 rk r� )r r� r ri r& r r r r� � s
�z(UploadNonSeekableInputManager._wrap_datar )T)r r! r" r7 r r\ rD rH rL rO rP rR r� r� �
__classcell__r r r� r r� a s
*r� c @ s\ e Zd ZdZg d�ZddgZdd� Z ddd �Zd
d � Zd d
� Z dd� Z
dd� Z dd� Z dS )�UploadSubmissionTaskz.Task for submitting tasks to execute an upload)�SSECustomerKey�SSECustomerAlgorithm�SSECustomerKeyMD5� RequestPayer�ExpectedBucketOwnerr� r� c C sD t ttg}|jjj}|D ] }|�|�r| S q td|t|�f ��)ao Retrieves a class for managing input for an upload based on file type
:type transfer_future: s3transfer.futures.TransferFuture
:param transfer_future: The transfer future for the request
:rtype: class of UploadInputManager
:returns: The appropriate class to use for managing a specific type of
input for uploads.
z&Input %s of type: %s is not supported.) r] r~ r� ra rc r& rD � RuntimeError�type)r rK � upload_manager_resolver_chainr& �upload_manager_clsr r r � _get_upload_input_manager_cls� s �
���z2UploadSubmissionTask._get_upload_input_manager_clsNc C sf | � |�|| j|�}|jjdu r|�|� |�||�s'| �||||||� dS | �||||||� dS )a�
:param client: The client associated with the transfer manager
:type config: s3transfer.manager.TransferConfig
:param config: The transfer config associated with the transfer
manager
:type osutil: s3transfer.utils.OSUtil
:param osutil: The os utility associated to the transfer manager
:type request_executor: s3transfer.futures.BoundedExecutor
:param request_executor: The request executor associated with the
transfer manager
:type transfer_future: s3transfer.futures.TransferFuture
:param transfer_future: The transfer future associated with the
transfer request that tasks are being submitted for
N)r� r% ra rd rL rO �_submit_upload_request�_submit_multipart_request)r �clientrN r< �request_executorrK r= �upload_input_managerr r r �_submit
s$ ��
�
�
�z UploadSubmissionTask._submitc
C sN |j j}| �|d�}| jj|t| j||�|�|j|j|j d�dd�|d� d S )Nr )r� r& �bucket�key�
extra_argsT)r' � main_kwargs�is_final��tag)
ra rc �_get_upload_task_tagr% �submit�
PutObjectTaskrP r� r� r� ) r r� rN r< r� rK r� rc �put_object_tagr r r r� 4 s( ����
�z+UploadSubmissionTask._submit_upload_requestc C s� |j j}| j�|t| j||j|j|jd�d��}g } | �|j�}
| � |d�} |j j
} t � }
|
� |j
| �}|�||�}|D ]!\}}| �| jj|t| j|||j|j||
d�d|id�| d�� q<| �|j�}| j�|t| j||j|j|d�|| d�d d
�� d S ) N)r� r� r� r� )r' r� � upload_part)r� r& r� r� rt r� � upload_id)r' r� �pending_main_kwargsr� )r� �partsT)r' r� r� r� )ra rc r% r� r r� r� r� �_extra_upload_part_argsr� rd r
�adjust_chunksize�multipart_chunksizerR �append�UploadPartTask� _extra_complete_multipart_argsr )r r� rN r< r� rK r� rc �create_multipart_future� part_futures�extra_part_args�upload_part_tagrd �adjusterrQ �
part_iteratorrt r& � complete_multipart_extra_argsr r r r� N sx ��� �� � ���������z.UploadSubmissionTask._submit_multipart_requestc C � t || j�S r )r �UPLOAD_PART_ARGS�r r� r r r r� � s z,UploadSubmissionTask._extra_upload_part_argsc C r� r )r �COMPLETE_MULTIPART_ARGSr� r r r r� � r6 z3UploadSubmissionTask._extra_complete_multipart_argsc C s d }|� |�r t}|S r )rH r )r r� rG r� r r r r� � s
z)UploadSubmissionTask._get_upload_task_tagr )
r r! r" r7 r� r� r� r� r� r� r� r� r� r r r r r� � s �
�'M r� c @ � e Zd ZdZdd� ZdS )r� z Task to do a nonmultipart uploadc C sB |�}|j d|||d�|�� W d � dS 1 sw Y dS )aP
:param client: The client to use when calling PutObject
:param fileobj: The file to upload.
:param bucket: The name of the bucket to upload to
:param key: The name of the key to upload to
:param extra_args: A dictionary of any extra arguments that may be
used in the upload.
)�Bucket�Key�BodyNr )r )r r� r& r� r� r� r� r r r �_main� s "�zPutObjectTask._mainN�r r! r" r7 r� r r r r r� � � r� c @ r� )r� z+Task to upload a part in a multipart uploadc C sR |�}|j d|||||d�|��} W d � n1 sw Y | d }
|
|d�S )a�
:param client: The client to use when calling PutObject
:param fileobj: The file to upload.
:param bucket: The name of the bucket to upload to
:param key: The name of the key to upload to
:param upload_id: The id of the upload
:param part_number: The number representing the part of the multipart
upload
:param extra_args: A dictionary of any extra arguments that may be
used in the upload.
:rtype: dict
:returns: A dictionary representing a part::
{'Etag': etag_value, 'PartNumber': part_number}
This value can be appended to a list to be used to complete
the multipart upload.
)r� r� �UploadId�
PartNumberr� N�ETag)r� r� r )r� ) r r� r& r� r� r� rt r� r� �response�etagr r r r� � s �� �
zUploadPartTask._mainNr� r r r r r� � r� r� ) r{ �botocore.compatr �s3transfer.compatr r �s3transfer.futuresr �s3transfer.tasksr r r r �s3transfer.utilsr
r r r
�objectr r# r8 r] r~ r� r� r� r� r r r r �<module> s, !+yQ/ F