|
Server : Apache System : Linux iad1-shared-b8-43 6.6.49-grsec-jammy+ #10 SMP Thu Sep 12 23:23:08 UTC 2024 x86_64 User : dh_edsupp ( 6597262) PHP Version : 8.2.26 Disable Function : NONE Directory : /lib/python3/dist-packages/boto/glacier/__pycache__/ |
Upload File : |
o
ckF[YC � @ s� d dl Z d dlZd dlZd dlZd dlZd dlZd dlmZ d dlZd dl m
Z
m Z m Z m
Z
mZ d dlmZmZmZ e� Ze�d�ZG dd� de�ZG dd � d e�ZG d
d � d ej�ZG d d
� d
e�ZG dd� de�Z G dd� de�Z dS )� N)�Queue)�DEFAULT_PART_SIZE�minimum_part_size� chunk_hashes� tree_hash� bytes_to_hex)�UploadArchiveError�DownloadArchiveError�TreeHashDoesNotMatchErrorzboto.glacier.concurrentc @ s2 e Zd Zedfdd�Zdd� Zdd� Zdd � Zd
S ) �ConcurrentTransferer�
c C s || _ || _g | _d S �N)�
_part_size� _num_threads�_threads)�self� part_size� num_threads� r �9/usr/lib/python3/dist-packages/boto/glacier/concurrent.py�__init__+ s
z ConcurrentTransferer.__init__c C sL t |�}| j|kr
| j}n
|}t�d| j|� tt�|t|� ��}||fS )NzfThe part size specified (%s) is smaller than the minimum required part size. Using a part size of: %s)r r �log�debug�int�math�ceil�float)r �
total_size�min_part_size_requiredr � total_partsr r r � _calculate_required_part_size0 s
�z2ConcurrentTransferer._calculate_required_part_sizec C s> t �d� | jD ]}d|_q| jD ]}|�� qt �d� d S )NzShutting down threads.FzThreads have exited.)r r r �should_continue�join)r �threadr r r �_shutdown_threads< s
z&ConcurrentTransferer._shutdown_threadsc C sD t �d� t|�D ] }|�||f� q t| j�D ]}|�t� qd S )NzAdding work items to queue.)r r �range�putr �
_END_SENTINEL)r r � worker_queuer �ir r r �_add_work_items_to_queueD s
�z-ConcurrentTransferer._add_work_items_to_queueN)�__name__�
__module__� __qualname__r r r r$ r* r r r r r * s
r c s@ e Zd ZdZedf� fdd� Zd dd�Zdd � Zd
d � Z� Z S )
�ConcurrentUploadera Concurrently upload an archive to glacier.
This class uses a thread pool to concurrently upload an archive
to glacier using the multipart upload API.
The threadpool is completely managed by this class and is
transparent to the users of this class.
r c s" t t| ��||� || _|| _dS )au
:type api: :class:`boto.glacier.layer1.Layer1`
:param api: A layer1 glacier object.
:type vault_name: str
:param vault_name: The name of the vault.
:type part_size: int
:param part_size: The size, in bytes, of the chunks to use when uploading
the archive parts. The part size must be a megabyte multiplied by
a power of two.
:type num_threads: int
:param num_threads: The number of threads to spawn for the thread pool.
The number of threads will control how much parts are being
concurrently uploaded.
N)�superr. r �_api� _vault_name)r �api�
vault_namer r �� __class__r r r V s
zConcurrentUploader.__init__Nc
C s� t �|�j}| �|�\}}dg| }t� }t� }| j�| j||�} | d }
| �|||� | � ||
||� z | �
|||� W n t yX } zt �
d� | j�| j|
� | �d} ~ ww t �
d� | j�| j|
tt|��|�} t �
d� | d S )a^ Concurrently create an archive.
The part_size value specified when the class was constructed
will be used *unless* it is smaller than the minimum required
part size needed for the size of the given file. In that case,
the part size used will be the minimum part size required
to properly upload the given file.
:type file: str
:param file: The filename to upload
:type description: str
:param description: The description of the archive.
:rtype: str
:return: The archive id of the newly created archive.
N�UploadIdzHAn error occurred while uploading an archive, aborting multipart upload.zCompleting upload.zUpload finished.� ArchiveId)�os�stat�st_sizer r r0 �initiate_multipart_uploadr1 r* �_start_upload_threads�_wait_for_upload_threadsr r r �abort_multipart_upload�complete_multipart_uploadr r ) r �filename� descriptionr r r � hash_chunksr( � result_queue�response� upload_id�er r r �uploadn s>
���
��
�
zConcurrentUploader.uploadc C s\ t |�D ]#}|�� }t|t�r t�d|� | �� td| ��|\}}|||< q| �� d S )N�?An error was found in the result queue, terminating threads: %s�0An error occurred while uploading an archive: %s)r% �get�
isinstance� Exceptionr r r$ r )r rB rC r �_�result� part_number� tree_sha256r r r r= � s
��
z+ConcurrentUploader._wait_for_upload_threadsc C sR t �d� t| j�D ] }t| j| j||||�}t�d� |� � | j
� |� q
d S �NzStarting threads.g�������?) r r r% r �UploadWorkerThreadr0 r1 �time�sleep�startr �append)r rC rE r( r@ rM r# r r r r< � s
�
�z(ConcurrentUploader._start_upload_threadsr
)
r+ r, r- �__doc__r r rG r= r< �
__classcell__r r r4 r r. L s �
2r. c s4 e Zd Z� fdd�Zdd� Zdd� Zdd� Z� ZS ) �TransferThreadc s$ t t| ��� || _|| _d| _d S )NT)r/ rY r �
_worker_queue�
_result_queuer! )r r( rC r4 r r r � s
zTransferThread.__init__c C sh | j r.z | jjdd�}W n ty Y q w |tu r | �� d S | �|�}| j�|� | j s| �� d S )N� )�timeout) r! rZ rJ �Emptyr' �_cleanup�_process_chunkr[ r& )r �workrN r r r �run� s �
�
zTransferThread.runc C � d S r
r )r ra r r r r` � � z TransferThread._process_chunkc C rc r
r �r r r r r_ � rd zTransferThread._cleanup)r+ r, r- r rb r` r_ rX r r r4 r rY � s
rY c s< e Zd Zddef� fdd� Zdd� Zdd� Zdd � Z� ZS )
rR � c
sL t t| ��||� || _|| _|| _t|d�| _|| _|| _ || _
| | _ d S )N�rb) r/ rR r r0 r1 � _filename�open�_fileobj�
_upload_id� _num_retries�_time_between_retries�_retry_exceptions)
r r2 r3 r@ rE r( rC � num_retries�time_between_retries�retry_exceptionsr4 r r r � s
zUploadWorkerThread.__init__c C s� d }t | jd �D ]<}z | �|�}W |S | jyE } z$t�d|d | j|d | jd | j|j|� t �
| j � |}W Y d }~q d }~ww |S )Nr\ zpException caught uploading part number %s for vault %s, attempt: (%s / %s), filename: %s, exception: %s, msg: %sr ) r% rl �
_upload_chunkrn r �errorr1 rh r5 rS rT rm )r ra rN r) rF r r r r` � s
�
� ��z!UploadWorkerThread._process_chunkc
C s� |\}}|| }| j �|� | j �|�}t�|��� }tt|��}||t|� d f}t �
d||� | j � | j
| j|t|�||�} | �� ||fS )Nr\ z Uploading chunk %s of size %s)rj �seek�read�hashlib�sha256� hexdigestr r �lenr r r0 � upload_partr1 rk r )
r ra rO r �
start_byte�contents� linear_hash�tree_hash_bytes�
byte_rangerD r r r rr � s �z UploadWorkerThread._upload_chunkc C s | j �� d S r
)rj �closere r r r r_ s zUploadWorkerThread._cleanup) r+ r, r- rL r r` rr r_ rX r r r4 r rR � s �rR c s> e Zd ZdZedf� fdd� Zdd� Zdd� Zd d
� Z� Z S ) �ConcurrentDownloaderz�
Concurrently download an archive from glacier.
This class uses a thread pool to concurrently download an archive
from glacier.
The threadpool is completely managed by this class and is
transparent to the users of this class.
r c s t t| ��||� || _dS )a
:param job: A layer2 job object for archive retrieval object.
:param part_size: The size, in bytes, of the chunks to use when uploading
the archive parts. The part size must be a megabyte multiplied by
a power of two.
N)r/ r� r �_job)r �jobr r r4 r r r s
z ConcurrentDownloader.__init__c
C s� | j j}| �|�\}}t� }t� }| �|||� | �||� z | �|||� W n ty; } zt� d|� |�d}~ww t� d� dS )z�
Concurrently download an archive.
:param filename: The filename to download the archive to
:type filename: str
z2An error occurred while downloading an archive: %sNzDownload completed.)
r� � archive_sizer r r* �_start_download_threads�_wait_for_download_threadsr r r )r r@ r r r r( rC rF r r r �download$ s ��z ConcurrentDownloader.downloadc C s� dg| }t |d��D}t|�D ]7}|�� }t|t�r*t�d|� | �� td| ��|\}} }
} |
||< || } |� | � |�
| � |� � qW d � n1 sQw Y t t
|��}
t�d| jj|
� | jj|
kry| �� td| jj|
f ��| �� dS )a
Waits until the result_queue is filled with all the downloaded parts
This indicates that all part downloads have completed
Saves downloaded parts into filename
:param filename:
:param result_queue:
:param total_parts:
N�wbrH rI z?Verifying final tree hash of archive, expecting: %s, actual: %szBTree hash for entire archive does not match, expected: %s, got: %s)ri r% rJ rK rL r r r$ r rt �write�flushr r r� �sha256_treehashr
)r r@ rC r rB �frM rN rO r � actual_hash�datar{ �
final_hashr r r r� 9 sH
���
� � � ��� z/ConcurrentDownloader._wait_for_download_threadsc C sJ t �d� t| j�D ]}t| j||�}t�d� |�� | j �
|� q
d S rQ ) r r r% r �DownloadWorkerThreadr� rS rT rU r rV )r rC r( rM r# r r r r� ` s
�z,ConcurrentDownloader._start_download_threads)
r+ r, r- rW r r r� r� r� rX r r r4 r r� s
�
'r� c s4 e Zd Zddef� fdd� Zdd� Zdd� Z� ZS )r� rf c s. t t| ��||� || _|| _|| _|| _dS )a
Individual download thread that will download parts of the file from Glacier. Parts
to download stored in work queue.
Parts download to a temp dir with each part a separate file
:param job: Glacier job object
:param work_queue: A queue of tuples which include the part_number and
part_size
:param result_queue: A priority queue of tuples which include the
part_number and the path to the temp file that holds that
part's data.
N)r/ r� r r� rl rm rn )r r� r( rC ro rp rq r4 r r r j s
z DownloadWorkerThread.__init__c C st d}t | j�D ]0}z | �|�}W |S | jy7 } zt�d|d | j� t�| j � |}W Y d}~qd}~ww |S )z�
Attempt to download a part of the archive from Glacier
Store the result in the result_queue
:param work:
Nz6Exception caught downloading part number %s for job %sr )
r% rl �_download_chunkrn r rs r� rS rT rm )r ra rN rM rF r r r r` � s
�
� ��z#DownloadWorkerThread._process_chunkc C s� |\}}|| }||| d f}t �d||� | j�|�}|�� }ttt|���}|d |kr:td||d |f ��||t �
|�|fS )z�
Downloads a chunk of archive from Glacier. Saves the data to a temp file
Returns the part number and temp file location
:param work:
r\ z Downloading chunk %s of size %s�TreeHashzBTree hash for part number %s does not match, expected: %s, got: %s) r r r� �
get_outputru r r r r
�binascii� unhexlify) r ra rO r r{ r rD r� r� r r r r� � s ���z$DownloadWorkerThread._download_chunk)r+ r, r- rL r r` r� rX r r r4 r r� i s �r� ) r8 r � threadingrv rS �logging� boto.compatr r� �boto.glacier.utilsr r r r r �boto.glacier.exceptionsr r r
�objectr' � getLoggerr r r. �ThreadrY rR r� r� r r r r �<module> s$
"o 4]