|
Server : Apache System : Linux iad1-shared-b8-43 6.6.49-grsec-jammy+ #10 SMP Thu Sep 12 23:23:08 UTC 2024 x86_64 User : dh_edsupp ( 6597262) PHP Version : 8.2.26 Disable Function : NONE Directory : /lib/python3/dist-packages/boto/s3/__pycache__/ |
Upload File : |
o
ckF[zE � @ s d dl Zd dlZd dlZd dlZd dlZd dlZd dlZd dlZd dl Z d dlm
Z
d dl Z d dl
mZmZmZmZ d dlmZ d dlmZ d dlmZ d dlmZ d dlmZ d d lmZ d d
l m Z d d l m Z m Z d d l m Z d d
l m Z G dd� de!�Z"dS )� N)�md5)�BytesIO�six�urllib� encodebytes��BotoClientError)�StorageDataError)�PleaseRetryException)�Provider)�KeyFile)�User)� UserAgent)� compute_md5� compute_hash)�find_matching_headers)�merge_headers_by_namec @ sN e Zd ZdZdZdZej�ddd�Z e
g d��Z e
� Z e D ] Z
e �e
�dd �� q e e
g d
��B Zd�d d
�Zdd� Zdd� Zedd� �Zdd� Zdd� Zeee�Zdd� Zdd� Zeee�Zd d � Z d d � Z ee e �Z d d!� Z d"d#� Z ee e �Z!d$d%� Z"d&d'� Z#d(d)� Z$d�d+d,�Z%d-d.� Z&d/d0� Z' 1 d�d2d3�Z(d�d4d5�Z) d�d7d8�Z*d*Z+d�d9d:�Z,d;d<� Z-e-Z.d�d>d?�Z/ @d�dAdB�Z0 * @d�dCdD�Z1dEdF� Z2dGdH� Z3d�dIdJ�Z4d�dKdL�Z5dMdN� Z6dOdP� Z7dQdR� Z8d�dSdT�Z9d�dUdV�Z:d�dWdX�Z;d�dYdZ�Z<d�d[d\�Z=d]d^� Z>d�d_d`�Z?d�dadb�Z@ *d�ddde�ZA f d�dgdh�ZB f d�didj�ZCd�dkdl�ZDd�dmdn�ZE @ d�dodp�ZF @ *d�dqdr�ZG @ * *d�dsdt�ZH @ * *d�dudv�ZI f d�dwdx�ZJ f d�dydz�ZKd�d{d|�ZL f * d�d}d~�ZM f * d�dd��ZN f * d�d�d��ZOd�d�d��ZP d�d�d��ZQd�d�� ZRd�d�d��ZS d�d�d��ZTd�d�d��ZUd S )��KeyaC
Represents a key (object) in an S3 bucket.
:ivar bucket: The parent :class:`boto.s3.bucket.Bucket`.
:ivar name: The name of this Key object.
:ivar metadata: A dictionary containing user metadata that you
wish to store with the object or that has been retrieved from
an existing object.
:ivar cache_control: The value of the `Cache-Control` HTTP header.
:ivar content_type: The value of the `Content-Type` HTTP header.
:ivar content_encoding: The value of the `Content-Encoding` HTTP header.
:ivar content_disposition: The value of the `Content-Disposition` HTTP
header.
:ivar content_language: The value of the `Content-Language` HTTP header.
:ivar etag: The `etag` associated with this object.
:ivar last_modified: The string timestamp representing the last
time this object was modified in S3.
:ivar owner: The ID of the owner of this object.
:ivar storage_class: The storage class of the object. Currently, one of:
STANDARD | REDUCED_REDUNDANCY | GLACIER
:ivar md5: The MD5 hash of the contents of the object.
:ivar size: The size, in bytes, of the object.
:ivar version_id: The version ID of this object, if it is a versioned
object.
:ivar encrypted: Whether the object is encrypted while at rest on
the server.
zapplication/octet-streamz�<?xml version="1.0" encoding="UTF-8"?>
<RestoreRequest xmlns="http://s3.amazonaws.com/doc/2006-03-01">
<Days>%s</Days>
</RestoreRequest>�Boto�key_buffer_sizei )z
cache-controlzcontent-dispositionzcontent-encodingzcontent-language� content-md5� content-typez x-robots-tag�expires�-�_)z
last-modified�content-length�date�etagNc C s� || _ || _i | _d | _| j| _d | _d | _d | _d | _ d | _
d| _ d | _ d | _
d | _d | _d | _d | _d | _d | _d | _d| _d | _d | _d | _i | _d S )NF)�bucket�name�metadata�
cache_control�DefaultContentType� content_type�content_encoding�content_disposition�content_language�filenamer � is_latest�
last_modified�owner�_storage_class�path�resp�mode�size�
version_id�source_version_id�
delete_marker� encrypted�ongoing_restore� expiry_date� local_hashes)�selfr r � r8 �-/usr/lib/python3/dist-packages/boto/s3/key.py�__init__i s2
z Key.__init__c C s< | j r
d| j j| jf }nd| j }t|t�s |�d�}|S )Nz <Key: %s,%s>z<Key: None,%s>�utf-8)r r �
isinstance�str�encode�r7 r r8 r8 r9 �__repr__� s
z Key.__repr__c C s | S �Nr8 �r7 r8 r8 r9 �__iter__� s z Key.__iter__c C s d }| j r| j jr| j jj}|S rA )r �
connection�provider)r7 rE r8 r8 r9 rE � s
z Key.providerc C s | j S rA �r rB r8 r8 r9 �_get_key� s z Key._get_keyc C �
|| _ d S rA rF �r7 �valuer8 r8 r9 �_set_key� �
z Key._set_keyc C s, d| j v r| j d rt�| j d �S d S d S �Nr )r6 �binascii�b2a_hexrB r8 r8 r9 �_get_md5� s �z Key._get_md5c C s8 |r t �|�| jd< d S d| jv r| j�dd � d S d S rM )rN �a2b_hexr6 �poprI r8 r8 r9 �_set_md5� s
�z Key._set_md5c C sP d| j v r$| j d r&| j d }t|t�s|�d�}t�|��d��d�S d S d S )Nr r; �
)r6 r< �bytesr> rN �
b2a_base64�decode�rstrip)r7 r r8 r8 r9 �_get_base64md5� s
�zKey._get_base64md5c C sH |rt |tj�s
|�d�}t�|�| jd< d S d| jv r"| jd= d S d S )Nr; r )r< r � string_typesrW rN �
a2b_base64r6 rI r8 r8 r9 �_set_base64md5� s
�zKey._set_base64md5c C s^ | j d u r,| jr,t| j�| j�d���}t|�r)t|d dd �r)|d j | _ | j S d| _ | j S )Nr; r r+ �STANDARD)r+ r �listr r> �len�getattr)r7 �
list_itemsr8 r8 r9 �_get_storage_class� s � �zKey._get_storage_classc C rH rA )r+ rI r8 r8 r9 �_set_storage_class� rL zKey._set_storage_classc C s2 t �|�}t|�}|d dkr|dd� }||fS )z�
A utility function to create the 2-tuple (md5hexdigest, base64md5)
from just having a precalculated md5_hexdigest.
���rT r )rN � unhexlifyr )r7 �
md5_hexdigest�digest� base64md5r8 r8 r9 �get_md5_from_hexdigest� s
zKey.get_md5_from_hexdigestc C s. | j jj}|jr|�|jd �| _d S d | _d S rA )r rD rE � server_side_encryption_header� getheaderr3 �r7 r- rE r8 r8 r9 �handle_encryption_headers� s
�
z Key.handle_encryption_headersc C sL | j jj}|jr |�|jd �| _| jd u r"|�� dkr$d| _d S d S d S d S )N�awsr] )r rD rE �storage_class_headerrk r+ �get_provider_namerl r8 r8 r9 �handle_storage_class_header� s
�
�z Key.handle_storage_class_headerFc C s^ | j jj}| jd u s |r|�|jd �| _|�|jd �| _|�|jd�dkr*d| _d S d| _d S )N�false�trueTF)r rD rE r0 rk �copy_source_version_idr1 r2 )r7 r- �forcerE r8 r8 r9 �handle_version_headers� s
�
zKey.handle_version_headersc C s� | j jj}|�|j�}|d u rd S |�dd�}|D ]+}dd� |�d�D �\}}|�dd�}|dkr=|�� d kr9d
nd | _q|d krD|| _ qd S )
N�,� c S s g | ]}|� � �qS r8 )�strip)�.0�ir8 r8 r9 �
<listcomp> s z.Key.handle_restore_headers.<locals>.<listcomp>�=�"� zongoing-requestrs TFz expiry-date)
r rD rE rk �restore_header�split�replace�lowerr4 r5 )r7 �responserE �header�parts�part�key�valr8 r8 r9 �handle_restore_headers s
��zKey.handle_restore_headersc C s dS )z�
Used by Key subclasses to do additional, provider-specific
processing of response headers. No-op for this base class.
Nr8 �r7 �headersr8 r8 r9 �handle_addl_headers s zKey.handle_addl_headersr c
C sD | j du r�d| _| jjj}| jjjd| jj| j|||d�| _ | j jdk s*| j jdkr:| j �� }|� | j j| j j
|��| j j }t j
�||�| _|�� D ]9\}}|�� dkr^d|vr^t|�| _qJ|�� d krqt�d
d |�} t| �| _qJ|�� tjv r�|| j|�� �d d
�< qJ| �| j � | �| j � | � | j � | � | j � � � dS dS )a�
Open this key for reading
:type headers: dict
:param headers: Headers to pass in the web request
:type query_args: string
:param query_args: Arguments to pass in the query string
(ie, 'torrent')
:type override_num_retries: int
:param override_num_retries: If not None will override configured
num_retries parameter for underlying GET.
:type response_headers: dict
:param response_headers: A dictionary containing HTTP
headers/values that will override any headers associated
with the stored object in the response. See
http://goo.gl/EWOPb for details.
N�r�GET��
query_args�override_num_retries�� �+ r z
Content-Rangez
content-rangez.*/(.*)z\1r r ) r- r. r rD rE � make_requestr �status�read�storage_response_error�reason�msg�boto�utils�get_aws_metadatar �itemsr� �intr/ �re�subr � base_fields�__dict__r� rv rm r� r� �
getheaders)
r7 r� r� r� �response_headersrE �bodyr rJ � end_ranger8 r8 r9 � open_read s@
�
�� � �z
Key.open_readc C s t d��)aK
Open this key for writing.
Not yet implemented
:type headers: dict
:param headers: Headers to pass in the write request
:type override_num_retries: int
:param override_num_retries: If not None will override configured
num_retries parameter for underlying PUT.
zNot Implementedr )r7 r� r� r8 r8 r9 �
open_writeP s zKey.open_writer� c C sN |dkrd| _ | j|||d� d S |dkr!d| _ | j||d� d S td| ��)Nr� )r� r� r� �w)r� r� zInvalid mode: %s)r. r� r� r )r7 r. r� r� r� r8 r8 r9 �open^ s
�
� zKey.openc C s* | j r
|s
| j �� d| _ d| _d| _dS )a�
Close this key.
:type fast: bool
:param fast: True if you want the connection to be closed without first
reading the content. This should only be used in cases where subsequent
calls don't need to return the content from the open HTTP connection.
Note: As explained at
http://docs.python.org/2/library/httplib.html#httplib.HTTPConnection.getresponse,
callers must read the whole response before sending a new request to the
server. Calling Key.close(fast=True) and making a subsequent request to
the server will work because boto will get an httplib exception and
close/reopen the connection.
NT)r- r� r. �closed)r7 �fastr8 r8 r9 �closem s
z Key.closec C s* | � � | j�| j�}|s| �� t�|S )a
By providing a next method, the key object supports use as an iterator.
For example, you can now say:
for bytes in key:
write bytes to a file or whatever
All of the HTTP connection stuff is handled for you.
)r� r- r� �
BufferSizer� �
StopIteration)r7 �datar8 r8 r9 �next� s
zKey.nextr c C s8 | � � |dkr| j�� }n| j�|�}|s| �� |S )Nr )r� r- r� r� )r7 r/ r� r8 r8 r9 r� � s zKey.readTc C sT |p| j j}|dkr| j|| jdd|d�S |dkr$| j|| jdd|d�S td| ��)a�
Change the storage class of an existing key.
Depending on whether a different destination bucket is supplied
or not, this will either move the item within the bucket, preserving
all metadata and ACL info bucket changing the storage class or it
will copy the item to the provided destination bucket, also
preserving metadata and ACL info.
:type new_storage_class: string
:param new_storage_class: The new storage class for the Key.
Possible values are:
* STANDARD
* REDUCED_REDUNDANCY
:type dst_bucket: string
:param dst_bucket: The name of a destination bucket. If not
provided the current bucket of the key will be used.
:type validate_dst_bucket: bool
:param validate_dst_bucket: If True, will validate the dst_bucket
by using an extra list request.
r] FT)�reduced_redundancy� preserve_acl�validate_dst_bucket�REDUCED_REDUNDANCYzInvalid storage class: %s)r r �copyr )r7 �new_storage_class�
dst_bucketr� � bucket_namer8 r8 r9 �change_storage_class� s
�
��zKey.change_storage_classc
C sB | j j�||�}|r
d}n| j}|j|| j j| j||||| jd�S )a�
Copy this Key to another bucket.
:type dst_bucket: string
:param dst_bucket: The name of the destination bucket
:type dst_key: string
:param dst_key: The name of the destination key
:type metadata: dict
:param metadata: Metadata to be associated with new key. If
metadata is supplied, it will replace the metadata of the
source key being copied. If no metadata is supplied, the
source key's metadata will be copied to the new key.
:type reduced_redundancy: bool
:param reduced_redundancy: If True, this will force the
storage class of the new Key to be REDUCED_REDUNDANCY
regardless of the storage class of the key being copied.
The Reduced Redundancy Storage (RRS) feature of S3,
provides lower redundancy at lower storage cost.
:type preserve_acl: bool
:param preserve_acl: If True, the ACL from the source key will
be copied to the destination key. If False, the
destination key will have the default ACL. Note that
preserving the ACL in the new key object will require two
additional API calls to S3, one to retrieve the current
ACL and one to set that ACL on the new object. If you
don't care about the ACL, a value of False will be
significantly more efficient.
:type encrypt_key: bool
:param encrypt_key: If True, the new copy of the object will
be encrypted on the server-side by S3 and will be stored
in an encrypted form while at rest in S3.
:type validate_dst_bucket: bool
:param validate_dst_bucket: If True, will validate the dst_bucket
by using an extra list request.
:rtype: :class:`boto.s3.key.Key` or subclass
:returns: An instance of the newly created key object
r� )�
storage_classr� � encrypt_key�src_version_id)r rD �lookupr� �copy_keyr r0 ) r7 r� �dst_keyr r� r� r� r� r� r8 r8 r9 r� � s
/� �zKey.copyc C s |dkr t | �| _| jS d S )N�Owner)r
r* )r7 r �attrsrD r8 r8 r9 � startElement s
zKey.startElementc C s� |dkr || _ d S |dkr|| _d S |dkr$|dkr d| _d S d| _d S |dkr-|| _d S |dkr8t|�| _d S |d krA|| _d S |d
krGd S |d krP|| _d S t| ||� d S ) Nr �ETag�IsLatestrs TF� LastModified�Size� StorageClassr� � VersionId) r r r( r) r� r/ r� r0 �setattr)r7 r rJ rD r8 r8 r9 �
endElement s&
zKey.endElementc C s t | jj| j|d��S )zt
Returns True if the key exists
:rtype: bool
:return: Whether the key exists on S3
�r� )�boolr r� r r� r8 r8 r9 �exists s z
Key.existsc C s | j j| j| j|d�S )z)
Delete this key from S3
)r0 r� )r �
delete_keyr r0 r� r8 r8 r9 �delete) s �z
Key.deletec C s | j �|�S rA )r �getr? r8 r8 r9 � get_metadata0 s zKey.get_metadatac C sf |� � dkr || jd< n|� � dkr|| jd< n|| j|< |� � tjv r1|| j|� � �dd�< d S d S )Nr � Content-Typer � Content-MD5r r )r� r r �base_user_settable_fieldsr� r� )r7 r rJ r8 r8 r9 � set_metadata3 s
�zKey.set_metadatac C s | j �|� d S rA )r �update)r7 �dr8 r8 r9 �update_metadata? s zKey.update_metadatac C s& | j d ur| j j|| j|d� d S d S �Nr� )r �set_aclr �r7 �acl_strr� r8 r8 r9 r� C s
�z Key.set_aclc C � | j d ur| j j| j|d�S d S r� )r �get_aclr r� r8 r8 r9 r� G �
�z Key.get_aclc C r� r� )r � get_xml_aclr r� r8 r8 r9 r� K r� zKey.get_xml_aclc C s" | j d ur| j j|| j|d�S d S r� )r � set_xml_aclr r� r8 r8 r9 r� O s
�zKey.set_xml_aclc C s | j �|| j|�S rA �r �set_canned_aclr r� r8 r8 r9 r� S � zKey.set_canned_aclc C sD | j j�d| j j| j�}|jdkr|�d�S | j�|j|j|� � ��)z�Return the redirect location configured for this key.
If no redirect is configured (via set_redirect), then None
will be returned.
�HEAD�� � x-amz-website-redirect-location)
r rD r� r r� rk rE r� r� r� )r7 r� r8 r8 r9 � get_redirectV s �
�zKey.get_redirectc C s^ |du ri }n|� � }||d< | jj�d| jj| j|�}|jdkr#dS | j�|j|j|� � ��)aO Configure this key to redirect to another location.
When the bucket associated with this key is accessed from the website
endpoint, a 301 redirect will be issued to the specified
`redirect_location`.
:type redirect_location: string
:param redirect_location: The location to redirect.
Nr� �PUTr� T)
r� r rD r� r r� rE r� r� r� )r7 �redirect_locationr� r� r8 r8 r9 � set_redirecte s �
�zKey.set_redirectc C s | j �d| j|�S )Nz public-readr� r� r8 r8 r9 � make_public~ r� zKey.make_publicr� c
C s� | j jj} |p | j}|du ri }n|�� }| r | || j< |
r*d| _| jr*| j|| j< | r1d|| j< t j
� || j | �}| j j�
||| j j| j||||||�
S )a$
Generate a URL to access this key.
:type expires_in: int
:param expires_in: How long the url is valid for, in seconds.
:type method: string
:param method: The method to use for retrieving the file
(default is GET).
:type headers: dict
:param headers: Any headers to pass along in the request.
:type query_auth: bool
:param query_auth: If True, signs the request in the URL.
:type force_http: bool
:param force_http: If True, http will be used instead of https.
:type response_headers: dict
:param response_headers: A dictionary containing HTTP
headers/values that will override any headers associated
with the stored object in the response. See
http://goo.gl/EWOPb for details.
:type expires_in_absolute: bool
:param expires_in_absolute:
:type version_id: string
:param version_id: The version_id of the object to GET. If specified
this overrides any value in the key.
:type policy: :class:`boto.s3.acl.CannedACLStrings`
:param policy: A canned ACL policy that will be applied to the
new key in S3.
:type reduced_redundancy: bool
:param reduced_redundancy: If True, this will set the storage
class of the new Key to be REDUCED_REDUNDANCY. The Reduced
Redundancy Storage (RRS) feature of S3, provides lower
redundancy at lower storage cost.
:type encrypt_key: bool
:param encrypt_key: If True, the new copy of the object will
be encrypted on the server-side by S3 and will be stored
in an encrypted form while at rest in S3.
:rtype: string
:return: The URL to access the key
Nr� �AES256)r rD rE r0 r� �
acl_headerr� ro rj r� r� �
merge_metar � generate_urlr )
r7 �
expires_in�methodr� �
query_auth�
force_httpr� �expires_in_absoluter0 �policyr� r� rE r8 r8 r9 r� � s,
6
�zKey.generate_url�
c C s | j |||||||d� dS )at
Upload a file to a key into a bucket on S3.
:type fp: file
:param fp: The file pointer to upload. The file pointer must
point at the offset from which you wish to upload.
ie. if uploading the full file, it should point at the
start of the file. Normally when a file is opened for
reading, the fp will point at the first byte. See the
bytes parameter below for more info.
:type headers: dict
:param headers: The headers to pass along with the PUT request
:type num_cb: int
:param num_cb: (optional) If a callback is specified with the
cb parameter this parameter determines the granularity of
the callback by defining the maximum number of times the
callback will be called during the file
transfer. Providing a negative integer will cause your
callback to be called with each buffer read.
:type query_args: string
:param query_args: (optional) Arguments to pass in the query string.
:type chunked_transfer: boolean
:param chunked_transfer: (optional) If true, we use chunked
Transfer-Encoding.
:type size: int
:param size: (optional) The Maximum number of bytes to read
from the file pointer (fp). This is useful when uploading
a file in multiple parts where you are splitting the file
up into different ranges to be uploaded. If not specified,
the default behaviour is to read all bytes from the file
pointer. Less bytes may be available.
�r� �cb�num_cbr� �chunked_transferr/ N)�_send_file_internal)r7 �fpr� r� r� r� r� r/ r8 r8 r9 � send_file� s '
�z
Key.send_filec s> �j jj�z��� � W n ty d � d�_Y nw �d u r%�js%dti�t�fdd��p.i D ���� �������� f dd�} |sEi }n|�� }t d|�D ]}
||
= qNt
|d< �j dvrc�j |�j
< t d |�rntd |��_t d
|�rytd
|��_t d |�} | r�t| �d kr�|| d
d u r�|| d
= n'td |��_n �jr�t��j�d
�_�jd u r��j�_�j|d < n�j|d < �jrÈj|d< �r�d|d< nt�j�|d< d�j j�� v r�tj d�} �d ur�| d< t d i | ��d
|d< d|d< t j � |�j!��}�j jj"d�j j#�j#|| |d�}
�j$|
dd� ��%|
�&� � d S ) NFr c 3 � � | ]
}|� | � fV qd S rA r8 �rz �alg�� hash_algsr8 r9 � <genexpr>
� � z*Key._send_file_internal.<locals>.<genexpr>c s< �d ur��� � kr���� n �d u r �jr ��d��i }tj�d|�r)d|d< tj�d|�r4d|d< | j||fi |�� |D ]
}| �||| � q@| � � �j
j j }d�j
j _ t
| dd�d k rf| �d� d}� r��ro�} n �jrv�j} nd} �r�| dkr�d
�j }
n �dkr�tt�| �j �d ��}
n �dk r�d }
nd}
d} � || � �} | r�| �jk r���| �}
n���j�}
t|
t�s�|
�d
�}
�d u r�d�_|
�rDt|
�}||7 }�r�| �d| � | �|
� | �d� n| �|
� �D ] }�| �|
� q�| �r
| |8 } | dk�r
n:� �r"| d7 } | |
k�s|
d k�r"� || � d} | �r1| �jk �r1��| �}
n���j�}
t|
t��sB|
�d
�}
|
s�|�_�D ] }�| �� �j |< �qI��rc| �d� | �d� � �rz|
dk�sp| dk�rz|dk�rz� || � | �|� |�j
j _ | � � }|�� }�� |���s��� |j |j!|��|S )Nz9Cannot retry failed request. fp does not support seeking.�hostrx � skip_hostzaccept-encoding�skip_accept_encodingr �
debuglevel� � � �?rd r; Tz%x;
z
z0
)"�tell�seek�read_from_stream�storage_data_errorr� r� r �
putrequest� putheader�
endheadersr rD �debugr` �set_debuglevelr/ r� r� �math�ceilr� r< rU r> r_ �sendr� rg r6 � getresponse� should_retryr� r� r� )� http_connr� r, r� r� �skipsr� �
save_debug�data_len�cb_size�cb_countr{ �
bytes_togo�chunk� chunk_lenr r� r� ) r� r� � digestersr� r� rE r7 r/ �sposr8 r9 �sender s� �
�
�
$
�z'Key._send_file_internal.<locals>.senderz
User-Agent)Nr] zContent-EncodingzContent-Languager� rx r r� �chunkedzTransfer-EncodingzContent-Lengthz
hmac-v4-s3)r� �hash_algorithmr/ �_sha256z 100-Continue�Expectr� )r( r� T)ru r8 )'r rD rE r �IOErrorr r �dictr� r r r+ r� ro r r$ r& r_ r# r, � mimetypes�
guess_typer" rh r= r/ �_required_auth_capability�hashlib�sha256r r� r� r� r r� r rv r� r� )r7 r� r� r� r� r� r� r/ r r( r� �content_type_headers�kwargsr- r8 )
r� r� r& r� r r� rE r7 r/ r' r9 r� � s~
� x
�
�
�
�zKey._send_file_internalc C s� | j jj}|s|jdv rdS |�d�rdS d|j kr dkr]n n;|�d�| _| j}t|t�r5|� d�}|�dd �}|�d d �}|d u r[|d u r[| jd
| kr[|�
d | j| jf ��dS |jd kr}|� � }|� |j|j
|�}|jd
v r}td|j |d��dS )N)i� i� T�locationr� r� r r; z/x-amz-server-side-encryption-customer-algorithmz x-goog-encryption-kms-key-namez"%s"z2ETag from S3 did not match computed MD5. %s vs. %si� )�RequestTimeoutzSaw %s, retrying)r� F)r rD rE r� rk r r r< rU rW r r� r� r� �
error_coder
) r7 r� r� rE r �-amz_server_side_encryption_customer_algorithm� goog_customer_managed_encryptionr� �errr8 r8 r9 r � sN
��
��
�
�zKey.should_retryc C s t ||d�\}}}|| _||fS )a�
:type fp: file
:param fp: File pointer to the file to MD5 hash. The file
pointer will be reset to the same position before the
method returns.
:type size: int
:param size: (optional) The Maximum number of bytes to read
from the file pointer (fp). This is useful when uploading
a file in multiple parts where the file is being split
in place into different parts. Less bytes may be available.
)r/ )r r/ )r7 r� r/ �
hex_digest�
b64_digest� data_sizer8 r8 r9 r � s
zKey.compute_md5c
C s� | j jj}
|
�� std|
�� ��| jr| jdkr td��|du r#i }|r*|||
j< |r8d| _|
j r8| j||
j < | j durV|sH| j �
| j�rHdS | j |||||d| d� dS dS )a"
Store an object using the name of the Key object as the key in
cloud and the contents of the data stream pointed to by 'fp' as
the contents.
The stream object is not seekable and total size is not known.
This has the implication that we can't specify the
Content-Size and Content-MD5 in the header. So for huge
uploads, the delay in calculating MD5 is avoided but with a
penalty of inability to verify the integrity of the uploaded
data.
:type fp: file
:param fp: the file whose contents are to be uploaded
:type headers: dict
:param headers: additional HTTP headers to be sent with the
PUT request.
:type replace: bool
:param replace: If this parameter is False, the method will first check
to see if an object exists in the bucket with the same key. If it
does, it won't overwrite it. The default value is True which will
overwrite the object.
:type cb: function
:param cb: a callback function that will be called to report
progress on the upload. The callback should accept two integer
parameters, the first representing the number of bytes that have
been successfully transmitted to GS and the second representing the
total number of bytes that need to be transmitted.
:type num_cb: int
:param num_cb: (optional) If a callback is specified with the
cb parameter, this parameter determines the granularity of
the callback by defining the maximum number of times the
callback will be called during the file transfer.
:type policy: :class:`boto.gs.acl.CannedACLStrings`
:param policy: A canned ACL policy that will be applied to the new key
in GS.
:type reduced_redundancy: bool
:param reduced_redundancy: If True, this will set the storage
class of the new Key to be REDUCED_REDUNDANCY. The Reduced
Redundancy Storage (RRS) feature of S3, provides lower
redundancy at lower storage cost.
:type size: int
:param size: (optional) The Maximum number of bytes to read from
the file pointer (fp). This is useful when uploading a
file in multiple parts where you are splitting the file up
into different ranges to be uploaded. If not specified,
the default behaviour is to read all bytes from the file
pointer. Less bytes may be available.
z$%s does not support chunked transferr zACannot determine the destination object name for the given streamNr� T)r� r/ ) r rD rE �supports_chunked_transferr rp r r� r� ro r� r ) r7 r� r� r� r� r� r� r� r� r/ rE r8 r8 r9 �set_contents_from_stream s.
=�
��z Key.set_contents_from_streamc
C s | j jj}
|pi }|r|||
j< |
rd||
j< | r!|�dtj� n1t|t �sR|�
� }|�dtj � |�
� |krM|�dtj� |�
� |krM|�|� t d��|�|� |r`d| _
|
jr`| j
||
j< t|d�ri|j| _| j dur�|sz|
�� rzd}d| _nbd}t|t �r�|�� }|j|�
� } | | _t�d |j�r�|j�d
�}|t�t� |��f}|s�| � || �}| j} n | r�| | _n|�
� }|�dtj � |�
� | | _|�|� | j} |d | _ |d | _ | jdu r�| j | _|s�| j � | j�r�dS | j!||||| || d � | jS dS )
a�
Store an object in S3 using the name of the Key object as the
key in S3 and the contents of the file pointed to by 'fp' as the
contents. The data is read from 'fp' from its current position until
'size' bytes have been read or EOF.
:type fp: file
:param fp: the file whose contents to upload
:type headers: dict
:param headers: Additional HTTP headers that will be sent with
the PUT request.
:type replace: bool
:param replace: If this parameter is False, the method will
first check to see if an object exists in the bucket with
the same key. If it does, it won't overwrite it. The
default value is True which will overwrite the object.
:type cb: function
:param cb: a callback function that will be called to report
progress on the upload. The callback should accept two
integer parameters, the first representing the number of
bytes that have been successfully transmitted to S3 and
the second representing the size of the to be transmitted
object.
:type num_cb: int
:param num_cb: (optional) If a callback is specified with the
cb parameter this parameter determines the granularity of
the callback by defining the maximum number of times the
callback will be called during the file transfer.
:type policy: :class:`boto.s3.acl.CannedACLStrings`
:param policy: A canned ACL policy that will be applied to the
new key in S3.
:type md5: A tuple containing the hexdigest version of the MD5
checksum of the file as the first element and the
Base64-encoded version of the plain checksum as the second
element. This is the same format returned by the
compute_md5 method.
:param md5: If you need to compute the MD5 for any reason
prior to upload, it's silly to have to do it twice so this
param, if present, will be used as the MD5 values of the
file. Otherwise, the checksum will be computed.
:type reduced_redundancy: bool
:param reduced_redundancy: If True, this will set the storage
class of the new Key to be REDUCED_REDUNDANCY. The Reduced
Redundancy Storage (RRS) feature of S3, provides lower
redundancy at lower storage cost.
:type encrypt_key: bool
:param encrypt_key: If True, the new copy of the object will
be encrypted on the server-side by S3 and will be stored
in an encrypted form while at rest in S3.
:type size: int
:param size: (optional) The Maximum number of bytes to read
from the file pointer (fp). This is useful when uploading
a file in multiple parts where you are splitting the file
up into different ranges to be uploaded. If not specified,
the default behaviour is to read all bytes from the file
pointer. Less bytes may be available.
:type rewind: bool
:param rewind: (optional) If True, the file pointer (fp) will
be rewound to the start before any bytes are read from
it. The default behaviour is False which reads from the
current position of the file pointer (fp).
:rtype: int
:return: The number of bytes written to the key.
r� r z8fp is at EOF. Use rewind option or seek() to data start.r� r NTFz^"[a-fA-F0-9]{32}"$r~ rx r� )"r rD rE r� rj r �os�SEEK_SETr< r r �SEEK_END�AttributeErrorr� ro �hasattrr r, r? r/ �getkeyr� �matchr ry �base64� b64encoderN re r r rh r� r )r7 r� r� r� r� r� r� r r� r� r� r/ �rewindrE r' r� r� r r8 r8 r9 �set_contents_from_fileo st
O
��zKey.set_contents_from_filec
C sJ t |d��}
| j|
|||||||| d� W d � S 1 s w Y dS )a#
Store an object in S3 using the name of the Key object as the
key in S3 and the contents of the file named by 'filename'.
See set_contents_from_file method for details about the
parameters.
:type filename: string
:param filename: The name of the file that you want to put onto S3
:type headers: dict
:param headers: Additional headers to pass along with the
request to AWS.
:type replace: bool
:param replace: If True, replaces the contents of the file
if it already exists.
:type cb: function
:param cb: a callback function that will be called to report
progress on the upload. The callback should accept two
integer parameters, the first representing the number of
bytes that have been successfully transmitted to S3 and
the second representing the size of the to be transmitted
object.
:type cb: int
:param num_cb: (optional) If a callback is specified with the
cb parameter this parameter determines the granularity of
the callback by defining the maximum number of times the
callback will be called during the file transfer.
:type policy: :class:`boto.s3.acl.CannedACLStrings`
:param policy: A canned ACL policy that will be applied to the
new key in S3.
:type md5: A tuple containing the hexdigest version of the MD5
checksum of the file as the first element and the
Base64-encoded version of the plain checksum as the second
element. This is the same format returned by the
compute_md5 method.
:param md5: If you need to compute the MD5 for any reason
prior to upload, it's silly to have to do it twice so this
param, if present, will be used as the MD5 values of the
file. Otherwise, the checksum will be computed.
:type reduced_redundancy: bool
:param reduced_redundancy: If True, this will set the storage
class of the new Key to be REDUCED_REDUNDANCY. The Reduced
Redundancy Storage (RRS) feature of S3, provides lower
redundancy at lower storage cost. :type encrypt_key: bool
:param encrypt_key: If True, the new copy of the object
will be encrypted on the server-side by S3 and will be
stored in an encrypted form while at rest in S3.
:rtype: int
:return: The number of bytes written to the key.
�rb�r� N)r� rK ) r7 r' r� r� r� r� r� r r� r� r� r8 r8 r9 �set_contents_from_filename! s = �$�z Key.set_contents_from_filenamec
C sD t |t�s
|�d�}t|�}
| j|
|||||||| d� } |
�� | S )ah
Store an object in S3 using the name of the Key object as the
key in S3 and the string 's' as the contents.
See set_contents_from_file method for details about the
parameters.
:type headers: dict
:param headers: Additional headers to pass along with the
request to AWS.
:type replace: bool
:param replace: If True, replaces the contents of the file if
it already exists.
:type cb: function
:param cb: a callback function that will be called to report
progress on the upload. The callback should accept two
integer parameters, the first representing the number of
bytes that have been successfully transmitted to S3 and
the second representing the size of the to be transmitted
object.
:type num_cb: int
:param num_cb: (optional) If a callback is specified with the
num_cb parameter this parameter determines the granularity of
the callback by defining the maximum number of times the
callback will be called during the file transfer.
:type policy: :class:`boto.s3.acl.CannedACLStrings`
:param policy: A canned ACL policy that will be applied to the
new key in S3.
:type md5: A tuple containing the hexdigest version of the MD5
checksum of the file as the first element and the
Base64-encoded version of the plain checksum as the second
element. This is the same format returned by the
compute_md5 method.
:param md5: If you need to compute the MD5 for any reason
prior to upload, it's silly to have to do it twice so this
param, if present, will be used as the MD5 values of the
file. Otherwise, the checksum will be computed.
:type reduced_redundancy: bool
:param reduced_redundancy: If True, this will set the storage
class of the new Key to be REDUCED_REDUNDANCY. The Reduced
Redundancy Storage (RRS) feature of S3, provides lower
redundancy at lower storage cost.
:type encrypt_key: bool
:param encrypt_key: If True, the new copy of the object will
be encrypted on the server-side by S3 and will be stored
in an encrypted form while at rest in S3.
r; rM )r< rU r> r rK r� ) r7 � string_datar� r� r� r� r� r r� r� r� r� r8 r8 r9 �set_contents_from_stringd s
9
�z Key.set_contents_from_stringc C s" | j ||||||||ddd�
dS )aN
Retrieves a file from an S3 Key
:type fp: file
:param fp: File pointer to put the data into
:type headers: string
:param: headers to send when retrieving the files
:type cb: function
:param cb: a callback function that will be called to report
progress on the upload. The callback should accept two
integer parameters, the first representing the number of
bytes that have been successfully transmitted to S3 and
the second representing the size of the to be transmitted
object.
:type cb: int
:param num_cb: (optional) If a callback is specified with the
cb parameter this parameter determines the granularity of
the callback by defining the maximum number of times the
callback will be called during the file transfer.
:type torrent: bool
:param torrent: Flag for whether to get a torrent for the file
:type override_num_retries: int
:param override_num_retries: If not None will override configured
num_retries parameter for underlying GET.
:type response_headers: dict
:param response_headers: A dictionary containing HTTP
headers/values that will override any headers associated
with the stored object in the response. See
http://goo.gl/EWOPb for details.
:type version_id: str
:param version_id: The ID of a particular version of the object.
If this parameter is not supplied but the Key object has
a ``version_id`` attribute, that value will be used when
retrieving the object. You can set the Key object's
``version_id`` attribute to None to always grab the latest
version from a version-enabled bucket.
N) r� r� r� �torrentr0 r� r� r r� )�_get_file_internal) r7 r� r� r� r� rQ r0 r� r� r8 r8 r9 �get_file� s /
�z Key.get_filec
s� |d u ri }| j jj} | j jjdkrd| j j_|
pg }
|r"|
�d� � d u r,|s,dti� t� fdd�� p5i D ��} |d u r@| j}|rI|
�d| � |r_|D ]}
|
�d|
tj� ||
�f � qMd �
|
�}
| j d
||
|d � d}|r�| j d u ryd}n| j }| j d u r�|d kr�d
| j
}n |dkr�tt�|| j
|d ��}n |dk r�d }nd}d}|||� z=| D ]8}|�|� |t|�7 }| D ] }| | �|� q�|r�|dkr�||kr� n|d7 }||ks�|d kr�|||� d}q�W n t�y } z|jtjk�rtd|j ��� d }~ww |�r |dk�s|dk�r |dk�r |||� | D ] }| | �� | j|< �q!| j d u �r?|�s?d|v�r?|| _ | �� | | j j_d S )Nrx r rQ r c 3 r rA r8 r r r8 r9 r � r z)Key._get_file_internal.<locals>.<genexpr>z versionId=%sz%s=%s�&r� r� rd r
r z$Out of space for destination file %s�Range) r rD r �appendr r. r0 r �parse�quote�joinr� r/ r� r� r r �writer_ r� r- �errno�ENOSPCr r rg r6 r� )r7 r� r� r� r� rQ r0 r� r� r r� r r& r� r r! r" r{ rU r �er8 r r9 rR � s�
�
�
����$
zKey._get_file_internalc C s | j ||||dd�S )a�
Get a torrent file (see to get_file)
:type fp: file
:param fp: The file pointer of where to put the torrent
:type headers: dict
:param headers: Headers to be passed
:type cb: function
:param cb: a callback function that will be called to report
progress on the upload. The callback should accept two
integer parameters, the first representing the number of
bytes that have been successfully transmitted to S3 and
the second representing the size of the to be transmitted
object.
:type cb: int
:param num_cb: (optional) If a callback is specified with the
cb parameter this parameter determines the granularity of
the callback by defining the maximum number of times the
callback will be called during the file transfer.
T)rQ )rS )r7 r� r� r� r� r8 r8 r9 �get_torrent_file) s zKey.get_torrent_filec C sJ | j dur#|r|j| ||||||d� dS | j|||||||d� dS dS )a�
Retrieve an object from S3 using the name of the Key object as the
key in S3. Write the contents of the object to the file pointed
to by 'fp'.
:type fp: File -like object
:param fp:
:type headers: dict
:param headers: additional HTTP headers that will be sent with
the GET request.
:type cb: function
:param cb: a callback function that will be called to report
progress on the upload. The callback should accept two
integer parameters, the first representing the number of
bytes that have been successfully transmitted to S3 and
the second representing the size of the to be transmitted
object.
:type cb: int
:param num_cb: (optional) If a callback is specified with the
cb parameter this parameter determines the granularity of
the callback by defining the maximum number of times the
callback will be called during the file transfer.
:type torrent: bool
:param torrent: If True, returns the contents of a torrent
file as a string.
:type res_upload_handler: ResumableDownloadHandler
:param res_download_handler: If provided, this handler will
perform the download.
:type response_headers: dict
:param response_headers: A dictionary containing HTTP
headers/values that will override any headers associated
with the stored object in the response. See
http://goo.gl/EWOPb for details.
:type version_id: str
:param version_id: The ID of a particular version of the object.
If this parameter is not supplied but the Key object has
a ``version_id`` attribute, that value will be used when
retrieving the object. You can set the Key object's
``version_id`` attribute to None to always grab the latest
version from a version-enabled bucket.
N)rQ r0 �rQ r0 r� )r rS ) r7 r� r� r� r� rQ r0 �res_download_handlerr� r8 r8 r9 �get_contents_to_fileD s
6
�
��zKey.get_contents_to_filec C s� z$t |d��} | j| |||||||d� W d � n1 s w Y W n
ty1 t�|� � w | jdur]ztj�| j�}
t tj�
|
��} t� | j | | f� W dS ty\ Y dS w dS )aO
Retrieve an object from S3 using the name of the Key object as the
key in S3. Store contents of the object to a file named by 'filename'.
See get_contents_to_file method for details about the
parameters.
:type filename: string
:param filename: The filename of where to put the file contents
:type headers: dict
:param headers: Any additional headers to send in the request
:type cb: function
:param cb: a callback function that will be called to report
progress on the upload. The callback should accept two
integer parameters, the first representing the number of
bytes that have been successfully transmitted to S3 and
the second representing the size of the to be transmitted
object.
:type num_cb: int
:param num_cb: (optional) If a callback is specified with the
cb parameter this parameter determines the granularity of
the callback by defining the maximum number of times the
callback will be called during the file transfer.
:type torrent: bool
:param torrent: If True, returns the contents of a torrent file
as a string.
:type res_upload_handler: ResumableDownloadHandler
:param res_download_handler: If provided, this handler will
perform the download.
:type response_headers: dict
:param response_headers: A dictionary containing HTTP
headers/values that will override any headers associated
with the stored object in the response. See
http://goo.gl/EWOPb for details.
:type version_id: str
:param version_id: The ID of a particular version of the object.
If this parameter is not supplied but the Key object has
a ``version_id`` attribute, that value will be used when
retrieving the object. You can set the Key object's
``version_id`` attribute to None to always grab the latest
version from a version-enabled bucket.
�wb)rQ r0 r` r� N)
r� ra � ExceptionrA �remover) �emailr� � parsedate_tzr� � mktime_tz�utimer ) r7 r' r� r� r� rQ r0 r` r� r� �modified_tuple�modified_stampr8 r8 r9 �get_contents_to_filename� s. 6 � ��
�
��z Key.get_contents_to_filenamec
C s< t � }| j|||||||d� |�� } |dur | �|�} | S )ao
Retrieve an object from S3 using the name of the Key object as the
key in S3. Return the contents of the object as a string.
See get_contents_to_file method for details about the
parameters.
:type headers: dict
:param headers: Any additional headers to send in the request
:type cb: function
:param cb: a callback function that will be called to report
progress on the upload. The callback should accept two
integer parameters, the first representing the number of
bytes that have been successfully transmitted to S3 and
the second representing the size of the to be transmitted
object.
:type cb: int
:param num_cb: (optional) If a callback is specified with the
cb parameter this parameter determines the granularity of
the callback by defining the maximum number of times the
callback will be called during the file transfer.
:type torrent: bool
:param torrent: If True, returns the contents of a torrent file
as a string.
:type response_headers: dict
:param response_headers: A dictionary containing HTTP
headers/values that will override any headers associated
with the stored object in the response. See
http://goo.gl/EWOPb for details.
:type version_id: str
:param version_id: The ID of a particular version of the object.
If this parameter is not supplied but the Key object has
a ``version_id`` attribute, that value will be used when
retrieving the object. You can set the Key object's
``version_id`` attribute to None to always grab the latest
version from a version-enabled bucket.
:type encoding: str
:param encoding: The text encoding to use, such as ``utf-8``
or ``iso-8859-1``. If set, then a string will be returned.
Defaults to ``None`` and returns bytes.
:rtype: bytes or str
:returns: The contents of the file as bytes or a string
r_ N)r ra �getvaluerW )
r7 r� r� r� rQ r0 r� �encodingr� rJ r8 r8 r9 �get_contents_as_string� s 6�
zKey.get_contents_as_stringc C s, | j |d�}|j�||� | j||d� dS )a
Convenience method that provides a quick way to add an email grant
to a key. This method retrieves the current ACL, creates a new
grant based on the parameters passed in, adds that grant to the ACL
and then PUT's the new ACL back to S3.
:type permission: string
:param permission: The permission being granted. Should be one of:
(READ, WRITE, READ_ACP, WRITE_ACP, FULL_CONTROL).
:type email_address: string
:param email_address: The email address associated with the AWS
account your are granting the permission to.
:type recursive: boolean
:param recursive: A boolean value to controls whether the
command will apply the grant to all keys within the bucket
or not. The default value is False. By passing a True
value, the call will iterate through all keys in the
bucket and apply the same grant to each key. CAUTION: If
you have a lot of keys, this could take a long time!
r� N)r� �acl�add_email_grantr� )r7 �
permission�
email_addressr� r� r8 r8 r9 rp s zKey.add_email_grantc C s0 | j |d�}|jj|||d� | j||d� dS )a
Convenience method that provides a quick way to add a canonical
user grant to a key. This method retrieves the current ACL,
creates a new grant based on the parameters passed in, adds that
grant to the ACL and then PUT's the new ACL back to S3.
:type permission: string
:param permission: The permission being granted. Should be one of:
(READ, WRITE, READ_ACP, WRITE_ACP, FULL_CONTROL).
:type user_id: string
:param user_id: The canonical user id associated with the AWS
account your are granting the permission to.
:type display_name: string
:param display_name: An option string containing the user's
Display Name. Only required on Walrus.
r� )� display_nameN)r� ro �add_user_grantr� )r7 rq �user_idr� rs r� r8 r8 r9 rt ) s
�zKey.add_user_grantc C sP t |�tkrt� }|D ] }|�|�� � q |S i }|D ]
}|| ||�� < q|S rA )�type�set�addr� )r7 r �
norm_metadata�kr8 r8 r9 �_normalize_metadataB s �zKey._normalize_metadatac C s~ i }| j D ]}t| |�r t| |�}|r |�dd�}|||�� < q| jj}| jD ]}|�dd�}| j| |d||�� f < q'|S )z�
Extracts metadata from existing URI into a dict, so we can
overwrite/delete from it to form the new set of metadata to apply to a
key.
r r z%s%s)�%_underscore_base_user_settable_fieldsrE r` r� r� rE �metadata_prefixr )r7 r� r �underscore_namerJ �
field_name�prefixr8 r8 r9 �_get_remote_metadataM s
�
�zKey._get_remote_metadatac
C s� | � |�}| � |�}| �� }|�|� |D ] }||v r ||= q| j}i }|D ] }|�d�s2|�d�r=|�dd��dd�} n|} || || < q&|}|j| j| jj| j|||d� d S )Nz x-goog-meta-z x-amz-meta-r )r r� r� )r{ r� r� r �
startswithr� r� r )
r7 �
metadata_plus�metadata_minusr� r� r �h�
src_bucket�rewritten_metadata� rewritten_hr8 r8 r9 �set_remote_metadatac s,
�
�
�zKey.set_remote_metadatac C sT | j jjd| j j| j| j| |dd�}|jdvr(| j jj}|�|j|j|� � ��dS )a
Restore an object from an archive.
:type days: int
:param days: The lifetime of the restored object (must
be at least 1 day). If the object is already restored
then this parameter can be used to readjust the lifetime
of the restored object. In this case, the days
param is with respect to the initial time of the request.
If the object has not been restored, this param is with
respect to the completion time of the request.
�POST�restore)r� r� r� )r� �� N)
r rD r� r � RestoreBodyr� rE r� r� r� )r7 �daysr� r� rE r8 r8 r9 r� | s
�
��z Key.restore)NN)F)Nr NN)r� NNN)r )NT)NFFFTrA )
r� NTFNFNNFF)NNr� NFN)NNr� NFNN)NTNr� NFNN) NTNr� NNFNFNF)NTNr� NNFF)NNr� FNNN) NNr� FNNNNN)NNr� )V�__name__�
__module__� __qualname__�__doc__r" r� r� �config�getintr� rw r� r| �frx r� r� r: r@ rC �propertyrE rG rK r� rP rS r rY r\ rh rb rc r� ri rm rq rv r� r� r� r� r� r� r� r� �__next__r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r r� r r r@ rK rN rP rS rR r^ ra rk rn rp rt r{ r� r� r� r8 r8 r8 r9 r 0 s
�
�
6
�
�%
�<
�P
�+
�
M
7
�X
� 3
�C
�B
�6
�
M
�@
�I
�
A
�
�r )#� email.utilsre r[ r2 r/ rA r� rH rN r r �
boto.utilsr� � boto.compatr r r r �boto.exceptionr r r
�
boto.providerr �boto.s3.keyfiler � boto.s3.userr
r r r r r �objectr r8 r8 r8 r9 �<module> s.