|
Server : Apache System : Linux iad1-shared-b8-43 6.6.49-grsec-jammy+ #10 SMP Thu Sep 12 23:23:08 UTC 2024 x86_64 User : dh_edsupp ( 6597262) PHP Version : 8.2.26 Disable Function : NONE Directory : /lib/python3/dist-packages/duplicity/__pycache__/ |
Upload File : |
o
�~�_j � @ s� d Z ddlmZ ddlmZ e�� ddlmZ ddlmZ ddlm Z ddlm
Z
ddlm Z dd l Z dd l
Z
dd
lmZ dd lmZ dd lmZ dd
lT dd
lT ddlmZ d ad ZG dd� de�Zdd� Zdd� Zdd� Zdd� Z d=dd�Z d=dd �Z d>d d �Z d=d d �Z d!d"� Z!d#d$� Z"d%d&� Z#d'd(� Z$d)d*� Z%G d+d,� d,e �Z&G d-d.� d.e �Z'G d/d0� d0e �Z(G d1d2� d2e �Z)G d3d4� d4e)�Z*G d5d6� d6e)�Z+G d7d8� d8e)�Z,d9d:� Z-d;d<� Z.d S )?a
Functions for producing signatures and deltas of directories
Note that the main processes of this module have two parts. In the
first, the signature or delta is constructed of a ROPath iterator. In
the second, the ROPath iterator is put into tar block form.
� )�division)�standard_library)�map)�next)�str)�range)�objectN)�
statistics)�util)�config)�*)�progressc @ s e Zd ZdS )�DiffDirExceptionN)�__name__�
__module__� __qualname__� r r �3/usr/lib/python3/dist-packages/duplicity/diffdir.pyr 6 s r c C s t | �S )z)
Alias for SigTarBlockIter below
)�SigTarBlockIter�� path_iterr r r �DirSig: s r c C s t | t�d��S )a
Return a tarblock full backup of items in path_iter
A full backup is just a diff starting from nothing (it may be less
elegant than using a standard tar file, but we can be sure that it
will be easy to split up the tar and make the volumes the same
sizes).
� )�DirDelta�io�StringIOr r r r �DirFullA s r c C s t | t�d�|�S )zN
Return full backup like above, but also write signature to sig_outfp
r )�DirDelta_WriteSigr r )r � sig_outfpr r r �DirFull_WriteSigM � r c C s^ t �� at|t�rtdd� |D ��}nt|�}t| |�}tj s'tj
r+t
j � � s+t
|�S t|�S )z�
Produce tarblock diff given dirsig_fileobj_list and pathiter
dirsig_fileobj_list should either be a tar fileobj or a list of
those, sorted so the most recent is last.
c S � g | ]}t |��qS r ��sigtar2path_iter��.0�xr r r �
<listcomp>^ � z DirDelta.<locals>.<listcomp>)r �StatsDeltaProcess�stats�
isinstance�list�combine_path_itersr# �get_delta_iterr �dry_runr
�tracker�has_collected_evidence�DummyBlockIter�DeltaTarBlockIter)r �dirsig_fileobj_list�sig_iter�
delta_iterr r r r T s
�
r c C sJ |r|� � }n |r|� � }nJ d��t�td�t�| �t�|�f � dS )zA
Called by get_delta_iter, report error in getting delta
r z)Both new and sig are None for some reason� Error %s getting delta for %sN)�get_relative_path�log�Warn�_r
�uexc�fsdecode)�exc�new_path�sig_path�sig_tar� index_stringr r r �delta_iter_error_handleri s
�rC c s^ | sJ ��r
| � � �| j� | �� }t�td�| |f � � ��fdd�}| �� rV|rV|�� rV|jdkrVd|_|�d�}t | �d��}�rLt
||| � � �}|� t
�||�� nLd|_�rutjjdkrmd t�d
�� �� �_nd d
�� � �_| �� s��r����� tr�t j|� � 7 _nt | �d��}�r�t
||| � � �}|� |� | �|� | jj|j_|S ) zj
Return new delta_path which, when read, writes sig to sig_fileobj,
if sigTarFile is not None
zGetting delta of %s and %sc sT t | ��_tjjdkrdt�d�� �� �_ndd�� � �_�� �t
� | �� dS )zG
Callback activated when FileWithSignature read to end
� �
signature/� /�
signature/N) �len�size�sys� version_info�majorr
r= �join�name�addfiler �BytesIO)�
sig_string��index�
sigTarFile�tir r �callback� s
z get_delta_path.<locals>.callback� signature�diff�rb�snapshotrD � snapshot/rF � snapshot/) � get_tarinforS �
get_ropathr9 �Debugr; �isreg�difftype�open�FileWithReadCounter�FileWithSignature�getsize�
setfileobj�librsync� DeltaFilerJ rK rL r
r= rM rN rO r* �SourceFileSize� copy_attribs�stat�st_size)r? r@ rT �
delta_pathrV � old_sigfp�newfpr rR r �get_delta_pathx sH
�
��
rp c C s� | j dkr'|r|r|�|� t�td�t�| �� � tjj t�
| �� �� dS |r0|r0|� |� t�td�t�| �� � tjj t�
| �� �� dS )zI
Look at delta path and log delta. Add stats if new_path is set
rZ zA %szM %sN)
ra � add_new_filer9 �Infor; r
r= r8 �InfoCode�
diff_file_new�escape�add_changed_file�diff_file_changed)rm r? r* r r r �log_delta_path� s"
� �
� �rx c c s� � t | |�}|rt�d|�}nd}|D ]�\}}t�td�|o#t�|j�|o*t�|j�f � |r3|js�|r�|� � r�|jdkr�t�
td�t� |� � � tj
jt�|� � �� |r|t|j��� }tjjdkrndt�|j� |_n dd �|j� |_|�|� t�|� t|j�V q|r�||kr�t�t t |||f�}|r�t ||t� |V qt j d
7 _ qt� |� qt�!� |r�|�!� dS dS ) a1
Generate delta iter from new Path iter and sig Path iter.
For each delta path of regular file type, path.difftype with be
set to "snapshot", "diff". sig_iter will probably iterate ROPaths
instead of Paths.
If sig_fileobj is not None, will also write signatures to sig_fileobj.
�wNzComparing %s and %sr zD %srD �deleted/s deleted/rF � )"�
collate2itersr
� make_tarfiler9 r_ r; �uindexrS �type�existsrr r= r8 rs �diff_file_deletedru �ROPathr] rJ rK rL rN rM rO r* �add_deleted_file�robust�check_common_errorrC rp rx �Errors�add_unchanged_file�close) �new_iterr5 � sig_fileobj�collatedrT r? r@ rU rm r r r r. � sP �
�
� �
� � �r. c c s � t �d| �}d|_|D ]n}t �|�}dD ]}|�|�r-|t|�d� |dd� }} nqtd|f ��tjj dkrFt
t � |�� d��}nt
|� d��}|d sW|dd� }t
|�}||_|d ksf|d
krw|�|� |�� rw|�|�|�� |V q | �� dS ) zK
Convert signature tar file object open for reading into path iter
�rr{ )rE r[ rz N���zBad tarinfo name %srD rF rW rZ )r
r} �debug�get_tarinfo_name�
startswithrH r rJ rK rL �tuple�fsencode�splitr� ra �init_from_tarinfor` rf � extractfiler� ) � sigtarobj�tf�tarinfo�tiname�prefixrN ra rS �ropathr r r r# � s0 �
�
r# c c s� � d\}} |s-zt | �}W n ty) |r d|fV |D ]}d|fV q Y dS w |j}|sTzt |�}W n tyP |rC|dfV | D ]}|dfV qEY dS w |j}||k r`|dfV d}n||krn||fV d\}}nd|fV d}q)a4
Collate two iterators.
The elements yielded by each iterator must be have an index
variable, and this function returns pairs (elem1, elem2), (elem1,
None), or (None, elem2) two elements in a pair will have the same
index, and earlier indicies are yielded later than later indicies.
�NNr{ N�r �
StopIterationrS )�riter1�riter2�relem1�relem2�index1�index2r r r r| sD �
�
�
�r| c # s| � �dd� ��� � �fdd�� � fdd�}dd� t� ttt�����D �}|r<|�� |d d V ||� |s)dS dS )
a�
Produce new iterator by combining the iterators in path_iter_list
This new iter will iterate every path that is in path_iter_list in
order of increasing index. If multiple iterators in
path_iter_list yield paths with the same index, combine_path_iters
will discard all paths but the one yielded by the last path_iter.
This is used to combine signature iters, as the output will be a
full up-to-date signature iter.
Nc s2 zt � | �}W n
ty Y dS w |j| |fS )zI
Represent the next element as a triple, to help sorting
Nr� )�
iter_index�path)�path_iter_listr r �
get_tripleV s � z&combine_path_iters.<locals>.get_triplec sr | d d }d}|t | �k r7| | }|d |kr-� |d �}|r)|| |< |d7 }n| |= ndS |t | �k sdS dS )zK
Update all elements with path_index same as first element
r r{ N)rH )� triple_list�
path_indexr� �
old_triple�
new_triple)r� r r �refresh_triple_list` s
�z/combine_path_iters.<locals>.refresh_triple_listc S s g | ]}|r|�qS r r r$ r r r r' r r( z&combine_path_iters.<locals>.<listcomp>r � )�reverser r, r rH �sort)r� r� r� r )r� r� r r- G s �
�r- c C sV t �� at|t�rt|�}nt|�}t| ||�}tj s#tj
r't
j � � s't
|�S t|�S )a
Like DirDelta but also write signature into sig_fileobj
Like DirDelta, sig_infp_list can be a tar fileobj or a sorted list
of those. A signature will only be written to newsig_outfp if it
is different from (the combined) sig_infp_list.
)r r) r* r+ r, �get_combined_path_iterr# r. r r/ r
r0 r1 r2 r3 )r �
sig_infp_list� newsig_outfp�
sig_path_iterr6 r r r r y s
r c C s t dd� | D ��S )zI
Return path iter combining signatures in list of open sig files
c S r! r r" r$ r r r r' � r( z*get_combined_path_iter.<locals>.<listcomp>)r- )r� r r r r� � r r� c @ s* e Zd ZdZdd� Zd
dd�Zdd� Zd S ) rc zH
File-like object which also computes amount read as it is read
c C �
|| _ dS )z FileWithReadCounter initializerN)�infile)�selfr� r r r �__init__� s
z FileWithReadCounter.__init__r� c
C sz z| j �|�}W n' ty/ } zd}t�td�t�|�t�| j j �f � W Y d }~nd }~ww t
r;t
j t |�7 _ |S )N� r7 )
r� �read�IOErrorr9 r: r; r
r< r= rN r* ri rH )r� �length�buf�exr r r r� � s
���zFileWithReadCounter.readc C s
| j �� S �N)r� r� �r� r r r r� � s
zFileWithReadCounter.closeN�r� )r r r �__doc__r� r� r� r r r r rc � s
rc c @ s. e Zd ZdZdZdd� Zd dd�Zdd � Zd
S ) rd zF
File-like object which also computes signature as it is read
i � c G s. ||| _ | _t�t|��| _d| _|| _dS )au
FileTee initializer
The object will act like infile, but whenever it is read it
add infile's data to a SigGenerator object. When the file has
been read to the end the callback will be called with the
calculated signature, and any extra_args if given.
filelen is used to calculate the block size of the signature.
N)r� rV rg � SigGenerator�get_block_size�sig_gen�activated_callback�
extra_args)r� r� rV �filelenr� r r r r� � s
zFileWithSignature.__init__r� c C s | j �|�}| j�|� |S r� )r� r� r� �update)r� r� r� r r r r� � s zFileWithSignature.readc C sJ | j s | �| j�r | �| j�s d| _ | j| j�� g| j�R � | j�� S )Nr{ ) r� r� � blocksizerV r� �getsigr� r� r� r� r r r r� � s �
zFileWithSignature.closeNr� )r r r r� r� r� r� r� r r r r rd � s
rd c @ � e Zd ZdZdd� ZdS )�TarBlockz5
Contain information to add next file to tar
c C s || _ || _dS )z8
TarBlock initializer - just store data
N)rS �data)r� rS r� r r r r� � s
zTarBlock.__init__N)r r r r� r� r r r r r� � � r� c @ sr e Zd ZdZdd� Zd dd�Zdd� Zd d
� Zd d � Zd
d� Z dd� Z
dd� Z dd� Z dd� Z
dd� Zdd� ZdS ) � TarBlockIterz�
A bit like an iterator, yield tar blocks given input iterator
Unlike an iterator, however, control over the maximum size of a
tarblock is available by passing an argument to next(). Also the
get_footer() is available.
c C s@ || _ d| _d| _d| _d| _d| _d| _d| _d| _d| _ dS )z*
TarBlockIter initializer
r FN)
�
input_iter�offset�process_waiting�process_next_vol_number�previous_index�previous_block�
remember_next�remember_value�remember_block� queued_data)r� r� r r r r� � s
zTarBlockIter.__init__r� c C s\ t |�|_|jdtjd�}t|jtj�\}}|dkr"dtj| }nd}t|d|||f �S )z<
Make tarblock out of tarinfo and file data
�replace)�errors�encodingr � r� s %s%s%s) rH rI �tobufr �
fsencoding�divmod�tarfile� BLOCKSIZEr� )r� rS r� � file_data�headers�blocks� remainder� filler_datar r r �tarinfo2tarblock� s
z TarBlockIter.tarinfo2tarblockc C s | j rJ �t dS )z?
Turn next value of input_iter into a TarBlock
N�r� �XXX)r� �valr r r �process s
zTarBlockIter.processc C s | j sJ �t dS )z�
Get more tarblocks
If processing val above would produce more than one TarBlock,
get the rest of them by calling process_continue.
Nr� r� r r r �process_continued s
z TarBlockIter.process_continuedc C s� | j dur
| j }d| _ |S | jr| �� }n| �t| j��}| j}| jt|j �7 _|j
| _ || _ | j
r>|j
| _|| _d| _
|S )z5
Return next block and update offset
NF)r� r� r� r� r r� r� r� rH r� rS r� r� r� r� r� )r� �result� block_numberr r r �__next__ s
zTarBlockIter.__next__c C s dS )Ni r r� r r r �
get_read_size+ s zTarBlockIter.get_read_sizec C � | j | jfS )zM
Return index of last tarblock, or None if no previous index
)r� r� r� r r r �get_previous_index2 � z TarBlockIter.get_previous_indexc C r� )zW
Next time next() is called, we will return data instead of processing
N)r� )r� r� r r r �queue_index_data8 s
z TarBlockIter.queue_index_datac C s d| _ d| _d| _dS )zL
When called, remember the index of the next block iterated
TN)r� r� r� r� r r r �remember_next_index> s
z TarBlockIter.remember_next_indexc C r� )zD
Retrieve index remembered with remember_next_index
)r� r� r� r r r � recall_indexF r� zTarBlockIter.recall_indexc C s& t | jtj�\}}d| _dtj| S )zA
Return closing string for tarfile, reset offset
r r� )r� r� r� �
RECORDSIZE)r� r� r� r r r �
get_footerL s zTarBlockIter.get_footerc C s | S r� r r� r r r �__iter__T s zTarBlockIter.__iter__N)r� )r r r r� r� r� r� r� r� r� r� r� r� r� r� r� r r r r r� � s
r� c @ r� )r2 z0
TarBlockIter that does no file reading
c C sd |� � }|j}|jr
|js| �||�S tr,t jd7 _t j|�� 7 _t �
dtj� | �||�S )z7
Get a fake tarblock from delta_ropath
r{ N) r] rS r �fileobjr� r* � SourceFilesri re r9 �Progress)r� � delta_ropathrU rS r r r r� \ s zDummyBlockIter.processN�r r r r� r� r r r r r2 X r� r2 c @ r� )r zK
TarBlockIter that yields blocks of a signature tar from path_iter
c C s� |� � }|�� r;t�|�d�t|�� ��}|�� }|�� dd� |j
� |_ t j
jdkr3t�|j �|_ | �|j
||�S dd� |j
� |_ t j
jdkrQt�|j �|_ | �|j
|�S )z@
Return associated signature TarBlock from path
rY rG rF rD r\ )r] r` rg �SigFilerb r� re r� r� rM rS rN rJ rK rL r
r= r� )r� r� rU �sfp�sigbufr r r r� t s
� zSigTarBlockIter.processNr r r r r r p r� r c @ s( e Zd ZdZdd� Zdd� Zdd� ZdS ) r3 z�
TarBlockIter that yields parts of a deltatar file
Unlike SigTarBlockIter, the argument to __init__ is a
delta_path_iter, so the delta information has already been
calculated.
c C s dd� }|� � }|j}|jr|js,|js||d� n |jdks!J �||d� | �||�S |�d�}| �|�\}}trCt j t
|�7 _ |rf|jdkrP||d� n|jdkr[||d� nJ d��| �|||�S d |j|j f }|d
|_ || _ || _
|| _d | _d | _| �|||�S )
z2
Get a tarblock from delta_ropath
c S s, | j dkr |d | _ dS d|| j f | _ dS )z(Add prefix to the name of a tarinfo file�.�/z%s/%sN)rN )r� r� r r r �
add_prefix� s
z-DeltaTarBlockIter.process.<locals>.add_prefix�deletedrZ rY rX r zUnknown difftypezmultivol_%s/%sz/1r{ r� )r] rS r r� ra r� rb �get_data_blockr* � RawDeltaSizerH rN �process_prefix�
process_fp�process_ropathr� r� ) r� r� r rU rS �fpr� �
last_block� full_namer r r r� � s8
zDeltaTarBlockIter.processc C s> | � � }|�|�}t|�|k r|�� rtd��|dfS |dfS )zH
Return pair (next data block, boolean last data block)
zError closing fileTF)r� r� rH r� r )r� r
� read_sizer� r r r r � s
z DeltaTarBlockIter.get_data_blockc C s� | j sJ �| j}|�� |j}}d| j| jf |_| �| j�\}}t r,t j
t |�7 _
|r>d| _d| _d| _d| _ d| _n| jd7 _| � |||�S )zA
Return next volume in multivol diff or snapshot
z%s/%dNr{ )
r� r r] rS r
r� rN r r r* r rH r� )r� r� rU rS r� r r r r r� � s
z#DeltaTarBlockIter.process_continuedN)r r r r� r� r r� r r r r r3 � s
/
r3 c C s� t |t�r t|jd�}nt |tdjf�rt|d�}n|}| D ]}|�|j� q |�| �� � |� � r4J �t |t�r?|�
� dS dS )z<
Write block_iter to filename, path, or file object
�wbr N) r+ �Pathrb rN r � __class__�writer� r� r� �setdata)�
block_iter�out_objr
�blockr r r �write_block_iter� s
�r c C s( | dk rdS t | d �d }t|tj�S )z�
Return a reasonable block size to use on files of length file_len
If the block size is too big, deltas will be bigger than is
necessary. If the block size is too small, making deltas and
patching can take a really long time.
i � i )�int�minr �
max_blocksize)�file_len�file_blocksizer r r r� � s r� r� r� )/r� �
__future__r �futurer �install_aliases�builtinsr r r r r r rJ � duplicityr r
r �duplicity.path�duplicity.lazyr
r* r0 � Exceptionr r r r r rC rp rx r. r# r| r- r r� rc rd r� r� r2 r r3 r r� r r r r �<module> sT
6
8!,2% {Z