1jxk d dl mZ d dlZd dlZd dlZd dlZd dlmZmZm Z m
Z
m Z d dl m
Z
mZmZmZmZ d dlmZmZ d dlmZ dZ d dlmZ d Zn # e$ r Y nw xY wd
Z G d d e Z G d
de Z G d d Z G d de Z G d de Z! G d de Z" G d d Z# G d d Z$dS ) )BytesION)msb_size stream_copyapply_delta_dataconnect_deltas delta_types)allocate_memory LazyMixinmake_shawriteclose) NULL_BYTE
BYTE_SPACE) force_bytesF) apply_deltaT) DecompressMemMapReaderFDCompressedSha1WriterDeltaApplyReader
Sha1WriterFlexibleSha1WriterZippedStoreShaWriterr FDStream
NullStreamc e Zd ZdZdZdZddZd Zd Zd Z e
dd
Z d Z d Z
d
Z eedd fdZddZdS )r a Reads data in chunks from a memory map and decompresses it. The client sees
only the uncompressed data, respective file-like read calls are handling on-demand
buffered decompression accordingly
A constraint on the total size of bytes is activated, simulating
a logical file within a possibly larger physical memory area
To read efficiently, you clearly don't want to read individual bytes, instead,
read a few kilobytes at least.
**Note:** The chunk-size should be carefully selected as it will involve quite a bit
of string copying due to the way the zlib is implemented. Its very wasteful,
hence we try to find a good tradeoff between allocation time and number of
times we actually allocate. An own zlib implementation would be good here
to better support streamed reading - it would only need to keep the mmap
and decompress it into chunks, that's all ... ) _m_zip_buf_buflen_br_cws_cwe_s_close_cbr_phii Nc || _ t j | _ d| _ d| _ ||| _ d| _ d| _ d| _ d| _
d| _ || _ dS )z|Initialize with mmap for stream reading
:param m: must be content data - use new if you have object data and no sizeNr F)
r zlib
decompressobjr r r r" r r r! r$ r% r# )selfmclose_on_deletionsizes ^/builddir/build/BUILD/cloudlinux-venv-1.0.11/venv/lib/python3.11/site-packages/gitdb/stream.py__init__z DecompressMemMapReader.__init__E sd &(( DG ' c > |dk sJ | d S )Nr" )_parse_header_infor) attrs r- _set_cache_z"DecompressMemMapReader._set_cache_U s) t||||
!!!!!r/ c . | d S N)r
r) s r- __del__z DecompressMemMapReader.__del__[ s
r/ c | d}|| _ | | }| t }|d| t
\ }}t
| }|| _ d| _ |dz
}t ||d | _ t | |z
| _ d| _ ||fS )zIf this stream contains object data, parse the header info and skip the
stream to a point where each read will yield object content
:return: parsed type_string, sizei Nr T)
r" readfindr splitr intr r r lenr r% )r) maxbhdrhdrendtypr, s r- r1 z)DecompressMemMapReader._parse_header_info^ s iioo)$$L&&z22 T4yy ! CL)) 3xx&( Dyr/ Fc \ t ||d }| \ }}|||fS )a Create a new DecompressMemMapReader instance for acting as a read-only stream
This method parses the object header from m and returns the parsed
type and size, as well as the created stream instance.
:param m: memory map on which to operate. It must be object data ( header + contents )
:param close_on_deletion: if True, the memory map will be closed once we are
being deletedr )r r1 )r) r* r+ instrC r, s r- newzDecompressMemMapReader.new{ s7 &a):A>>++-- TD$ r/ c | j S )z8:return: random access compatible data we are working on)r r7 s r- datazDecompressMemMapReader.data s wr/ c | j r7t | j d r| j d| _ dS dS )zClose our underlying stream of compressed bytes if this was allowed during initialization
:return: True if we closed the underlying stream
:note: can be called safely
r
FN)r# hasattrr r
r7 s r- r
z DecompressMemMapReader.close sF
; tw((
DKKK r/ c " | j | j k r| j j sd| _ t | j d rT| j j t j k r9| t j
| j j t j k 9nq| j j se| j t | j
k rH| t j
| j j s | j t | j
k H| j | _ | j S )z
:return: number of compressed bytes read. This includes the bytes it
took to decompress the header ( if there was one )r status)r r" r unused_datarJ rL r' Z_OKr; mmapPAGESIZEr$ r? r r7 s r- compressed_bytes_readz,DecompressMemMapReader.compressed_bytes_read s 0 8tw ty'< DHty(++
-i&$)33IIdm,,, i&$)33
)/ -DITW 4M4MIIdm,,, )/ -DITW 4M4M wDH
yr/ SEEK_SETr c |dk s|t t dd k rt d t j | _ dx| _ x| _ x| _ | _ | j
r d| _
| ` dS dS )zgAllows to reset the stream to restart reading
:raise ValueError: If offset and whence are not 0r rR Can only seek to position 0FN) getattros
ValueErrorr' r( r r r r! r$ r% r" r) offsetwhences r- seekzDecompressMemMapReader.seek s Q;;&GB
A$>$>>> :;; ; &(( 7888498ty49 9 DI r/ c R |dk r| j | j z
}n t || j | j z
}|dk rdS d}| j r| j |k r<| j | }| xj |z c_ | xj |z
c_ |S | j }|| j z }| xj | j z
c_ d| _ d | _ | j j }|r,| j t | z
| _
| j
|z | _ n | j
}| j | _
||z | _ | j | j
z
dk r| j
dz | _ | j | j
| j }| j
t | z | _ | j || }t j dv r*t j dk st | j j }n3t | j j t | j j z }| xj t | |z
z
c_ | xj t | z
c_ |r||z }|r[t | t | z
|k r8| j | j k r(|| |t | z
z
}|S )Nr: r r/ )z1.2.7z1.2.5darwin)r" r minr r r; r unconsumed_tailr! r? r r
decompressr' ZLIB_VERSIONsysplatformrM r$ )r) r, dattailcwsindatadcompdatunused_datalens r- r; zDecompressMemMapReader.read s !887TX%DDtTWtx /00D 1993 9
!|t##innT** $ D
inn&& $DL( y( # CII-DI D(DII)C DId
DI 9ty 1 $ $ A
DI 49,- IF + 9''55 2 2 23<8;S;S !: ; ;NN !: ; ;c$)BW>X>X XN
S[[>11 CMM! &X~H 8X S1T99dh>P>P $X "677 7Hr/ r6 F)r\ )__name__
__module__ __qualname____doc__ __slots__
max_read_sizer. r4 r8 r1 classmethodrF rH r
rQ rU rV r[ r; r/ r- r r . s : : !I M( ( ( ( " " " :
[
- - -b #*'"j!"<"< i i i i i ir/ r c e Zd ZdZdZdZd Zd Zd Ze seZ
neZ
ddZ e e
d d fd
Zed Zed Zed
Zed ZdS )r a A reader which dynamically applies pack deltas to a base object, keeping the
memory demands to a minimum.
The size of the final object is only obtainable once all deltas have been
applied, unless it is retrieved from a pack index.
The uncompressed Delta has the following layout (MSB being a most significant
bit encoded dynamic size):
* MSB Source Size - the size of the base against which the delta was created
* MSB Target Size - the size of the resulting data after the delta was applied
* A list of one byte commands (cmd) which are followed by a specific protocol:
* cmd & 0x80 - copy delta_data[offset:offset+size]
* Followed by an encoded offset into the delta data
* Followed by an encoded size of the chunk to copy
* cmd & 0x7f - insert
* insert cmd bytes from the delta buffer into the output stream
* cmd == 0 - invalid operation ( or error in delta stream )
)_bstream _dstreams
_mm_target_sizer ic t | dk s
J d |d | _ t |dd | _ d| _ dS )zInitialize this instance with a list of streams, the first stream being
the delta to apply on top of all following deltas, the last stream being the
base object onto which to apply the deltasr: z+Need at least one delta and one base streamr\ Nr )r? rv tuplerw r )r) stream_lists r- r. zDeltaApplyReader.__init__h sT ; !###%R####B
{3B3 /00r/ c p t | j dk r| | S t | j }| dk r d| _ t
d | _ d S | | _ t
| j | _ t
| j j }t | j j |j | j j dt j z | j j }| || | j d d S )Nr: r )r? rw _set_cache_brute_r rboundry r rx rv r, r r; r rO rP applyr[ )r) r3 dclbbufr s r- _set_cache_too_slow_without_cz.DeltaApplyReader._set_cache_too_slow_without_cr s t~ ! # #))$// /
T^,, ::<<1 DJ -a 0 0DO F ZZ\\
)$*55 t} 122DM&
DM4FdmH[\\\ % $ Q r/ c N t }d}| j D ]m}| d }t | \ }}t || \ }}| ||d |||f t || }n| j j } |}t | j dk rt | | x} }t | }
t | j j |
j | dt j
z t | } d} t t | t | j D ]\ \ }
}}}}t |j |z
}| |
t |j |j |j dt j
z dt! v rt# |
|| n%t% |
||t | | j | |
} }
|
d | d |} |
| _ | | _ dS )z*If we are here, we apply the actual deltasr i Nr: r~
c_apply_delta)listrw r; r appendmaxrv r, r? r r r rO rP zipreversedglobalsr r r[ rx ry )r) r3 buffer_info_listmax_target_sizedstreambufrY src_size target_size base_sizer tbuffinal_target_sizedbufddatas r- r z"DeltaApplyReader._set_cache_brute_ s%
66~ @ @G,,s##C '}} FH"*3"7"7 FK # #S\68[$Q R R R !/; ? ?OO
M& % t~ " "&))_&E&E EI
y))DM&
IsT]?RSSS {++ !>A(K[B\B\^fgkgu^v^v>w>w , , : 1T68[7 $GL6$9::E KK ek7<t}AT U U U '))++ dE40000 xE
DJOOO
t$D IIaLLL IIaLLL + &
r/ r c | j | j z
}|dk s||k r|}| j | }| xj t | z
c_ |S )Nr: )ry r rx r; r? )r) countblrH s r- r; zDeltaApplyReader.read sX
Z$(
" 199
E ##E** CII r/ rR c |dk s|t t dd k rt d d| _ | j d dS )zhAllows to reset the stream to restart reading
:raise ValueError: If offset and whence are not 0r rR rT N)rU rV rW r rx r[ rX s r- r[ zDeltaApplyReader.seek sU Q;;&GB
A$>$>>> :;; ; Q r/ c t | dk rt d |d j t v r t d|d j z | | S )a
Convert the given list of streams into a stream which resolves deltas
when reading from it.
:param stream_list: two or more stream objects, first stream is a Delta
to the object that you want to resolve, followed by N additional delta
streams. The list's last stream must be a non-delta stream.
:return: Non-Delta OPackStream object whose stream can be used to obtain
the decompressed resolved data
:raise ValueError: if the stream list cannot be handled zNeed at least two streamsr\ zNCannot resolve deltas if there is no base object stream, last one was type: %s)r? rW type_idr type)clsr| s r- rF zDeltaApplyReader.new ss { a 899 9 r? "k 1 1 `cnoqcrcwwy y
y s; r/ c | j j S r6 )rv r r7 s r- r zDeltaApplyReader.type s
}!!r/ c | j j S r6 )rv r r7 s r- r zDeltaApplyReader.type_id s
}$$r/ c | j S )z3:return: number of uncompressed bytes in the stream)ry r7 s r- r, zDeltaApplyReader.size s zr/ Nr )rm rn ro rp rq k_max_memory_mover. r r has_perf_modr4 r; rU rV r[ rs rF propertyr r r, rt r/ r- r r B s 0I * DH' H' H'V 4' 3 #*'"j!"<"< [ 4 " " X" % % X% X r/ r c * e Zd ZdZdZd Zd ZddZdS ) r zpSimple stream writer which produces a sha whenever you like as it degests
everything it is supposed to writesha1c , t | _ d S r6 )r r r7 s r- r. zSha1Writer.__init__2 s JJ r/ c T | j | t | S )z{:raise IOError: If not all bytes could be written
:param data: byte object
:return: length of incoming data)r updater? r) rH s r- r zSha1Writer.write7 s&
4yyr/ Fc j |r| j S | j S )z]:return: sha so far
:param as_hex: if True, sha will be hex-encoded, binary otherwise)r hexdigestdigest)r) as_hexs r- shazSha1Writer.shaD s4 )9&&(( (y !!!r/ Nrl )rm rn ro rp rq r. r r rt r/ r- r r , sU * *I
" " " " " "r/ r c " e Zd ZdZdZd Zd ZdS )r zZWriter producing a sha1 while passing on the written bytes to the given
write functionwriterc H t | || _ d S r6 )r r. r )r) r s r- r. zFlexibleSha1Writer.__init__T s! D!!! r/ c f t | | | | d S r6 )r r r r s r- r zFlexibleSha1Writer.writeX s0 t$$$ Dr/ N)rm rn ro rp rq r. r rt r/ r- r r N sA I r/ r c T e Zd ZdZdZd Zd Zd Zd Z e e
dd fd Z d
Z d S ) r z=Remembers everything someone writes to it and generates a sha)r r c t | t | _ t j t j | _ d S r6 )r r. r r r' compressobj Z_BEST_SPEEDr r7 s r- r. z ZippedStoreShaWriter.__init__b s: D!!!99#D$566r/ c , t | j | S r6 )rU r r2 s r- __getattr__z ZippedStoreShaWriter.__getattr__g s tx&&&r/ c t | | }| j | j | |S r6 )r r r r compress)r) rH alens r- r zZippedStoreShaWriter.writej s@ d++ tx((../// r/ c h | j | j d S r6 )r r r flushr7 s r- r
zZippedStoreShaWriter.closep s( tx~~''(((((r/ rR r c |dk s|t t dd k rt d | j d dS )z`Seeking currently only supports to rewind written data
Multiple writes are not supportedr rR rT N)rU rV rW r r[ rX s r- r[ zZippedStoreShaWriter.seeks sL Q;;&GB
A$>$>>> :;; ;
ar/ c 4 | j S )zA:return: string value from the current stream position to the end)r getvaluer7 s r- r z ZippedStoreShaWriter.getvalue{ s x """r/ N)
rm rn ro rp rq r. r r r
rU rV r[ r rt r/ r- r r ] s GG I7 7 7
' ' ' ) ) ) #*'"j!"<"< # # # # #r/ r c H e Zd ZdZdZ ed Z fdZd Zd Z xZ
S )r zDigests data written to it, making the sha available, then compress the
data and write it to the file descriptor
**Note:** operates on raw file descriptors
**Note:** for this to work, you have to use the close-method of this instance)fdr r z+Failed to write all bytes to filedescriptorc t || _ t j t j | _ d S r6 )superr. r r' r r r )r) r __class__s r- r. z FDCompressedSha1Writer.__init__ s9
#D$566r/ c | j | | j | }t | j | }|t
| k r| j t
| S )zZ:raise IOError: If not all bytes could be written
:return: length of incoming data)r r r r r r r? exc)r) rH cdata
bytes_writtens r- r z FDCompressedSha1Writer.write sd
!!$'' dgu--
CJJ & &(N4yyr/ c | j }t | j | t | k r| j t
| j S r6 )r r r r r? r r
)r) remainders r- r
z FDCompressedSha1Writer.close sE HNN$$ ) $ $I 6 6(NTW~~ r/ ) rm rn ro rp rq IOErrorr r. r r
__classcell__)r s @r- r r s{ U U
&I '?
@
@C7 7 7 7 7
r/ r c < e Zd ZdZdZd Zd Zd dZd Zd Z d Z
d
S ) r zA simple wrapper providing the most basic functions on a file descriptor
with the fileobject interface. Cannot use os.fdopen as the resulting stream
takes ownership_fd_posc " || _ d| _ d S Nr r )r) r s r- r. zFDStream.__init__ s r/ c t | xj t | z
c_ t j | j | d S r6 )r r? rV r r r s r- r zFDStream.write s2 SYY
4 r/ r c |dk r$t j | j }t j | j | }| xj t | z
c_ |S r )rV pathgetsize _filepathr; r r r? )r) r bytess r- r; z
FDStream.read sP A::GOODN33E %(( SZZ r/ c | j S r6 )r r7 s r- filenozFDStream.fileno s xr/ c | j S r6 )r r7 s r- tellz
FDStream.tell s
yr/ c . t | j d S r6 )r
r r7 s r- r
zFDStream.close s
dhr/ Nr ) rm rn ro rp rq r. r r; r r r
rt r/ r- r r s I ! ! ! r/ r c : e Zd ZdZ e ZddZd Zd ZdS )r zVA stream that does nothing but providing a stream interface.
Use it like /dev/nullr c dS )N rt )r) r, s r- r; zNullStream.read s rr/ c d S r6 rt r7 s r- r
zNullStream.close s r/ c t | S r6 )r? r s r- r zNullStream.write s 4yyr/ Nr ) rm rn ro rp r{ rq r; r
r rt r/ r- r r s[ I
r/ r )%ior rO rV rd r' gitdb.funr r r r r
gitdb.utilr r
r r r
gitdb.constr r gitdb.utils.encodingr r gitdb_speedups._perfr r ImportError__all__r r r r r r r r rt r/ r- r s
. - - - - - - - , , , , , , AAAAAALL D %Q Q Q Q QY Q Q Qh` ` ` ` `y ` ` `T " " " " " " " "D # # # # #: # # #F# # # # # Z # # # P D
s A
AA