3
'j__ @ sr d dl Z d dlZd dlZd dlZd dlZd dlZd dlZd dlmZ d dl m
Z
m Z m Z dd Z
G dd dZdS ) N)ProcessPoolExecutor)file_is_binarysos_get_command_outputfile_is_certificatec
C s t j| }tjj|d}tt ddd |_xp|j D ]d}tjj||j}tjj |}tjj |}tjj
||g}||krt d| d| |j || q6W tjj||jj
dd j
d d
S Q R X d S ) NZcleanerZfully_trusted_filterc S s | S )N )memberpathr r /usr/lib/python3.6/__init__.py# s z!extract_archive..z"Attempted path traversal in tarflez != / z.tarr )tarfileopenosr joingetattrZextraction_filterZ
getmembersnameabspath commonprefix Exceptionextractsplit) archive_pathtmpdirarchiver r Z member_pathZ
abs_directoryZ
abs_targetprefixr r r extract_archive s r c @ sx e Zd ZdZdZdZdZdZdZdZ i Z
dd Z dd Z d d
Z
d d Zd
d ZdRddZdd Zedd Zedd Zedd Zdd Zd d Zd d Zd d! ZdSd"d#ZdTd$d%Z dUd&d'Z dVd(d)Z d*d+ Z ed,d- Z d.d/ Z!d0d1 Z"d2d3 Z#d4d5 Z$dWd6d7Z%d8d9 Z&d:d; Z'dd? Z)d@dA Z*dBdC Z+dDdE Z,dFdG Z-dHdI Z.dJdK Z/dLdM Z0dNdO Z1dPdQ Z2dS )XSoSObfuscationArchivea A representation of an extracted archive or an sos archive build
directory which is used by SoSCleaner.
Each archive that needs to be obfuscated is loaded into an instance of this
class. All report-level operations should be contained within this class.
r Z undeterminedFc C s || _ | j | _|| _| j jdd jdd | _| j| _tjd| _tjd| _ | j
| _ d| _ | j
d| _|| _|| _f | _| jd | j d
| j d S ) Nr r z.tarr sossos_uiF zLoaded z as type r
)r final_archive_pathr r archive_nameui_namelogging getLoggersoslogui_log_load_skip_list skip_list is_extracted
_load_self archive_rootkeep_binary_filestreat_certificatesparserslog_info description)selfr r r. r/ r r r __init__C s
z SoSObfuscationArchive.__init__c C sV xP| j D ]F}y|j|}W q tk
rL } z| jd| W Y d d }~X qX qW |S )Nz Error obfuscating string data: )r0 Zparse_string_for_keysr r1 )r3 Z string_dataparsererrr r r obfuscate_stringW s &z&SoSObfuscationArchive.obfuscate_stringc C s | j |jdd }|j|jdd |}||kr|j|d }tjj||}tjj|shtj|| n&| j tj|}tj | tj
|| d S )Nr r r r
r
) r7 r replacer r r islinkrenamereadlinkremovesymlink)r3 Z
short_namefilenameZ_ob_short_nameZ _ob_filenamearc_pathZ_ob_pathZ
_target_obr r r obfuscate_filenamea s
z(SoSObfuscationArchive.obfuscate_filenamec C s
|| _ d S )N)r0 )r3 r0 r r r set_parsersx s z!SoSObfuscationArchive.set_parsersc C s4 t jd| _t jd| _x| jD ] }|j q W d S )Nr r )r% r& r' r( r0 Zload_map_entries)r3 r5 r r r load_parser_entries{ s z)SoSObfuscationArchive.load_parser_entriesNc C s d}|j s||fS |dkr"| j}x^|D ]V}y|j|\}}||7 }W q( tk
r| } z| jd| |j W Y dd}~X q(X q(W ||fS )a Run a line through each of the obfuscation parsers, keeping a
cumulative total of substitutions done on that particular line.
Positional arguments:
:param line str: The raw line as read from the file being
processed
:param parsers: A list of parser objects to obfuscate
with. If None, use all.
Returns the fully obfuscated line and the number of substitutions made
r Nzfailed to parse line: )stripr0 Z
parse_liner log_debugr )r3 liner0 countr5 _countr6 r r r obfuscate_line s
*z$SoSObfuscationArchive.obfuscate_linec ' s | j xp|D ]f}| jdtj d| y tjj|| jd | j rRw| j rp| j rp| j
w| jrt |s| j rwtjj |rwt
|}|r|dkr| j
w| jdkrw| jdkr| j
w| jdkr| j|}tjj|| jd fdd | jD }|s8| jd
p,| d w| jd pF| d
}tjd| jd}t|ddddx}xp|D ]h}y&| j||\}} || 7 }|j| W n: tk
r }
z | jd d|
W Y d d }
~
X nX q|W W d Q R X |jd
|rtj|j| | j | W d Q R X | j | W q tk
rv }
z&| jdtj d| d|
W Y d d }
~
X qX qW | j | j | j fS )Nz pid=z: obfuscating )startZcertificatekeyZkeepr< Z obfuscatec s( g | ] }t fd d|jD s|qS )c 3 s | ]}|j V qd S )N)match).0_skip)rel_namer r s zGSoSObfuscationArchive.obfuscate_arc_files...)anyZ
skip_patterns)rK Z_p)rM r r
s z=SoSObfuscationArchive.obfuscate_arc_files..zSkipping obfuscation of z" due to matching file skip patternz Obfuscating r w)modedirrzutf-8r8 )encodingerrorszUnable to obfuscate z: z': caught exception on obfuscating file )!rB rD r getpidr relpathextracted_pathshould_skip_filer. should_remove_file remove_filer r9 r r/ certificate_to_textr0 tempfileZNamedTemporaryFiler r rH writer seekshutilZcopyfiler update_sub_countr@ files_obfuscated_counttotal_sub_countremoved_file_count) r3 flistr> Zis_certificateZ_parsersZsubsZtfilefnamerE Zcntr6 r )rM r obfuscate_arc_files st
8
6z)SoSObfuscationArchive.obfuscate_arc_filesc C s t dS )z=Check if the archive is a well-known type we directly supportN)NotImplementedError)clsr? r r r
check_is_type s z#SoSObfuscationArchive.check_is_typec C s d| j jj kS )Nr ) __class____name__lower)r3 r r r is_sos s z SoSObfuscationArchive.is_sosc C s
d| j kS )NZinsights) type_name)r3 r r r is_insights s z!SoSObfuscationArchive.is_insightsc C s | j rtj| j| _d S )N)
is_tarfiler r r tarobj)r3 r r r r, s z SoSObfuscationArchive._load_selfc C s g S )a/ Return a list of ObfuscationArchives that represent additional
archives found within the target archive. For example, an archive from
`sos collect` will return a list of ``SoSReportArchive`` objects.
This should be overridden by individual types of ObfuscationArchive's
r )r3 r r r get_nested_archives s z)SoSObfuscationArchive.get_nested_archivesc C s> | j r0| jj}|j r |jS tjj|jp.tjS tjj | j
S )z|Set the root path for the archive that should be prepended to any
filenames given to methods in this class.
) rr rs Z firstmemberisdirr r r dirnamesepr r )r3 Ztoplevelr r r get_archive_root
s z&SoSObfuscationArchive.get_archive_rootc C s" | j j| jd dd| dS )z9Helper to easily format ui messages on a per-report basisz :z<50 N)r( infor$ )r3 msgr r r
report_msg s z SoSObfuscationArchive.report_msgc C s& d|rd| nd d| j d| S )Nz[cleaner:r! [z]] )r# )r3 r{ callerr r r _fmt_log_msg s z"SoSObfuscationArchive._fmt_log_msgc C s | j j| j|| d S )N)r' debugr )r3 r{ r r r r rD s z SoSObfuscationArchive.log_debugc C s | j j| j|| d S )N)r' rz r )r3 r{ r r r r r1 s z SoSObfuscationArchive.log_infoc C s | j j| j|| d S )N)r' errorr )r3 r{ r r r r log_error# s z SoSObfuscationArchive.log_errorc C s ddddddgS )zyProvide a list of files and file regexes to skip obfuscation on
Returns: list of files and file regexes
z
proc/kallsymsz
sosreport-z sys/firmwarezsys/fszsys/kernel/debugz
sys/moduler )r3 r r r r) & s z%SoSObfuscationArchive._load_skip_listc C s& y t j| jS tk
r dS X d S )NF)r rr r r )r3 r r r rr 4 s z SoSObfuscationArchive.is_tarfilec C sB | d}| j d| d td| dt| tj| |S )zConvert a certificate to text. This is used when cleaner encounters
a certificate file and the option 'treat_certificates' is 'obfuscate'.
z.textz Converting certificate file 'z ' to textz openssl storeutl -r -noout -out z -text -certs )r1 r strr r< )r3 rg Zout_fnr r r r] ; s
z)SoSObfuscationArchive.certificate_to_textc C s< | j |}|r8| jd| d tj| | jd7 _dS )zRemove a file from the archive. This is used when cleaner encounters
a binary file, which we cannot reliably obfuscate.
zRemoving binary file 'z' from archiver N)
get_file_pathr1 r r< re )r3 rg Z
full_fnamer r r r\ H s
z!SoSObfuscationArchive.remove_filec C s6 | j s&| js| j | _tjj| j|S tjj| j|S )zBased on the type of archive we're dealing with, do whatever that
archive requires to a provided **relative** filepath to be able to
access it within the archive
)r+ r- rx r r r rY )r3 rg r r r format_file_nameS s
z&SoSObfuscationArchive.format_file_namec C s | j dkrZ| jrZ| j|}y| jj|j jdS tk
rV | jd| d dS X n`y(t | j|ddd
}|j S Q R X W n6 t
k
r } z| jd | d
| dS d}~X nX dS ) zReturn the content from the specified fname. Particularly useful for
tarball-type archives so we can retrieve prep file contents prior to
extracting the entire archive
Fzutf-8zUnable to retrieve z: no such file in archiver! rT )rU NzFailed to get contents of z: ) r+ rr r rs Z extractfilereaddecodeKeyErrorrD r r )r3 rg r> Zto_readr6 r r r get_file_content^ s
z&SoSObfuscationArchive.get_file_contentc C s~ | j r,|s| jd | j | _d| _d | _n| j| _tj dkrh| j d xtj
| jD ]\}}}yx8|D ]0}tj j ||}tj
|j}tj||t
jB qpW x|D ]~}tj j ||} tj j| stj j| rqtj| tj stj| tj r| j d| j| jd tj| t
jt
jB qW W q\ tk
rb }
z| j d|
W Y d d }
~
X q\X q\W | j d| j d S )
Nz
Extracting...Tr z)Verifying permissions of archive contentsz Adding owner rw permissions to r z!Error while trying to set perms: zExtracted path is r
)rr r| extract_selfrY r+ rs r r getuidrD walkr r statst_modechmodS_IRWXUexistsr9 accessR_OKW_OKr S_IRUSRS_IWUSRr ) r3 quietrv dirsfiles_dirZ_dirnameZ
_dir_permsr> rg r6 r r r r u s6
&z SoSObfuscationArchive.extractc C s. | j j| j|}|| _tj| j | || _ dS )zRename the top-level directory to new_name, which should be an
obfuscated string that scrubs the hostname from the top-level dir
which would be named after the unobfuscated sos report
N)rY r8 r# r r: )r3 new_name_pathr r r rename_top_dir s z$SoSObfuscationArchive.rename_top_dirc C s | j r| jjdrdS dS dS )zReturn the compression type used by the archive, if any. This is
then used by SoSCleaner to generate a policy-derived compression
command to repack the archive
xzZgzN)rr r endswith)r3 r r r get_compression s
z%SoSObfuscationArchive.get_compressionc C s d}| j d }i }|rL|d| 7 }|d| 7 }|dkrDddi}ndd i}| jd
| tj|fd |i|$}|j| j tjj| jd d
W dQ R X |S )zIPack the extracted archive as a tarfile to then be re-compressed
rQ z-obfuscated.tarr} .r Zpreset Z
compresslevel zBuilding tar file rR r )ZarcnameN) rY rD r r addr r r r# )r3 methodrR ZtarpathZ
compr_argsZtarr r r build_tar_file s
z$SoSObfuscationArchive.build_tar_filec C s y| j || _W n4 tk
rD } z| jd| W Y dd}~X nX | jd| j y | j W n< tk
r } z | jd| | jd W Y dd}~X nX dS )zExecute the compression command, and set the appropriate final
archive path for later reference by SoSCleaner on a per-archive basis
z(Exception while re-compressing archive: NzCompressed to z'Failed to remove extraction directory: z/Failed to remove temporary extraction directory)r r" r rD remove_extracted_pathr| )r3 r r6 r r r compress s z SoSObfuscationArchive.compressc C st y"| j d| j tj| j W nL tk
rn tj| jtj tj j
| jr^tj | j n tj| j Y nX dS )zAfter the tarball has been re-compressed, remove the extracted path
so that we don't take up that duplicate space any longer during
execution
z Removing N) rD rY ra ZrmtreeOSErrorr r r r r isfiler< )r3 r r r r s z+SoSObfuscationArchive.remove_extracted_pathc C s2 t d }|jt| j| j}|j }|S Q R X dS )z}Extract an archive into our tmpdir so that we may inspect it or
iterate through its contents for obfuscation
r N)r Zsubmitr r r result)r3 Z_poolZ _path_futurer r r r r s
z"SoSObfuscationArchive.extract_selfc c s| xvt j| jD ]f\}}}x,|D ]$}t jj||}t jj|r |V q W x,|D ]$}t jj||}t jj|rL|V qLW qW dS )z.Iterator for a list of symlinks in the archiveN)r r rY r r r9 )r3 rv r r r Z_dirpathr> _fnamer r r get_symlinks s
z"SoSObfuscationArchive.get_symlinksc c sT xNt j| jD ]>\}}}x2|D ]*}t jj||jd}t jj|s |V q W qW dS )zIterator for a list of files in the archive, to allow clean to
iterate over.
Will not include symlinks, as those are handled separately
r N)r r rY r r lstripr9 )r3 rv _r r> r r r r get_files s
z SoSObfuscationArchive.get_filesc C s. g }x$t j| jD ]\}}}|j| qW |S )z3Return a list of all directories within the archive)r r rY append)r3 Zdir_listrv r r r r get_directory_list s z(SoSObfuscationArchive.get_directory_listc C s | j d7 _ | j|7 _dS )zCalled when a file has finished being parsed and used to track
total substitutions made and number of files that had changes made
r N)rc rd )r3 rF r r r rb s z&SoSObfuscationArchive.update_sub_countc C s* t jj| j|jd}t jj|r&|S dS )zReturn the filepath of a specific file within the archive so that
it may be selectively inspected if it exists
r r! )r r r rY r r )r3 rg r r r r r s z#SoSObfuscationArchive.get_file_pathc C sZ t jj| j| r,t jj| j| r,dS x(| jD ] }|j|sNtj||r4dS q4W dS )a% Checks the provided filename against a list of filepaths to not
perform obfuscation on, as defined in self.skip_list
Positional arguments:
:param filename str: Filename relative to the extracted
archive root
TF) r r r r r9 r*
startswithrerJ )r3 r> rL r r r rZ s
z&SoSObfuscationArchive.should_skip_filec C sV ddddddddd g }x |D ]}t j||r d
S q W | j|}tjj|rRt|S d S ) aa Determine if the file should be removed or not, due to an inability
to reliably obfuscate that file based on the filename.
:param fname: Filename relative to the extracted archive root
:type fname: ``str``
:returns: ``True`` if the file cannot be reliably obfuscated
:rtype: ``bool``
z.*\.gz$z.*\.xz$z
.*\.bzip2$z .*\.tar\..*z.*\.txz$z.*\.tgz$z.*\.bin$z .*\.journal$z.*\~$TF)r rJ r r r r r )r3 rg Zobvious_removesZ_arc_regZ
_full_pathr r r r[ - s
z(SoSObfuscationArchive.should_remove_file)N)N)N)N)N)F)3rm
__module__ __qualname____doc__rc rd re rp r2 Z is_nestedZ
prep_filesr4 r7 r@ rA rB rH rh classmethodrk propertyro rq r, rt rx r| r rD r1 r r) rr r] r\ r r r r r r r r r r r r rb r rZ r[ r r r r r 3 s\
R
&
r )r% r ra r r r^ r concurrent.futuresr Z
sos.utilitiesr r r r r r r r r s