3
'jp @ s d dl Z d dlZd dlZd dlmZ d dlmZ d dlmZm Z yd dl
Z
dZ W n e k
rh dZ Y nX yd dl
Z
dZW n e k
r dZY nX G dd dZdS ) N)getpass)_sos)
is_executableTIMEOUT_DEFAULTTFc @ s e Zd ZdZdZdZdZdZdZdZ dZ
dZ dZ dZ
dZdZdZdZdZdZdZdZdZdZdZdZdZdZdZ ddd g ddddd
d d
dddddddd Z dSd d
Z dd Z dd Z e!dd Z"dd Z#dd Z$dd Z%dd Z&d d Z'd d Z(d d! Z)d"d# Z*d$d% Z+d&d' Z,d(d) Z-dTd*d+Z.d,d- Z/d.d/ Z0d0d1 Z1d2d3 Z2d4d5 Z3d6d7 Z4d8d9 Z5d:d; Z6dd? Z8d@dA Z9dUdBdCZ:dDdE Z;dVdGdHZdMdN Z?dXdOdPZ@dYdQdRZAdS )Z UploadTargetz
This class is designed to upload files to a distribution
defined location. These files can be either sos reports,
sos collections, or other kind of files like: vmcores,
application cores, logs, etc.
z
Upload a file (can be an sos report, a must-gather, or others) to
a distribution defined remote location
zGeneric UploadZgenericN/zhttps://s3.amazonaws.com Fauto) upload_filecase_id low_priorityZprofiles
upload_urlupload_directory upload_user upload_pass
upload_methodupload_no_ssl_verifyupload_protocolupload_s3_endpointupload_s3_regionupload_s3_bucketupload_s3_access_keyupload_s3_secret_keyupload_s3_object_prefix
upload_targetc C s" t jd| _|| _|| _|| _d S )NZsos_ui)loggingZ getLoggerui_logparsercmdlineargs)selfr r r r! /usr/lib/python3.6/__init__.py__init__a s zUploadTarget.__init__c C s dS )aS This should be overridden by upload targets
This is called by sos upload on each target type that exists, and
is meant to return True when the upload target matches a criteria
that indicates that is the local upload target that should be used.
Only the first upload target to determine a match is selectedFr! )r r! r! r" check_distributionh s z UploadTarget.check_distributionc C s | j S )N)upload_target_id)r r! r! r"
get_target_idr s zUploadTarget.get_target_idc C s | j r | j S | jj S )z-Returns the upload target's name as a string.)upload_target_name__name__lower)clsr! r! r" nameu s zUploadTarget.namec C s* | j d | j d | j d j| j d jdS )N cmdlineoptspolicy)r, r- r r ) hook_commonsr r )r r! r! r" get_commons| s
zUploadTarget.get_commonsc C s
|| _ dS )zMSet common host data for the Upload targets
to reference
N)commons)r r0 r! r! r" set_commons s zUploadTarget.set_commonsc C s || _ | j | _| jd }| jd }|jr2|j |j| _|j| _|j| _|j| _ d| _
|j | _ |j | _ |j
| _
|j| _|j| _|j| _|j r|j r| j r|jdk r| j | j n*|jdkr| j | j | j | j | jj d d S )Nr, r- r s3) r. r/ r0 r Z_configure_low_priorityr
r r r upload_passwordupload_archive_namer r r r r r Zbatchquietget_upload_urlr prompt_for_upload_userprompt_for_upload_passwordprompt_for_upload_s3_bucket prompt_for_upload_s3_endpoint prompt_for_upload_s3_access_key prompt_for_upload_s3_secret_keyr info)r r. Z cmdline_optsr- r! r! r" pre_work s:
zUploadTarget.pre_workc C s4 | j s0d| j d| j d}tt|| _dS )zuShould be overridden by targets to determine if an access key needs
to be provided for upload or not
z0Please provide the upload access key for bucket z via endpoint z: N)get_upload_s3_access_keyget_upload_s3_bucketget_upload_s3_endpointinput_r )r msgr! r! r" r; s z,UploadTarget.prompt_for_upload_s3_access_keyc C s0 | j s,d| j d| j d}t|| _dS )ztShould be overridden by targets to determine if a secret key needs
to be provided for upload or not
z0Please provide the upload secret key for bucket z via endpoint z: N)get_upload_s3_secret_keyr@ rA r r )r rD r! r! r" r< s z,UploadTarget.prompt_for_upload_s3_secret_keyc C sH | j sB| jr*| jjdr*| jdd | _ nttd}|jd| _ | j S )zpShould be overridden by targets to determine if a bucket needs to
be provided for upload or not
zs3:// Nz"Please provide the upload bucket: r )r r
startswithrB rC strip)r
user_inputr! r! r" r9 s z(UploadTarget.prompt_for_upload_s3_bucketc C s> | j }| js8d| j d| d}tt|}|p4|| _| jS )zsShould be overridden by targets to determine if an endpoint needs
to be provided for upload or not
z.Please provide the upload endpoint for bucket z (default: z): )_upload_s3_endpointr r@ rB rC )r Zdefault_endpointrD rI r! r! r" r: s
z*UploadTarget.prompt_for_upload_s3_endpointc C s* | j s&d| j d}tt|| _dS )zcShould be overridden by targets to determine if a user needs to
be provided or not
z Please provide upload user for z: N)get_upload_userr6 rB rC r )r rD r! r! r" r7 s z#UploadTarget.prompt_for_upload_userc C s6 | j r2| j | jkr2d| j d}t|| _dS )zrShould be overridden by targets to determine if a password needs to
be provided for upload or not
z'Please provide the upload password for z: N)get_upload_passwordrK _upload_userr r3 )r rD r! r! r" r8 s z'UploadTarget.prompt_for_upload_passwordc C sL || _ | js| j | _| js$td| j }| jjtd| j | S )a
Entry point for sos attempts to upload the generated archive to a
target or user specified location.
Currently there is support for HTTPS, SFTP, and FTP. HTTPS uploads are
preferred for target-defined defaults.
Targets that need to override uploading methods should override the
respective upload_https(), upload_sftp(), and/or upload_ftp() methods
and should NOT override this method.
:param archive: The archive filepath to use for upload
:type archive: ``str``
In order to enable this for a target, that target needs to implement
the following:
Required Class Attrs
:_upload_url: The default location to use. Note these MUST include
protocol header
:_upload_user: Default username, if any else None
:_upload_password: Default password, if any else None
The following Class Attrs may optionally be overidden by the Target
:_upload_directory: Default FTP server directory, if any
The following methods may be overridden by ``Target`` as needed
`prompt_for_upload_user()`
Determines if sos should prompt for a username or not.
`get_upload_user()`
Determines if the default or a different username should be used
`get_upload_https_auth()`
Format authentication data for HTTPS uploads
`get_upload_url_string()`
Print a more human-friendly string than vendor URLs
zBNo upload destination provided by upload target or by --upload-urlzAttempting upload to ) r4 r
r6 Exception_determine_upload_typer r= rC get_upload_url_string)r archiveZ upload_funcr! r! r" upload_archive s ,
zUploadTarget.upload_archivec C sv | j | j| j| jd}| jd j|kr6|| jd j S d| jkrHtd| jjd\}}||krntd| || S )a Based on the url provided, determine what type of upload to attempt.
Note that this requires users to provide a FQDN address, such as
https://myvendor.com/api or ftp://myvendor.com instead of
myvendor.com/api or myvendor.com
)ZftpsftpZhttpsr2 r, z://z#Must provide protocol in upload URLz&Unsupported or unrecognized protocol: )
upload_ftp upload_sftp upload_https upload_s3r0 r r
rN split)r ZprotsZprotrC r! r! r" rO 3 s
z#UploadTarget._determine_upload_typec C s& |s | j }|s| j }tjj||S )ah Formats the user/password credentials using basic auth
:param user: The username for upload
:type user: ``str``
:param password: Password for `user` to use for upload
:type password: ``str``
:returns: The user/password auth suitable for use in requests calls
:rtype: ``requests.auth.HTTPBasicAuth()``
)rK rL requestsauthZ
HTTPBasicAuth)r userpasswordr! r! r" get_upload_https_authI s
z"UploadTarget.get_upload_https_authc C s t jddp| jp| jS )zHelper function to determine if we should use the target default
upload access key or one provided by the user
:returns: The access_key to use for upload
:rtype: ``str``
ZSOSUPLOADS3ACCESSKEYN)osgetenvr _upload_s3_access_key)r r! r! r" r? \ s z%UploadTarget.get_upload_s3_access_keyc C s | j s| j | j S )zHelper function to determine if we should use the target default
upload endpoint or one provided by the user
:returns: The S3 Endpoint to use for upload
:rtype: ``str``
)r r: )r r! r! r" rA g s z#UploadTarget.get_upload_s3_endpointc C s | j p
| jS )zHelper function to determine if we should use the target default
upload region or one provided by the user
:returns: The S3 region to use for upload
:rtype: ``str``
)r _upload_s3_region)r r! r! r" get_upload_s3_regionr s z!UploadTarget.get_upload_s3_regionc C sb | j rH| j jdrH| j dd jdd}|d | _t|dkrH|d | _| jsV| j | jp`| jS )zHelper function to determine if we should use the target default
upload bucket or one provided by the user
:returns: The S3 bucket to use for upload
:rtype: ``str``
zs3://rF Nr r )r
rG rX r lenr r9 _upload_s3_bucket)r Zbucket_and_prefixr! r! r" r@ { s
z!UploadTarget.get_upload_s3_bucketc C s | j p
| jS )zHelper function to determine if we should use the target default
upload object prefix or one provided by the user
:returns: The S3 object prefix to use for upload
:rtype: ``str``
)r _upload_s3_object_prefix)r r! r! r" get_upload_s3_object_prefix s z(UploadTarget.get_upload_s3_object_prefixc C s t jddp| jp| jS )zHelper function to determine if we should use the target default
upload secret key or one provided by the user
:returns: The S3 secret key to use for upload
:rtype: ``str``
ZSOSUPLOADS3SECRETKEYN)r^ r_ r _upload_s3_secret_key)r r! r! r" rE s z%UploadTarget.get_upload_s3_secret_keyc C sH | j r<| jr<| jr<| jr<| j }| j }d| d| | _| j pF| jS )zHelper function to determine if we should use the target default
upload url or one provided by the user
:returns: The URL to use for upload
:rtype: ``str``
zs3://r )r
r r r r@ rg _upload_url)r bucketprefixr! r! r" r6 s zUploadTarget.get_upload_urlc C s d}t j|d|}|S )Nz ([^:]+://[^:]+:)([^@]+)(@.+)z \1********\3)resub)r urlpatternZobfuscated_urlr! r! r" _get_obfuscated_upload_url s z'UploadTarget._get_obfuscated_upload_urlc C s | j | j S )zUsed by upload targets to potentially change the string used to
report upload location from the URL to a more human-friendly string
)rp r6 )r r! r! r" rP s z"UploadTarget.get_upload_url_stringc C s t jddp| jp| jS )zHelper function to determine if we should use the target default
upload user or one provided by the user
:returns: The username to use for upload
:rtype: ``str``
Z
SOSUPLOADUSERN)r^ r_ r rM )r r! r! r" rK s z UploadTarget.get_upload_userc C s t jddp| jp| jS )a_ Helper function to determine if we should use the target default
upload password or one provided by the user
A user provided password, either via option or the 'SOSUPLOADPASSWORD'
environment variable will have precendent over any target value
:returns: The password to use for upload
:rtype: ``str``
ZSOSUPLOADPASSWORDN)r^ r_ r3 _upload_password)r r! r! r" rL s
z UploadTarget.get_upload_passwordc C s: t dstdy ddl}W n, tk
rH } ztd|W Y dd}~X nX d}|sZ| j }|sf| j }| j jdd}d | d
| }|j|d d }d
dd|j |j
g} |j | dd}
|
dkrd}n|
dkr|j | d
d|j |j
g} |j | dddk}|sr|j
td| j nX|
dkr:td| j dn8|
dkrXtd| j n|
d krrtd |j |s|j
td | j d | j d | j } |j | d!|j |j
d"g}
|j |
d#d}|dkr|j d$ dS |dkrtd%|dkrtd&|j |dkr&td'td(|j dS ))a Attempts to upload the archive to an SFTP location.
Due to the lack of well maintained, secure, and generally widespread
python libraries for SFTP, sos will shell-out to the system's local ssh
installation in order to handle these uploads.
Do not override this method with one that uses python-paramiko, as the
upstream sos team will reject any PR that includes that dependency.
rS z SFTP is not locally supportedr NzFSFTP upload requires python3-pexpect, which is not currently installedFzsftp://r z sftp -oStrictHostKeyChecking=no @zutf-8)encodingzsftp>z password:zConnection refused )timeoutTrc zPermission denied
z#Incorrect username or password for zConnection refused by z. Incorrect port? z!Timeout hit trying to connect to z,Unexpected error trying to connect to sftp: z Unable to connect via SFTP to zput z100%zNo such file or directory Zbyez Timeout expired while uploadingz Unknown error during upload: z&Unable to write archive to destinationz!Unexpected response from server: )r rN pexpect ImportErrorrK rL r6 replaceZspawnZTIMEOUTZEOFexpectZsendlinecloserP Zbeforer4 _get_sftp_upload_name)r r[ r\ r| errZsftp_connectedZsftp_urlZsftp_cmdretZ sftp_expectsidxZ pass_expectsZput_cmdZ put_expectsZ put_successr! r! r" rU sv
zUploadTarget.upload_sftpc C s* | j jdd }| jr&tjj| j|}|S )a$ If a specific file name pattern is required by the SFTP server,
override this method in the relevant Upload Target. Otherwise the
archive's name on disk will be used
:returns: Filename as it will exist on the SFTP server
:rtype: ``str``
r rc )r4 rX r r^ pathjoin)r Zfnamer! r! r" r 7 s z"UploadTarget._get_sftp_upload_nameTc C s t j| j || j |tdS )zIf upload_https() needs to use requests.put(), use this method.
Targets should override this method instead of the base upload_https()
:param archive: The open archive file object
)datarZ verifyru )rY putr6 r] r )r rQ r r! r! r" _upload_https_putD s z UploadTarget._upload_https_putc C s i S )zJDefine any needed headers to be passed with the POST request here
r! )r r! r! r" _get_upload_headersO s z UploadTarget._get_upload_headersc C s: d|j jdd || j fi}tj| j || j |tdS )zIf upload_https() needs to use requests.post(), use this method.
Targets should override this method instead of the base upload_https()
:param archive: The open archive file object
filer rc )filesrZ r ru r )r+ rX r rY Zpostr6 r] r )r rQ r r r! r! r" _upload_https_postT s
z UploadTarget._upload_https_postc C s t s tdt| jd}| jd jdkr2| j}n | jd j}| jd jdk}|dkrd| j||}n | j ||}|j
dkr|j
d krtd
td |j
d |j d
S Q R X dS )zAttempts to upload the archive to an HTTPS location.
:returns: ``True`` if upload is successful
:rtype: ``bool``
:raises: ``Exception`` if upload was unsuccessful
z7Unable to upload due to missing python requests libraryrbr, r Fr i z/Authentication failed: invalid user credentialszPOST request returned z: TN)r r ) REQUESTS_LOADEDrN openr4 r0 r _upload_methodr r r Z status_codereason)r Zarcmethodr rr! r! r" rV c s
zUploadTarget.upload_httpsc C s ddl }ddl}|s | j }|dkr,td|jdd}|sD| j }|sP| j }|s`| jp^| j}y.|j |||dd}|std|j
| W n |j k
r } ztd | |W Y dd}~X n |j k
r } ztd
| |W Y dd}~X n |j
k
r } z|t|j d } | d kr4td | d
|| dkrPtd| d
|| dkrjtd| |tdt| |W Y dd}~X nX t| jd$}
|jd| jjdd |
W dQ R X |j dS )aT Attempts to upload the archive to either the target defined or user
provided FTP location.
:param url: The URL to upload to
:type url: ``str``
:param directory: The directory on the FTP server to write to
:type directory: ``str`` or ``None``
:param user: The user to authenticate with
:type user: ``str``
:param password: The password to use for `user`
:type password: ``str``
:returns: ``True`` if upload is successful
:rtype: ``bool``
:raises: ``Exception`` if upload in unsuccessful
r NzPno FTP server specified by upload target, use --upload-url to specify a locationzftp://r rt )ru z3connection failed, did you set a user and password?z timeout hit while connecting to zunable to connect to Z503zcould not login as ''Z530zinvalid password for user 'Z550z"could not set upload directory to z#error trying to establish session: r zSTOR r rc Tr )ftplibsocketr6 rN r~ rK rL r _upload_directoryZFTPcwdru ZgaierrorZ
error_permstrrX r r4 Z
storbinaryquit) r rn Z directoryr[ r\ r r Zsessionr errnoZ_arcfiler! r! r" rT sL
$zUploadTarget.upload_ftpc
C s t s td|s| j }|s$| j }|s6| j jd}|s| j }|dkr`|jdr`|dd }|dkr|jd r|r| dnd}|s| j }|s| j
}t j d||||d}y(|| j
jdd
}|j| j
|| dS tk
r } ztd t| | W Y dd} ~ X nX dS ) a Attempts to upload the archive to an S3 bucket.
:param endpoint: The S3 endpoint to upload to
:type endpoint: str
:param region: The S3 region to upload to
:type region: str
:param bucket: The name of the S3 bucket to upload to
:type bucket: str
:param prefix: The prefix for the S3 object/key
:type prefix: str
:param access_key: The access key for the S3 bucket
:type access_key: str
:param secret_key: The secret key for the S3 bucket
:type secret_key: str
:returns: True if upload is successful
:rtype: bool
:raises: Exception if upload is unsuccessful
z4Unable to upload due to missing python boto3 libraryr r rc Nr2 )Z endpoint_urlZ region_nameZaws_access_key_idZaws_secret_access_keyTzFailed to upload to S3: r ) BOTO3_LOADEDrN rA rb r@ rH rg rG endswithr? rE boto3Zclientr4 rX r
r )
r ZendpointZregionrj rk Z
access_keyZ
secret_keyZ s3_clientkeyer! r! r" rW s: zUploadTarget.upload_s3)NNN)NN)NN)T)T)NNNN)NNNNNN)Br(
__module__ __qualname____doc__Zdescr' r% Z _upload_fileri r rM rq r rJ re r` rh ra rf r
r r3 r r r r r r r Z arg_defaultsr# r$ r& classmethodr+ r/ r1 r> r; r< r9 r: r7 r8 rR rO r] r? rA rb r@ rg rE r6 rp rP rK rL rU r r r r rV rT rW r! r! r! r" r s
*
8
c
F r )r^ rl r r Zsosr rC Z
sos.utilitiesr r rY r r} r r r r! r! r! r" s