3
'ju @ s d dl Z d dlZd dlZd dlZd dlZd dlmZ d dlmZ d dl m
Z
d dl m Z d dl
mZ d dlmZ d dlmZ d d lmZ d d
lmZmZmZmZ d d lmZ eeeee d Z G d
d dZ dS ) N)quote)load)
InitSystem)JujuSSH)SSHControlPersist)LocalTransport) OCTransport)SaltStackMaster)CommandTimeoutExceptionConnectionExceptionUnsupportedHostExceptionInvalidTransportException)sos_parse_version)localZcontrol_persistZocZ saltstackZjujuc @ s e Zd ZdjddZedd Zdd Zd d
Zd d Zed
d Z edd Z
dd Z dd Z dd Z
dd ZdkddZed d Zd d Zd d! Zd"d# Zd$d% Zd&d' Zd(d) Zd*d+ Zd,d- Zd.d/ Zdld0d1Zd2d3 Zd4d5 Z d6d7 Z d8d9 Z dmdd? Z d@dA Z!dBdC Z"dDdE Z#dFdG Z$dHdI Z%dJdK Z&dLdM Z'dNdO Z(dPdQ Z)dRdS Z*dTdU Z+dVdW Z,dXdY Z-dZd[ Z.d\d] Z/d^d_ Z0d`da Z1dbdc Z2ddde Z3dfdg Z4dhdi Z5dS )nSosNodeNTc C s |j | _|| _|d | _| j |d | _|d | _|d | _|d | _| jj | _ | jj
| _
d| _ d | _ d | _
d | _i | _i | _|p| jj| _| jj r| jj r| j| j_|r|| j_d | _d| _d| _g | _d g g g g |d d| _d | _tj d
| _ tj d | _ | j || _ | j j!r,| j j!d k| _y| j j"| j W n6 t#k
rt } z| j$d
| W Y d d }~X nX | j% | _ d | _&| j j| _| j r| jj'rd}| j(r|r| j s| j j) d S | j r| j* rd| j _+| j j+r| j, | j- d S )NZ cmdlineoptstmpdirhostlen need_sudo sos_optionsFsos_cmd)versionenableddisabledoptionspresetsr sosreportZsosZsos_uirootz Unable to open remote session: ).stripaddresscommonsopts_assign_config_optsr r r r node_config_fileinherit_config_filer hostclusterhostname sos_env_vars _env_varspassword _passwordZ
nopasswd_sudoZsudo_pwsos_path retrievedZhash_retrieved file_listsos_infosos_binloggingZ getLoggersoslogui_log_load_remote_transport
_transportZ default_userZconnect Exception log_errordetermine_host_policy _sudo_binaryZno_local connected
disconnectcheck_in_container
containerizedcreate_sos_container_load_sos_info)selfr r r) Z
local_sudoZ
load_factserr rA /usr/lib/python3.6/sosnode.py__init__, st
zSosNode.__init__c C s | j r| j jS dS )NT)r4 r9 )r? rA rA rB r9 r s zSosNode.connectedc C s | j j dS )zDWrapper to close the remote session via our transport agent
N)r4 r: )r? rA rA rB r: y s zSosNode.disconnectc C st | j dkr d| _t| j |S | jjtkrSet the manifest section that this node will write to
r& policyZ sos_versionr final_sos_command rG N)manifest add_fieldrK r$ os_release_namer. r4 name)r? rb rA rA rB set_node_manifest s zSosNode.set_node_manifestc C s< t jjdr| jd dS t jjddk r8| jd dS dS )zV
Tries to identify if we are currently running in a container or not.
z/run/.containerenvz/Found /run/.containerenv. Running in container.TZ containerNz/Found env var 'container'. Running in containerF)rR pathexists log_debugrS get)r? rA rA rB r; s
zSosNode.check_in_containerc C s | j jr| j j| jj| j | jjd}| j|dd}|d dkr|d dkrd|d krf| jd t d
|d kr| jd t | j| j j
dd}|d dkr| j d | j j d
dS | jd|d t | jd|d t dS )zJIf the host is containerized, create the container we'll be using
)imageZauthZ
force_pullT) need_rootrN r } z unable to retrieve auth tokenrO zHCould not pull image. Provide either a username and password or authfilezunknown: Not foundz%Specified image not found on registryzTemporary container z createdz(Could not start container after create: z$Could not create container on host: F)r rm )
r$ r< r= r rk get_container_authZforce_pull_imagerU r6 r5 Zrestart_sos_containerlog_infoZsos_container_name)r? cmdresrW rA rA rB r= s2
z SosNode.create_sos_containerc C sB | j jr$| jjd j| j j| j jS | jjd j| j jp>| jjS )zrDetermine what the auth string should be to pull the image used to
deploy our temporary container
default) r Z
registry_userr$ ZruntimesZfmt_registry_credentialsZregistry_passwordZfmt_registry_authfileZregistry_authfileZcontainer_authfile)r? rA rA rB rn s
zSosNode.get_container_authFc C s: y | j d| |d}|d dkS tk
r4 dS X dS )z3Checks for the presence of fname on the remote nodezstat )rl rN r FN)rU r5 )r? Zfnamerl rq rA rA rB file_exists s
zSosNode.file_existsc C s | j rd| j kr| j S | jS )NrD )r& r )r? rA rA rB rK s zSosNode._hostnamec C s d}t j|d|S )zUAttempts to obfuscate sensitive information in log messages such as
passwordsz>(?P(pass|key|secret|PASS|KEY|SECRET).*?=)(?P.*?\s)z\g****** )resub)r? rL ZregrA rA rB _sanitize_log_msg
s zSosNode._sanitize_log_msgc C s | j j| j| dS )z:Format a ui message that includes host name and formattingN)r2 inforM )r? rL rA rA rB ui_msg s zSosNode.ui_msgc C s8 t j d d }d| j d| d| }| jj| dS )z#Used to print and log info messagesrJ [:z] N)inspectstackrK r1 rw )r? rL callerlmsgrA rA rB ro s zSosNode.log_infoc C s8 t j d d }d| j d| d| }| jj| dS )z$Used to print and log error messagesrJ ry rz r{ z] N)r| r} rK r1 error)r? rL r~ r rA rA rB r6 s zSosNode.log_errorc C sB | j |}tj d d }d| j d| d| }| jj| dS )z$Used to print and log debug messagesrJ ry rz r{ z] N)rv r| r} rK r1 debug)r? rL r~ rA rA rB ri s
zSosNode.log_debugc C sX | j jdtjjddk}| jr(|r(|S | jjr>dt| S | jrT| j d| S |S )zIf we need to provide a sudo or root password to a command, then
here we prefix the command with the correct bits
ZUSERr zsu -c )
r( rj rR rS r r become_rootr r r^ )r? rp Zis_rootrA rA rB _format_cmd' s
zSosNode._format_cmdc C s d}d}| j jdkrN| j jj| j j}|dk rdj|d }|d r|d }n8| j| j jddd}|d dkr|d j jd
\}}}|rt |jdd kr|d 7 }y |d
|jdd 7 }W n2 t
k
r } z| j d
| W Y dd}~X nX || j d< | j d r"| j
d| j d n$| j| jjks<| jd d| _dS | jdrXd| _| j d}| j|ddd}|d dkr| j|d | jdr| j dS )zLQueries the node for information about the installed version of sos
N.r releaseT)
use_containerrl rN r rO - z.0z#Unable to fully parse sos release: zsos version is z!sos is not installed on this nodeFz4.0z
sos reportz -lz3.6)r$ Zcontainer_version_commandpackage_managerZ pkg_versionZ sos_pkg_namejoinrU r rV lenr5 ri r. ro r r Zprimaryr6 r9 check_sos_versionr/ _load_sos_plugins_load_sos_presets) r? verZrelpkgZpkgs_r@ rp sosinforA rA rB r> 4 sF
"
zSosNode._load_sos_infoc C sr | j d}| j|ddd}|d dkrnxD|d j D ]4}|j jdr6|jdd j }| jd j| q6W d S )
Nz --list-presetsT)r rl rN r rO zname:rJ r )r/ rU
splitlinesr
startswithrV r. append)r? rp rq lineZpnamerA rA rB r h s zSosNode._load_sos_presetsc C s d}d}d}d}d}|d | }|d | }|d | } |d d }
| j ||| jd< | j ||| jd < | j | || jd
< | j |
|d | jd < d S )
Nz,The following plugins are currently enabled:z-The following plugins are currently disabled:z4The following options are available for ALL plugins:z+The following plugin options are available:z Profiles:z(.*?)z
r r r TZprofiles)_regex_sos_helpr. ) r? r ZENABLEDZDISABLEDZ ALL_OPTIONSZOPTIONSZPROFILESZ enableregZdisregZoptregZproregrA rA rB r q s zSosNode._load_sos_pluginsc
C s g }xt j||t jD ]}x|j D ]z}|sy |j }|rH|j|d W q tk
r| } z| jd| W Y d d }~X qX q$|jd} |jdd | D q$W qW |S )Nr zError parsing sos help: ,c s s | ]}|j r|j V qd S )N)r ).0prA rA rB s z*SosNode._regex_sos_help..) rt findallSr rV r r5 ri extend)
r? Zregexr Zis_listrq resultr Zlsr@ rrA rA rB r s $
zSosNode._regex_sos_helpc C sX y | j d| | jj|S tk
rR } z| jd| d| dS d}~X nX dS )z1Reads the specified file and returns the contentsz
Reading file zException while reading z: ra N)ro r4 read_filer5 r6 )r? Zto_readr@ rA rA rB r s zSosNode.read_filec C sz | j r(| jd| jd j | jd S ti | jjt d| jj | j
dd}|rh| jd|j d |S | j d t d S )
z[Attempts to identify the host installation against supported
distributions
zusing local policy r_ Tz/etc/os-release)cachesysrootZinitZ
probe_runtimeZ remote_execZ remote_checkzloaded policy z for hostz4Unable to determine host installation. Ignoring nodeN)
r ro r rd r r r r r4 rU r r6 r )r? r$ rA rA rB r7 s
z SosNode.determine_host_policyc C sP y| j d }t|t|kS tk
rJ } z| jd| dS d}~X nX dS )a Checks to see if the sos installation on the node is AT LEAST the
given ver. This means that if the installed version is greater than
ver, this will still return True
:param ver: Version number we are trying to verify is installed
:type ver: ``str``
:returns: True if installed version is at least ``ver``, else False
:rtype: ``bool``
r z Error checking sos version: FN)r. r r5 r6 )r? r Z _node_verr@ rA rA rB r s
zSosNode.check_sos_versionc C s | j s
dS | j jj|dk S )z2Checks if a given package is installed on the nodeFN)r$ r Z pkg_by_name)r? r rA rA rB is_installed s zSosNode.is_installed rF c
C s | j rt| j rt| jd y$| jj| j}|s<| jd tW n4 tk
rr } z| jd| W Y dd}~X nX |r| jj r| jj
|}|r| j |}|r| j } | j
|}| jj|||||S )a Runs a given cmd, either via the SSH session or locally
:param cmd: The full command to be run
:type cmd: ``str``
:param timeout: Time in seconds to wait for `cmd` to complete
:type timeout: ``int``
:param use_shell: If a shell is needed to run `cmd`, set to True
:type use_shell: ``bool`` or ``auto`` for transport-determined
:param use_container: Run this command in a container *IF* the host
is a containerized host
:type use_container: ``bool``
:param env: Pass environment variables to set for this `cmd`
:type env: ``dict``
z-Node is disconnected, attempting to reconnectzFailed to reconnect to nodez!Error while trying to reconnect: N)r9 r ri r4 Z reconnectr* r r5 r$ r< Zformat_container_commandr rZ updaterU )
r? rp timeout use_shellrl r envZ reconnectedr@ Z_cmd_envrA rA rB rU s&
zSosNode.run_commandc C sv y6| j }|r| j| n
| jd | jr4| j | _W n2 tk
rh } z| jd| W Y dd}~X nX | j dS )z.Run an sos report on the node, then collect itz'Unable to determine path of sos archivez Error during sos execution: N)execute_sos_commandfinalize_sos_pathr6 r+ retrieve_sosreportr, r5 cleanup)r? rg r@ rA rA rB r s
"zSosNode.sosreportc C s || j d kS )z/Verifies if the given preset exists on the noder )r. )r? presetrA rA rB _preset_exists s zSosNode._preset_existsc s& t fdd| jd | jd gD S )z/Verifies if the given plugin exists on the nodec 3 s | ]} |kV qd S )NrA )r s)pluginrA rB r s z)SosNode._plugin_exists..r r )anyr. )r? r rA )r rB _plugin_exists s zSosNode._plugin_existsc C s || j d kS )z6Checks to see if the plugin is default enabled on noder )r. )r? r rA rA rB _check_enabled s zSosNode._check_enabledc C s || j d kS )z7Checks to see if the plugin is default disabled on noder )r. )r? r rA rA rB _check_disabled s zSosNode._check_disabledc C sR |j dd }| j|s dS | j|r6|| jjkr6dS | j|rN|| jd kS dS )zAttempts to verify that the given option is available on the node.
Note that we only get available options for enabled plugins, so if a
plugin has been force-enabled we cannot validate if the plugin option
is correct or notr r Fr T)rV r r r enable_pluginsr r. )r? optplugrA rA rB _plugin_option_exists s
z SosNode._plugin_option_existsc s dj fdd|D S )z^Returns a comma delimited list for sos plugins that are confirmed
to exist on the noder c 3 s | ]} j |r|V qd S )N)r )r o)r? rA rB r s z,SosNode._fmt_sos_opt_list..)r )r? r rA )r? rB _fmt_sos_opt_list s zSosNode._fmt_sos_opt_listc C s
|| _ dS )zVExpose the node to the cluster profile determined for the
environment
N)r% )r? r% rA rA rB set_cluster" s zSosNode.set_clusterc s> | j jr0| js| j j| _n| jd| j j d | j jr`x&| j jD ]}|| jkrB| jj| qBW | j jrx,| j jD ] | jkrr| j j | j < qrW | j jrxJ| j jD ]> t fdd| j
D s d| j j }| j
j| qW | j j | r| j j | j j
| W dQ R X n | j j | j j| W dQ R X dS )a+ This is used to modify the sos report command run on the nodes.
By default, sos report is run without any options, using this will
allow the profile to specify what plugins to run or not and what
options to use.
This will NOT override user supplied options.
zCluster specified preset z> but user has also defined a preset. Using user specification.c 3 s | ]} |kV qd S )NrA )r r )r rA rB r D s z2SosNode.update_cmd_from_cluster..rQ N)r% Z
sos_presetr ro Z sos_pluginsr r r Zsos_plugin_optionsr plugoptsZcheck_node_is_primarylockZset_primary_optionsZset_node_options)r? r ZoptionrA )r rB update_cmd_from_cluster( s,
z SosNode.update_cmd_from_clusterc C sJ t | jj| _t | jj| _t | jj| _t | jj| _t | jj| _dS )zFrom the global opts configuration, assign those values locally
to this node so that they may be acted on individually.
N)listr only_plugins skip_pluginsr r r )r? rA rA rB r! P s
zSosNode._assign_config_optsc s j d } j }|r*| dt| d}g } jdr` jjdkr`|jdtt jj jdr jjr|jdtt jj jdr܈ jj r|jd jj
r|jd
jj r|jd t jj jd r8 jj r|jd
tdj
jj jjr8|jdtdj
jj jdrj jjrj|jdtt jj jds jdr jjdkr|jd jj jjr|jd jj jdr jjr|jd jj jjr |jd j |jdtjj
jj j}x: jD ]0} j| }|jd| d|rP|nd q,W j r fd d j D }|r|jd tdj
d d! |D j r j j r|jd"t j n j d# j d$ j r fd%d j D }t!|t! j kr@t"t# j t#| } j d&| d' j$ j } | rf|jd(t| | ddj
| _% j&d) j% j'j(d* j% d+S j)r fd,d j)D }
t!|
t! j)krt"t# j)t#|
} j d-| d. j$|
} | r |jd/t| j*r fd0d j*D }t!|t! j*krvt"t# j*t#| }
j d1|
d2 j$|}|r|jd3t| | ddj
| _% j&d) j% j'j(d* j% d+S )4zZUse host facts and compare to the cluster type to modify the sos
command if neededr r z3.6 z
--threads=z3.7z--plugin-timeout=z3.8z--allow-system-changesz
--no-env-varsz--since=z4.1z--skip-commands=r z
--skip-files=z4.2z--cmd-timeout=z4.3z4.2-13rF z--container-runtime=z
--namespaces=z4.5.2z--journal-size=z--low-priorityr z--ra c s8 g | ]0} j |jd d r j|jdd r|qS )r r rQ )r rV r )r r )r? rA rB
s z,SosNode.finalize_sos_cmd..z-k c s s | ]
}|V qd S )NrA )r r rA rA rB r s z+SosNode.finalize_sos_cmd..z --preset=zRequested to enable preset z" but preset does not exist on nodec s g | ]} j |r|qS rA )r )r r )r? rA rB r s zRequested plugins z. were requested to be enabled but do not existz--only-plugins=zFinal sos command set to r` Nc s g | ]} j |r|qS rA )r )r r )r? rA rB r s zRequested to skip plugins z%, but plugins are already not enabledz--skip-plugins=c s. g | ]&}| j kr j|r j|r|qS rA )r r r )r r )r? rA rB r s
z Requested to enable plugins z1, but plugins are already enabled or do not existz--enable-plugins=)+r. determine_sos_labelr r r Zthreadsr strZplugin_timeoutZallow_system_changesZ no_env_varsZsinceZ
skip_commandsr Z
skip_filesZ cmd_timeoutZcontainer_runtimeZ
namespacesZ journal_sizeZ low_priorityr replacerR rg r$ Z sos_bin_pathr/ r r r r ri r r r setr r ro rb rc r r )r? r labelZsos_optsr rY r ZplugsZnot_onlyZonlyskipZnot_skipZskiplnZnot_onenablerA )r? rB finalize_sos_cmd[ s
&$
zSosNode.finalize_sos_cmdc C s d}|| j j| 7 }| jjr<||s,| jj n d| jj 7 }|sDdS | jd| | jdrdd}n d}| jjdd d| }| d
| S ) z?Determine what, if any, label should be added to the sos reportra r Nz Label for sos report set to z3.6z--labelz--namer r rQ )r% Zget_node_labelr r ri r r rV )r? r ZlcmdrA rA rB r s
zSosNode.determine_sos_labelc C s^ | j j}|r|j|d}|j d }| jd| || _|jdd | _| jjd| j dS ) z\Use host facts to determine if we need to change the sos path
we are retrieving fromra r zFinal sos path: r[ rJ Zcollected_archiveNr] ) r$ Zsos_path_stripr rV ro r+ archiverb rc )r? rg ZpstriprA rA rB r s zSosNode.finalize_sos_pathc C sT |d
kr dS |dkr d|kr dS |dkr,dS t |dkrJ|jddd S d | S ) NrJ z+sos report process received SIGKILL on noder\ zsudo attempt failed z4sos report terminated unexpectedly. Check disk spacer
zsos exited with code r] )r rV )r? ZrcstdoutrA rA rB determine_sos_error s zSosNode.determine_sos_errorc C s: | j d yd}d}d}| jjr2d| jj }nR| jjr| jsvd| jjdd" d}| jj| jj | d| }nd| jj }|r| j
d | | _
| j | j
| jj d
d
d
| j
d }|d d
krxZ|d j D ]J}tj|dr|j }|jd#r|jdd }q|jdr|j d }qW |r| jjd| t|dkrT| jjdd n,t|dkrr| jjdd n| jjdd n| jjdd n:| j|d |d }| jd |d d |d t||S tk
r | jd Y nF tk
r4 } z(| jd | | j d | W Y d!d!}~X nX d!S )$z2Run sos report and capture the resulting file pathzGenerating sos report...Fra z--config-file=z/tmp/r[ rJ z.confr T)r r rl r r rN r rO z*sosreport-*tar* sha256 md5 zThe checksum is: ry checksum Z
checksum_typeZmd5@ Zsha256unknownz Error running sos report. rc = z msg = zTimeout exceededz'Exception during sos report execution: zError running sos report: Nr] )r r )rx r r" r# r r rV r4 Zcopy_file_to_remoteZ config_filer rU r r' r fnmatchr r rb rc r r ri r5 r
r6 ro )r? rg r Zconfig_file_argZ
remote_configrq r r@ rA rA rB r sd
zSosNode.execute_sos_commandc C s | j d }tjj||jdd }yD| j|rR| jd| d| | jj||S | j d| d dS t
k
r } z| j d| d | dS d
}~X nX d
S ) z7Copies the specified file from the host to our temp dirr[ rJ zCopying remote z
to local z Attempting to copy remote file z%, but it does not exist on filesystemFzFailed to retrieve z: Nr] ) r rR rg r rV rs ro r4
retrieve_fileri r5 )r? rg Zdestdirdestr@ rA rA rB r G s
zSosNode.retrieve_filec C s dj |j }y|t|jddkr8| jd| d dS | j|rv| jd| d| }| j|d d
}|d d kS | jd
| d dS tk
r } z| jd| d| dS d}~X nX dS )z|Removes the spciefied file from the host. This should only be used
after we have retrieved the file already
ra r[ r zRefusing to remove path z0: appears to be incorrect and possibly dangerousFzRemoving file zrm -f T)rl rN r z!Attempting to remove remote file z%, but it does not exist on filesystemzFailed to remove z: N)r rV r ri rs ro rU r5 )r? rg rp rq erA rA rB remove_fileV s
zSosNode.remove_filec C s | j s| jjr>y| j| j W n tk
r< | jd dS X | jd| j | j d y| j
| j}W n* tk
r } z| j| dS d}~X nX |r| j d | j j | jj
dd d S | j d
dS ) z,Collect the sos report archive from the nodez Failed to make archive readableFzRetrieving sos report from zRetrieving sos report...Nz!Successfully collected sos reportr[ rJ Tz Failed to retrieve sos reportr] )r r r make_archive_readabler+ r5 r6 ro r rx r r- r rV )r? rW r@ rA rA rB r l s&
zSosNode.retrieve_sosreportc C sT | j dks| jrdS d| j kr6| jd| j d dS | j| j }|sP| jd dS )zvRemove the sos report archive from the node, since we have
collected it and it would be wasted space otherwiseNr zNode sos report path z/ looks incorrect. Not attempting to remove pathzFailed to remove sos report)r+ r ri r r6 )r? ZremovedrA rA rB remove_sos_archive s
zSosNode.remove_sos_archivec C sN | j | jr.x dD ]}| j| j| rP qW | jj }|rJ| j|dd dS )z r r r r r7 r r rU r r r r r r r r r r! r r r r r r r r r r r r rA rA rA rB r * sj
E
(
4
'( < r ) r r| r0 rR rt Zshlexr Z sos.policiesr Zsos.policies.init_systemsr Z sos.collector.transports.jujur Z(sos.collector.transports.control_persistr Z sos.collector.transports.localr Zsos.collector.transports.ocr Z"sos.collector.transports.saltstackr Zsos.collector.exceptionsr
r r r
Z
sos.utilitiesr rH r rA rA rA rB
s(