3
cj @ s d dl Z d dlZd dlZd dlZd dlZd dlZd dlZd dlmZ d dl m
Z
m Z d dl m
Z
mZmZmZmZ d dlmZmZmZmZmZ d dlmZ d dlmZ d dlm Z d d l m Z d d
l m Z m!Z! d d l"m#Z# d d l$m%Z% d d
l&m'Z' d dl(m)Z) dZ*dZ+dZ,dZ-e*e+e,gZ.dZ/dZ0dZ1dZ2dZ3dZ4dZ5dZ6ej7e8Z9dd d fd d d fd d!d fd"Z:e G d#d$ d$e
Z;G d%d& d&e<Z=G d'd( d(e<Z>G d)d* d*e<Z?d+f fd,d-Z@e3fd.d/ZAed0d1d2d3d4gZBed5d6d7gZCG d8d de'e jDd9ZEd:d; ZFeeEeGf d<d=d>ZHd?d@ ZIdVeGeJdBdCdDZKdEdF ZLdWdHdIZMG dJdK dKeNZOdLdM ZPeEeGeJdNdOdPZQeGeeE dQdRdSZReGd<dTdUZSdS )X N)
namedtuple)Enumunique)AnyDictListOptionalTuple)
atomic_helperdmiimporternet
type_utils) user_data)util)
write_json)Distro)
EventScope EventType) launch_index)Paths)CloudInitPickleMixin)eventsZdisabledlocalr
passZ
FILESYSTEMNETWORK
DataSourcez|EXPERIMENTAL: The structure and format of content scoped under the 'ds' key may change in subsequent releases of cloud-init.zredacted for non-root userz
cloud-nameZ_unsetunknownz aws-chinac C s | dkS )Naws )cr r /usr/lib/python3.6/__init__.py? s r" zaws-govc C s | dkS )Nr r )r r r r! r" @ s z azure-chinac C s | dkS )NZazurer )r r r r! r" A s )zcn-zus-gov-Zchinac @ s2 e Zd ZdZdZdZdZdZdZe ddd Z
d
S ) NetworkConfigSourcezb
Represents the canonical list of network config sources that cloud-init
knows about.
cmdlinedsZ
system_cfgZfallbackZ initramfs)returnc C s | j S )N)value)selfr r r! __str__R s zNetworkConfigSource.__str__N) __name__
__module__ __qualname____doc__CMD_LINEDS
SYSTEM_CFGZFALLBACK INITRAMFSstrr) r r r r! r# E s r# c @ s e Zd ZdZdS ) DatasourceUnpickleUserDataErrorzERaised when userdata is unable to be unpickled due to python upgradesN)r* r+ r, r- r r r r! r3 V s r3 c @ s e Zd ZdS )DataSourceNotFoundExceptionN)r* r+ r, r r r r! r4 Z s r4 c @ s e Zd ZdZdS )InvalidMetaDataExceptionz8Raised when metadata is broken, unavailable or disabled.N)r* r+ r, r- r r r r! r5 ^ s r5 c
C s t j| }g }g }x| j D ]\}}|r6|d | }n|}|j |ksR|j |kr\|j| t|tr|jdr|j| |jdd||< t|t r t
|||} |j | j d |j | j d | ||< q W t
||d< t
||d< |S )zProcess all instance metadata cleaning it up for persisting as json.
Strip ci-b64 prefix and catalog any 'base64_encoded_keys' as a list
@return Dict copy of processed metadata.
/zci-b64:r6 base64_encoded_keyssensitive_keys)copydeepcopyitemslowerappend
isinstancer2
startswithreplacedictprocess_instance_metadataextendpopsorted)
metadatakey_pathr9 md_copyr8 Z sens_keyskeyvalZ sub_key_pathZ
return_valr r r! rC b s,
rC c C s | j dg s| S tj| }xh| j dD ]Z}|jd}|}x6|D ].}||kr>t|| tr>||d kr>|| }q>W ||kr&|||< q&W |S )zRedact any sensitive keys from to provided metadata dictionary.
Replace any keys values listed in 'sensitive_keys' with redact_value.
r9 r7 )getr: r; splitr? rB )rG Z redact_valuerI rH
path_partsobjpathr r r! redact_sensitive_keys s
rS URLParamsZmax_wait_secondsZtimeout_secondsZ num_retriessec_between_retriesDataSourceHostnamehostname
is_defaultc
@ s e Zd ZU eZdZdZdZdZdZ dZ
e j e j
e je jfZee df d|ZdZdZdZejejejejej hiZ ejejhiZ de fd e fd
i fd}d~ddddf Z eee!e"f df dZ#dZ$ee!df $dZ%dZ&de'e(ddd Z)e*dd d d Z+d d! Z,e-d"d#d$Z.e-d"d%d&Z/d'd( Z0d)d* Z1f fd+d,Z2e-d"d-d.Z3dd0d1Z4e-d"d2d3Z5d4d5 Z6dd6d7Z7d8d9 Z8d:d; Z9e:dd? Ze:dDdE Z?dFdG Z@e:dHdI ZAdJdK ZBe:dLdM ZCdNdO ZDdPdQ ZEdRdS ZFdTdU ZGdVdW ZHdXdY ZIdZd[ ZJd\d] ZKd^d_ ZLe:d`da ZMe:dbdc ZNddde ZOddfdgZPdhdi ZQeRe djdkdlZSeRe e-dmdndoZTdpdq ZUdrds ZVeWddtduZXe:dvdw ZYdxdy ZZdzd{ Z[dS )r z en_US.UTF-8Z_undefN.rL
ec2_metadata network_jsonrG userdata userdata_raw
vendordatavendordata_raw vendordata2vendordata2_rawFcombined_cloud_config
merged_cfgmerged_system_cfgsecurity-credentials user-datar vendor-datads/vendor_data)distropathsc C s~ || _ || _|| _d | _i | _d | _d | _d | _d | _d | _ t
j | j d| j fi | _
| j
s`i | _
|sttj| j| _n|| _d S )NZ
datasource)sys_cfgrj rk r] rG r^ r_ ra r` rb r Zget_cfg_by_pathdsnameds_cfgudZUserDataProcessorud_proc)r( rl rj rk rp r r r! __init__ s" zDataSource.__init__)ci_pkl_versionr& c C s t | dsd| _t | ds d| _t | ds0d| _t | dsJt| ddd t | d r| jdk ryt| j W n6 tk
r } ztj d
| t
|W Y dd}~X nX dS ) z(Perform deserialization fixes for Paths.ra Nrb skip_hotplug_detectF check_if_fallback_is_allowedc S s dS )NFr r r r r! r" = s z&DataSource._unpickle..r] z:Unable to unpickle datasource: %s. Ignoring current cache.) hasattrra rb rs setattrr] r2 AttributeErrorLOGdebugr3 )r( rr er r r! _unpickle3 s
zDataSource._unpicklec C s
t j| S )N)r obj_name)r( r r r! r) O s zDataSource.__str__)r& c C s dS )z#Check if running on this datasourceTr )r( r r r! ds_detectR s zDataSource.ds_detectc C sN | j j t j kr$tjd| dS | jjdg | j gkrJtjd| dS dS )aI Override if either:
- only a single datasource defined (nothing to fall back to)
- commandline argument is used (ci.ds=OpenStack)
Note: get_cmdline() is required for the general case - when ds-identify
does not run, _something_ needs to detect the kernel command line
definition.
zOMachine is configured by the kernel commandline to run on single datasource %s.TZdatasource_listz5Machine is configured to run on single datasource %s.F)rm r=
parse_cmdlinerx ry rl rN )r( r r r! override_ds_detectV s z DataSource.override_ds_detectc C s@ | j r| j S | j r,tjd| | j S tjd| dS dS )z&Overrides runtime datasource detectionz8Detected platform: %s. Checking for active instance dataz#Datasource type %s is not detected.FN)r _get_datar} rx ry )r( r r r! _check_and_get_datam s z DataSource._check_and_get_datac C s | j j}| j }| j}|d }ddg||t| j| j| j| j| j|d d |d d |d d | j| j |d |||d d |||d d
| j| j |d |d d
iS )z2Return a dictionary of standardized metadata keys.sys_infov1 subplatformZdistr rL pythonuname platformvariant)Z
_beta_keyszavailability-zoneavailability_zonecloud_idz
cloud-name
cloud_namerj Zdistro_versionZdistro_releaser Zpublic_ssh_keysZpython_versionz instance-id instance_idZkernel_releasezlocal-hostnamelocal_hostnamemachineregionr Zsystem_platformr )
get_hostnamerW get_instance_idr canonical_cloud_idr r
platform_typeget_public_ssh_keysr )r(
instance_datar r r Zsysinfor r r! _get_standardized_metadata{ s8
z%DataSource._get_standardized_metadatac C sP | j s
dS |r|}n| j}x&|D ] \}}t| |r t| || q W |sLd| _ dS )zReset any cached metadata attributes to datasource defaults.
@param attr_defaults: Optional tuple of (attr, value) pairs to
set instead of cached_attr_defaults.
NF) _dirty_cachecached_attr_defaultsru rv )r( Z
attr_defaultsZ attr_valuesZ attributer' r r r! clear_cached_attrs s
z DataSource.clear_cached_attrsc C s" d| _ | j }|s|S | j |S )zDatasources implement _get_data to setup metadata and userdata_raw.
Minimally, the datasource should return a boolean True on success.
T)r r persist_instance_data)r( Z return_valuer r r! get_data s zDataSource.get_dataTc C st |r&t jj| jjr&t| | jjd t| drbtj t
| d}|j dd |j dd d|i}n^dd| j ii}t| drt
| d}|t
kr||d d< t| d rt
| d }|t
kr||d d < t|d d
< tj | j|d < d |d d
< tj |d |d
< d|d
d
< tj |d< |j| j| y"tj|}ttj|| jd}W nb tk
rv } ztj dt | dS d}~X n2 t k
r } ztj dt | dS d}~X nX | jj d} |d j dd}
t jj!| jj"d} tj#| d|
|
d d} | d|
}
t jj$| r t jj%| } tj&|
| dd | rH| |
krHtj'| t(| |d d | jj d }t(|t)| dS ) aP Process and write INSTANCE_JSON_FILE with all instance metadata.
Replace any hyphens with underscores in key names for use in template
processing.
:param write_cache: boolean set True to persist obj.pkl when
instance_link exists.
@return True on successful write, False otherwise.
Zobj_pklZ_crawled_metadataz user-dataNz vendor-datar% Z meta_datar\ r[ Z_docrd z, (str, bool).
is_default is a bool and
it's true only if hostname is localhost and was
returned by util.get_hostname() as a default.
This is used to differentiate with a user-defined
localhost hostname.
Optionally return (None, False) when
metadata_only is True and local-hostname data is not available.
Z localdomainZ localhostFzlocal-hostnameNT.r zip-%sr rL z%s.%s)rG rN rV r r Zget_fqdn_from_hostsfindr2 rO r
Zis_ipv4_addressZ
gethostbyaddrrA r r ) r( ZfqdnZ
resolve_ipZ
metadata_onlyZ defdomainZdefhostZdomainrX ZtoksrW Z
hosts_fqdnZlhostr r r! r sF
zDataSource.get_hostnamec C s | j j| dS )N)Z data_source)rj get_package_mirror_info)r( r r r! r _ s z"DataSource.get_package_mirror_info)source_event_typesc C sX i }xN|D ]F}x@| j j D ]2\}}||kr|j|s>t ||< || j| qW q
W |S )N)supported_update_eventsr< rN setadd)r( r supported_eventseventZ update_scopeZ
update_eventsr r r! get_supported_eventsb s
z DataSource.get_supported_events)r r& c C s | j |}xH|j D ]<\}}tjd|jdjdd |D | jd| tff qW |rp| j | j }|rpdS tjd| djdd |D d S )
a Refresh cached metadata if the datasource supports this event.
The datasource has a list of supported_update_events which
trigger refreshing all cached metadata as well as refreshing the
network configuration.
@param source_event_types: List of EventTypes which may trigger a
metadata update.
@return True if the datasource did successfully update cached metadata
due to source_event_type.
z:Update datasource metadata and %s config due to events: %sz, c S s g | ]
}|j qS r )r' ).0r r r r!
s z;DataSource.update_metadata_if_supported..z
_%s_configTz(Datasource %s not updated for events: %sc S s g | ]
}|j qS r )r' )r r r r r! r s F) r r< rx ry r' r r r r )r( r r ZscopeZmatched_eventsresultr r r! update_metadata_if_supportedo s"
z'DataSource.update_metadata_if_supportedc C s dS )NFr )r( rl r r r! check_instance_id s z DataSource.check_instance_idc C s dS )a check_if_fallback_is_allowed()
Checks if a cached ds is allowed to be restored when no valid ds is
found in local mode by checking instance-id and searching valid data
through ds list.
@return True if a ds allows fallback, False otherwise.
Fr )r( r r r! rt s z'DataSource.check_if_fallback_is_allowedc C sR |d kr t }|d krt}x4| D ],}|d kr,q ||kr8|S tjd|| |S q W |S )Nz%invalid dsmode '%s', using default=%s)DSMODE_NETWORK
VALID_DSMODESrx r )Z
candidatesdefaultZvalid candidater r r! _determine_dsmode s
z DataSource._determine_dsmodec C s d S )Nr )r( r r r! network_config s zDataSource.network_configc C s dS )a( setup(is_new_instance)
This is called before user-data and vendor-data have been processed.
Unless the datasource has set mode to 'local', then networking
per 'fallback' or per 'network_config' will have been written and
brought up the OS at this point.
Nr )r( is_new_instancer r r! setup s zDataSource.setupc C s dS )a activate(cfg, is_new_instance)
This is called before the init_modules will be called but after
the user-data and vendor-data have been fully processed.
The cfg is fully up to date config, it contains a merged view of
system config, datasource config, user config, vendor config.
It should be used rather than the sys_cfg passed to __init__.
is_new_instance is a boolean indicating if this is a new instance.
Nr )r( Zcfgr r r r! activate s zDataSource.activaterM )r] N)r^ N)r_ N)r` N)ra N)rb N)
rc rd re rf r] rg r r_ rh ri )N)T)F)FFF)NN)\r* r+ r, r Zdsmoder rm r r r r r# r. r1 r0 r/ Znetwork_config_sourcesr r r r r r r r BOOT_NEW_INSTANCEZBOOTZ BOOT_LEGACYZHOTPLUGr Zdefault_update_eventsr r r2 r r r rs Z_ci_pkl_versionr r rq r r{ r) boolr} r r r r r r r r r r r propertyr r r r r r r r r r r r r r r r r r r r r r r r r r r rt staticmethodr r r r r r r r! r s
&
R<
H$
) metaclassc C s g }| s |S t | tr | j S t | ttfr4t| S t | trxN| j D ]B\}}t |tr`|g}t |ttfrHx|D ]}|rt|j| qtW qHW |S )N)r? r2
splitlineslistr rB r< r> )Z pubkey_datakeysZ_keynameZklistZpkeyr r r! r s
r )r& c C s t |||}dd |D }t|kr&dnd} tjd| | xt||D ]\}
} tjd|
jdd d | |
f d
| |
f |d } yV| H tjd | | | ||}
|
jt j
grd
| |
f | _ |
t j
| fS W d Q R X W qD tk
r tjtd| Y qDX qDW ddj| }t|d S )Nc S s g | ]}t j|qS r )r r| )r r r r r! r s z find_source..Znetworkr z#Searching for %s data source in: %sz search-%sr r6 z searching for %s data from %szno %s data found from %s)name descriptionmessageparentz%Seeing if we can get any data from %szfound %s data from %szGetting data from %s failedz4Did not find any data source, searched classes: (%s)z, ) list_sources DEP_NETWORKrx ry zipr ZReportEventStackrA r r r r r r| r r r r r4 )rl rj rk Zds_depscfg_listpkg_listZreporterds_listZds_namesr r clsZmyrepsmsgr r r! find_source s.
r c C s g }t jd| || xt| D ]l}tj|}tj||dg\}}|sLt jd| x8|D ]0}tj|} t| d}
|
|} | rR|j| P qRW qW |S )zReturn a list of classes that have the same depends as 'depends'
iterate through cfg_list, loading "DataSource*" modules
and calling their "get_datasource_list".
Return an ordered list of classes that match (if any)
zLLooking for data source in: %s, via packages %s that matches dependencies %sZget_datasource_listzDCould not import %s. Does the DataSource exist and is it importable?) rx ry r Z"match_case_insensitive_module_name find_moduleerror
import_moduler rD ) r
dependsr Zsrc_listr% Zds_nameZm_locsZ _looked_locsZm_locmodZlisterZmatchesr r r! r s,
r system-uuid)fieldr& c C s* | sdS t j|}|sdS | j |j kS )NF)r Z
read_dmi_datar= )r r Z dmi_valuer r r! instance_id_matches_system_uuid9 s
r c C sl | st } |st }|t kr(| t kr$| S |S x2tj D ]&\}}|\}}|j|r2|| r2|S q2W | t krh| S |S )z@Lookup the canonical cloud-id for a given cloud_name and region.)r CLOUD_ID_REGION_PREFIX_MAPr< r@ )r r r prefixZ
cloud_id_testr Z valid_cloudr r r! r G s r Tc C sj | sdS t | tr| S t | tr*tj| S t | trV|dkrNt| jdddS tdtdt | dS )aL data: a loaded object (strings, arrays, dicts).
return something suitable for cloudinit vendordata_raw.
if data is:
None: return None
string: return string
list: return data
the list is then processed in UserDataProcessor
dict: return convert_vendordata(data.get('cloud-init'))
NTz
cloud-initF)recursez'vendordata['cloud-init'] cannot be dictz$Unknown data type for vendordata: %s)
r? r2 r r: r; rB convert_vendordatarN r r )datar r r r! r Z s
r c @ s e Zd ZdS )BrokenMetadataN)r* r+ r, r r r r! r r s r c C s8 g }t | }x&|D ] \}}|t |kr|j| qW |S )N)r r> )r r Zret_listZdepsetr
Zdepsr r r! list_from_depends} s r )rQ fnamer& c C sp yt j| }W n" tk
r0 tjtd| dS X ytj||ddd W n" tk
rj tjtd| dS X dS )z[Use pickle to serialize Datasource to a file as a cache.
:return: True on success
z Failed pickling datasource %sFwb )Zomoder z Failed pickling datasource to %sT)pickledumpsr r r rx r )rQ r Z pk_contentsr r r! r s r )r r& c C s d}yt j| dd}W n< tk
rR } z tjj| rBtjd| | W Y dd}~X nX |s\dS y
tj |S t
k
rz dS tk
r t j td| dS X dS )zBUse pickle to deserialize a instance Datasource from a cache file.NF)decodez failed loading pickle in %s: %sz#Failed loading pickled blob from %s) r Z load_filer r rR isfilerx r r# r r3 r )r Zpickle_contentsrz r r r! pkl_load s
r' c C s t j } tjd| }tjd| }tjd| }|p6|p6|}|p>|}|rv|jdj }t jd| d| dd| d d
|r|jdr|jdS d S ) zCheck if command line argument for this datasource was passed
Passing by command line overrides runtime datasource detection
z ds=([^\s;]+)zci\.ds=([^\s;]+)zci\.datasource=([^\s;]+)rL z7Defining the datasource on the commandline using ci.ds=z or ci.datasource=z23.2zUse ds=z instead)
deprecatedZdeprecated_versionZ
extra_messager6 )r Z get_cmdlineresearchgroupstripZ deprecate)r$ Z
ds_parse_0Z
ds_parse_1Z
ds_parse_2r% r( rm r r r! r~ s
r~ )r )T)Tabcr: r Zloggingr r# r) collectionsr enumr r Ztypingr r r r r Z cloudinitr
r r r
r r ro r Zcloudinit.atomic_helperr Zcloudinit.distrosr Zcloudinit.eventr r Zcloudinit.filtersr Zcloudinit.helpersr Zcloudinit.persistencer Zcloudinit.reportingr ZDSMODE_DISABLEDZ DSMODE_LOCALr Z DSMODE_PASSr ZDEP_FILESYSTEMr Z DS_PREFIXr ZREDACT_SENSITIVE_VALUEr r r Z getLoggerr* rx r r# r r3 r4 r5 rC rS rT rV ABCMetar r r2 r r r r r r IOErrorr r r r' r~ r r r r! s
#
+ %