MjG d dl Z d dlZd dlZd dlZd dlZd dlZd dlZd dlZd dl m Z m
Z
d dl m Z m
Z
d dlmZ d dlmZ d dlmZmZmZmZmZ d dlmZmZmZm Z d dl m Z d d l m Z d d
l!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z) d d l*m+Z+m,Z, d d l-m.Z. d d
l/m0Z0 d dl1m2Z2 ed Z3 e4g d Z5 ej6 d Z7de8fdZ9 G d dej: j; Z< G d dej= Z>d Z?d Z@e G d d ZAdeBfd ZCd ZDd-d ZEd-de eA fd ZFd-d ZGd! ZHd" ZId# ZJd$ ZKd% ZLd& ZMd' ZNd( ZOePd)k r eQ ejR ZSe3T d*eS ee&eS 5 e d+ e d, e# ej ejU e? ej ejV e@ e jW ZXeXY eO ddd dS # 1 swxY w Y dS dS ). N)datetime timedelta)ListOptional) dataclass)or_) ScrapeResult DomainAlertssetup_database
session_scopecleanup_old_data) get_domains setup_loggersave_pid_and_lock intersect)init_sentry_client)get_pkg_version)PING_TIMEOUT_STATUS_CODE
SENTRY_DNS!ERROR_DOMAINS_PING_RETRY_INTERVAL ERROR_DOMAINS_ALERT_INTERVAL
WMT_LOCK_FILEPING_CONNECTIONS&LICENSE_EXPIRED_FAREWELL_LETTER_MARKERLICENSE_CHECK_PAUSE)NotifierSupportedNotificationTypes) ErrorReport)cfg)CloudlinuxLicenseLib wmt_scanner) is_loopback
is_private
is_link_local is_reservedz
100.64.0.0/10returnc t j | n# t $ r Y dS w xY wt dd }||t fdt
D rdS j dk r t v rdS dS )z
True if `host` is a literal IP in a non-routable / blocked range.
Hostnames return False -- they are filtered after resolution by _SafeResolver.
F ipv4_mappedNc 3 8 K | ]}t | V d S N)getattr).0propips /builddir/build/BUILDROOT/cl-web-monitoring-tool-3.2.6-1.el8.cloudlinux.x86_64/opt/cloudlinux/venv/lib/python3.11/site-packages/wmt/main.py z!_is_blocked_ip..B s-
@
@72t
@
@
@
@
@
@ T ) ipaddress
ip_address
ValueErrorr+ any_BLOCKED_IP_PROPERTIESversion_CGNAT_NETWORK)hostmappedr. s @r/ _is_blocked_ipr< 4 s
!$
'
' uu R
-
-F
@
@
@
@)?
@
@
@@@ t zQ2 //t 5s
&&c & e Zd ZdZd ZddZd ZdS )
_SafeResolverz
Wraps aiohttp's ThreadedResolver and drops any resolved address that falls
in a blocked range, closing the DNS-rebinding-to-internal-host vector.
c 2 t j |i || _ d S r* )aiohttpThreadedResolver _resolver)selfargskwargss r/ __init__z_SafeResolver.__init__O s 14B6BBr1 r Nc K |"| j || d {V }n"| j ||| d {V }d |D }|st d| |S )Nc < g | ]}t |d |S )r: )r< )r, hs r/
z)_SafeResolver.resolve..W s) BBBaqy(A(ABBBBr1 zNo routable address for )rB resolveOSError)rC r: portfamilyhostssafes r/ rK z_SafeResolver.resolveR s > .00t<<<<<<< r> I sS
C C C % % % % %r1 r> c $ e Zd ZdZd fd Z xZS )_SafeTCPConnectorz
Rejects IP-literal hosts in a blocked range before connecting. aiohttp's
stock _resolve_host short-circuits IP literals and skips the resolver, which
would otherwise leave redirect targets like http://169.254.169.254/ unfiltered.
Nc K t | rt d| t ||| d {V S )Nz Blocked non-routable address: )traces)r< rL super
_resolve_host)rC r: rM r[ __class__s r/ r] z _SafeTCPConnector._resolve_hostg s` $ CA4AABB BWW**4f*EEEEEEEEEr1 r* )rS rT rU rV r]
__classcell__)r^ s @r/ rY rY ` sQ F F F F F F F F F Fr1 rY c t j d t j D }t dt
| d S )Nc & i | ]\ }}|d k ||S ) report_emailrW )r, kvs r/
z reload_conf..o s( PPP41aA ._fetch s IKKE
G";;z<;HH D18u1E1E UDH
:I+!),TTY[[55H-I)J)J %2
N
N
N'
6N9E9LN N N N N N ,8
G
G
G (
#FFFFFF
Gs; B< A!B)B< )
B33B< 6B37B< <.D,$DDzhttps://zhttp:// )
startswithlenry rw ) rv r r semaphorer https_urlhttp_url https_result http_results `` r/ pingr s' & & & & & & & & G G G G G G( >>* % % 'I 3s:'7'7#88HH
^^I
&
& '"SY %99IHH(3((I&H #VI . ....... '* $ % , , E& & & & & & & & & & & & & &H #F8 , ,,,,,,, &) #M& & & & & & & & & & & & & & & & & & & & & & & & & & & & & &s BC%9C%%
C/2C/c # K t }||t | z }t dt | |D ]*}t j | st || || V +dS )z
ping_timeout: specified in config timeout time (s) for request
semaphore: semaphore obj to handle asyncio tasks
ping_target_domains: mostly needed for re-pinging error domains
Nz Those domains will be pinged: %s)r setri debugrk r is_domain_ignoredr )r r r ping_target_domainsdomainsdomains r/ executorsr s mmG&3*+++
LL3S\\BBB A A$V,, Av|Wi@@ @ @ @A Ar1 c K t t | ||| }t | dk rg S d |D } t j t j | | d {V S # t j $ rj g }|D ]*}| s| +|D ]3} | | d {V # t j
$ r Y 0w xY w|cY S w xY w)Nr c 6 g | ]}t j | S rW )asyncio create_task)r, coros r/ rJ z scrape_sites.. s# > > >4W
&
& > > >r1 r ) listr r r wait_forgatherr donecancelappendCancelledError) ping_site_timeout
ping_intervalr r r
coroutinestasksresultstasks r/ scrape_sitesr s1 i 17IGZ[[\\J
:! > >: > > >E %gne&D,,D03D0:*F11F58F5c t j t t z
}t | 5 }| t j t j t | t t j
|k t j dk }d |D cddd S # 1 swxY w Y dS )z
- gets websites that must NOT be included in alert email:
less than ERROR_DOMAINS_ALERT_INTERVAL passed or is_resolved marker was not
changed from last alerting
)hoursFc g | ] }|j
S rW r r, rows r/ rJ z%get_recent_alerts.. s 888 888r1 N)r nowr r r r r
r r r r keysr
alert_timer )r
alert_domainsrepeat_intervalr recently_alerteds r/ get_recent_alertsr
s lnny7S'T'T'TTO v 9'"== )=>>
VL(,,T-2D2D2F2F-G-GHH //A$0E9; ;< < 98'7888 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9s B%C++C/2C/c Z t dt t | d | D }t
t j t j d|it j | S )z
prepares needed error report object with error domains
to be alerted and sends this mail
returns alerted domains
zAlerts will be sent for %sc
g | ]W\ }}t |d t t t | t | XS )z, )rv code count_errors)r joinmaprk r r )r, r codess r/ rJ zalert..$ sh
FE 3sCJJ//00 U
r1 error_report target_email
from_emailreportnotification_type)
ri rj rk r r rh r r r r r ALERTnotify) domains_datar s r/ alertr s KK,D **,,--..0 0 0 * / / 1 1
L
%> L
5:
< < < =CFHHH r1 c ` t | | fd| D }|sOt dt t | t dS t | t j
}t | 5 }| t t j }d |D }|D ]}||v r^| t t j |k t' |d d| t || ddd dS # 1 swxY w Y dS )aJ
- gets recently alerted domains (those that must not be alerted again)
and does not include them for alerting
- calls alerting for left domains
- updates DomainAlerts table:
if website was not alerted -> adds new record
if website was alerted before -> updates alert time and is_resolved marker
c $ i | ] \ }}|v ||
S rW rW )r, rc rd r s r/ re z flush_alerts..@ s* \\\A!K[B[B[1B[B[B[r1 zDAll domains "%s" were alerted or still not resolved in last %d hoursNc g | ] }|j
S rW r r s r/ rJ z flush_alerts..N s 000 000r1 F)r r )r r )r rh ri rj rk r r r r r r r r r
with_entitiesr allr r r r ) r r domains_to_alertr r websitesurlsr r s @r/ flush_alertsr 6 s ) ??\\\\ )<)<)>)>\\\ Z] / / 1 122330 2 2 2
,..C v J'== ..
]</
0
0
SUU 10x000& J JF~~
l++VL 0F :;;VDCU C C CDDDD LC H H HIIII
J J J J J J J J J J J J J J J J J J Js 4C!F##F'*F'c 0 | rt j j rdS dS )NTF)r alert_notifications_enabled)
error_domainss r/ should_be_repingedr X s < t 5r1 c t j t r5t d t j t d S d S )Nz CloudLinux license was updated)ospathexistsr ri rj removerW r1 r/ cleanup_farewell_letter_markerr ^ sM w~~<== : 4555
899999: :r1 c d t j t s_t d t
t j t j i t j dS dS # t $ r t d Y dS w xY w)z@
Sends farewell letter once (if it was not sent before)
z/Going to send last email about expired license!r z$Error while managing farewell letterN)r r r r ri warningr r r r r FAREWELLr Exception exceptionrW r1 r/ manage_license_farewellr d s Aw~~ DEE P NNL M M M - >"<"E
G
G
G HNvxxxxx
P P A A A?@@@@@@As BB $B/.B/c
K t j }t j j dz } t t | t }t
t j dd rt j | }n t |t }t j | 4 d{V }t j
| }t t j j ||| d{V }t! || } t# | rt$ dt) | t j j dz }|t* k rt j |z
}
t j t/ t1 t* |
z
d
d d{V |
t* k rot t j j |||| d{V } t! || | } t5 | | }
nnt5 | | }
| } n|
rWt$ d
t) t7 |
t9 ||
ng } ddd d{V n# 1 d{V swxY w Y n*# t: $ r t$ d Y nw xY w t j j dz }t j |z
}t/ t1 ||z
d
d }t j | d{V ||k rnmns# t j j dz }t j |z
}t/ t1 ||z
d
d }t j | d{V ||k rnmw xY w| S )aF
Scanner logic:
1. Scrapes domains and obtains ping results;
2. Manage ping results (e.g: saving to DB)
3. In case error domains found -> start re-pinging
Re-pinging:
- in min(ping_interval, 5 mins)
- flush alerts if needed
4. Sleep for ping_interval until next ping iteration
< allow_private_targetsF)limit)r resolver) connectorNz8Those domains are unsuccessful: %s
Try to re-ping themTr
)r z1Domains with unsuccessful status code found: "%s"z Error during ping iteration!) r r r r r
r r+ r@ TCPConnectorrY r>
ClientSessionr Semaphorer r r r ri rj rk r sleepminr r r r r r r )previously_erroredr r ping_interval_secondsconnections_limitr r r ping_resultr elapsed_for_pingping_retry_result retry_errorsr elapsed
sleep_times r/ scrape_iterationr t s
IKKE G 1B 6<&((( 0 2 2 37 3U ; ; ] ,3DEEEII)0AMOO\\\I(9=== * ( * ( * ( * ( * ( * ( * ( )*;<( %m $EHX$XZ[ \ \^` a a ,.OOO6B # 46KWV_4A4F4F4H4H7 7 7 1 1 1 1 1 1 - ,? &(9=;M;M;O;O, , L -6m\,R,RM ! P )22Dm(T(T
-:* / 2 ! 8KK S #D ););)=)=$>$> ? ?A A A 777 &("'U* ( * ( * ( * ( * ( * ( * ( * ( * ( * ( * ( * ( * ( * ( * ( * ( * ( * ( * ( * ( * ( * ( * ( * ( * ( * ( * (V 9 9 97888889 $'G$9B$> !ikkE)G S!6!@! D DbIIJ-
++ + + + + + + +...
$'G$9B$> !ikkE)G S!6!@! D DbIIJ-
++ + + + + + + +...
sP BK ?G$J6$K 6
K K K K M! $K/,M! .K//M! !A0Oc K t } g }d} t rd}t || d{V }n~|dz
}|dk rt d |dk rt
d}t j j dz }t |t }t j | d{V )z
Main loop for wmt_scanner_solo service
each 'while: True' iteration returns errored domains
(domains that responded with non-200 status code)
r TNr~ z)Seems your CloudLinux license is expired! r )
r r get_license_statusr ri r r r r r r r r )r r license_attemptr r s r/ scrape_loopr s
FO, ! ! 4 4 6 6 , O'78JF'S'S!S!S!S!S!S!S q O !## JKKK !##')))"#$'G$9B$> ! 24GHHJ-
++ + + + + + + + ,r1 __main__zPID: %szweb-monitoring-toolzcl-web-monitoring-toolr* )Zr r3 r rq r@ signalr concurrent.futuresr r r typingr r dataclassesr
sqlalchemyr wmt.dbr r
r r r
wmt.common.utilsr r r r clsentryr clsentry.utilsr wmt.common.constr r r r r r r r wmt.common.notificationr r wmt.common.reportr
wmt.commonr cllicenser ri frozensetr7
ip_networkr9 boolr< abcAbstractResolverr> r
rY ro rs ru r{ r r r r r r r r r r r r r rS rk getpidpidrj SIGUSR1SIGTERMget_event_looplooprun_until_completerW r1 r/ r< s:
( ( ( ( ( ( ( ( ! ! ! ! ! ! ! ! ! ! ! ! ! ! U T T T T T T T T T T T ' ' ' ' ' ' * * * * * * I H H H H H H H ) ) ) ) ) ) * * * * * * m $ $ ##`#`#`aa &%o66D *% % % % %GK0 % % %.
F
F
F
F
F,
F
F
F9 9 9 * * * * * * * *c / / /d A A A A jno kA (' ' ' 'T 9 9 9 4 J J JD : : :
A
A
A K K K \, , ,6 z
#ibikk
C
KK 3 =# . . / /0*?+CDD% ' ' '
fnk222
fnh///%w%''
.../ / / / / / / / / / / / / / / / / / s ;A/G77G;>G;