MjU d Z dZddlZddlZddlZddlmZ ddlmZm Z m
Z
ddl m Z m Z ddl
mZ ddlmZmZmZmZmZ dd lmZ ddlZdd
lmZ dd lm Z dd l m Z dd
l m Z m!Z! ddl"m#Z#m$Z$m%Z% ddl&m'Z' ddl(m)Z) ddl*m+Z+m,Z, ddl-m.Z. dZ/ ddl0m1Z2 n# e3$ rZ4e4Z/dZ2Y dZ4[4ndZ4[4ww xY we G d d Z5e G d d Z6e G d d Z7 G d de Z8de9d e8fd Z: G d d e Z;d e9fd Zd#k rH e. ej? e= e< ej@ $ e; ZA eA dS dS )%z'
This module contains Autotracer class
z ssa.modules N) defaultdict) dataclassfieldasdict)datetime timedelta) fnmatchcase)IteratorOptional
NamedTupleListIterable)urlparse)setup_database)SSAError)Common)iter_domains_dataiter_domain_url_data)statusdisabled_usersmisconfiguration_checks)
load_tunables)autotracing_tunables_schema)autotracing_logautotracing_stats_file) sentry_initc Z e Zd ZU dZeed< eed< eed< eed< eed< eed< eed< d S )
URLzURL data containeruri avg_duration max_durationtotal_slow_reqs
total_reqs
belongs_tobelongs_to_userN)__name__
__module__ __qualname____doc__str__annotations__int M/opt/cloudlinux/venv/lib64/python3.11/site-packages/ssa/modules/autotracer.pyr r , sc HHHOOOOOOr. r c e Zd ZU dZdZeed< dZeed< dZeed< dZ eed< dZ
eed< dZ eed< dZ eed < dZ
eed
< dZeed < dZeed < dZeed
< dS ) RulesRejectszRejects counters containerr non_wp throttleddensityslowness max_slownessdisablednginx frequency server_limit domain_limit no_domainN)r& r' r( r) r2 r, r+ r3 r4 r5 r6 r7 r8 r9 r: r; r< r- r. r/ r1 r1 9 s $$FCOOOIsGSHcL#HcE3NNNIsL#L#Isr. r1 c \ e Zd ZU dZeed< dZeed< dZeed< e e
Z e
ed< dS ) Statsz AutoTracer statistics container
rules_versionr urls_processed
urls_selected)default_factoryrejectsN) r& r' r( r) r* r+ r@ r, rA r r1 rC r- r. r/ r> r> J sa ))NCM3 !E, ? ? ?G\?????r. r> c ( e Zd ZU dZeed< eed< dS )URLSz"
Representation of an URL
domain_nameuri_pathN)r& r' r( r) r* r+ r- r. r/ rE rE T s1 MMMMMr. rE urlreturnc t | }|j r
d|j nd}|j r |j | nd}|j dd }| d d }t
|| S )z
Split URL into domain_name and uripath including query string
:param url: URL of format protocol://domain/path;parameters?query#fragment
:return: namedtuple URL(domain_name, uripath)
? /zwww.:r )r querypathnetlocreplacesplitrE )rH fragmentsqsr _no_www_netloc_no_port_netlocs r/ url_splitrX ] s
I"+/ 9 Y_ rB%.^
r? rf urls_scheduledr rA request_number Exceptionsave_iteration_stats)rh exc
start_toolrH eri s r/ __call__zAutoTracer.__call__z s # % % % % K 5sz B B B FFFFF w~~ ?@@
L !!G)**
!JKKK F /
! K 1 2 2 2 GG K
Says if nginx is enabled for a specific user
NF)
skip_nginxrn NginxUserCache
is_enabled)rh r s r/ nginx_is_enabledzAutoTracer.nginx_is_enabled s3 ? A u*955@ @ur. c t d | j D }| j d| | j |z
S )z>Recalculate limit per server taking into account running tasksc B g | ] }|d dk |d dk | S )r runninguserz
*autotracing*r- ).0tasks r/
z. s8 q q qdh 91L1LQUV\Q]apQpQpTQpQpQpr. z&Number of running autotracing tasks %s)lenre ra rt per_server_limit)rh
running_counts r/ per_server_limit_recalculatedz(AutoTracer.per_server_limit_recalculated sM q qdo q q q
BMRRR$}44r. full_listc |.t j }| j d| t j t
| j z
fd|D S )zQ
Excludes tasks older than N days from the general list of tasks
NzTask list loaded %s)daysc g | ]S}| d | d ndt k Q|TS )
createtimeNr )getr, timestamp)r r
n_days_agos r/ r z1AutoTracer.excludes_old_tasks.. sg
*.((<*@*@*L &&&RSWZ[e[o[o[q[qWrWrrr
rrrr. )rn re ra rt r nowr min_retracing_interval)rh r r s @r/ excludes_old_tasksz AutoTracer.excludes_old_tasks s{ +--I K 3Y ? ? ? \^^iT5P&Q&Q&QQ
!
r. current_urlc \ t | }| j D ]}t |d }|j |j k o|j |j k }t |j |j ot |j |j }|s|r%| j d||d dS dS )z
Excludes url from the list if it completely matches the current url
or if domain names match and "*" follows the domain name in the list
rH z2Skipped: URL %s was traced recently. Matched by %sTF)rX re rF rG r ra rt )rh r c task_datat direct_matchwildcard_matchs r/ exclude_thesame_urlsz AutoTracer.exclude_thesame_urls s
k " " I)E*++A =AM9VajAJ>VL (
F F n;WXWacdcmKnKnN
~
!!H e$
tt
ur. url_total_reqsdomain_total_reqsc | j rMt j t j || }| j d| || j k S dS )z-Check that URL density passes given thresholdzCalculated density %sT)r4 npamincorrcoefra rt density_threshold)rh r r url_densitys r/ pass_by_densityzAutoTracer.pass_by_density sT < 8 '"+n>O"P"PQQK K 5{ C C C !77 7tr.
url_durationsc d| j z
}t j || }| j d|| || j k S )z
The measure of "slowness" for URL is:
at least N% of its requests take more than X seconds.
N% -- self.slow_duration_percentage
X -- self.slow_duration_threshold
d z.Calculated %sth percentile %s for min duration)slow_duration_percentager
percentilera rt slow_duration_threshold)rh r reversed_percentilereversed_percentile_values r/ pass_by_slowness_percentilez&AutoTracer.pass_by_slowness_percentile sX "D$A A$&M-AT$U$U! < %
)D,HHHr. c t j || j }| j d| j | || j k S )a
The opposite to pass_by_slowness_percentile method.
The measure of "much slowness" for URL is:
at least N% of its requests must take less than X seconds.
N% -- self.max_slow_duration_percentage
X -- self.max_slow_duration_threshold
z.Calculated %sth percentile %s for max duration)r r max_slow_duration_percentagera rt max_slow_duration_threshold)rh r percentile_values r/ pass_by_max_slowness_percentilez*AutoTracer.pass_by_max_slowness_percentile sO = 8YZZ < -
4#CCCr. url_throttled_reqsc |dS t | t | z dz }| j d| || j k S )zY
Check that percent of throttled requests per URL passes given threshold
NFr z Calculated throttled percent %s)sumra rt allowed_throttling_percentage)rh r r throttled_percents r/ pass_by_allowed_throttlingz%AutoTracer.pass_by_allowed_throttling
sX %5 !3 4 4s>7J7J Jc Q ;=NOOO D$FFFr. wp_statusc 2 | j rt | S dS )zx
Check that URLs of a particular domain should be analyzed.
For now we skip non-wordpress sites
T)only_wpbool)rh r s r/ pass_by_enginezAutoTracer.pass_by_engine s
< # ?? "tr. c # K t | j D ]}d}| |j sQ| j d |j }| j j xj |z
c_ | j xj
|z
c_
pt | j |j D ][\ }}}|| j
v r| j d| | j xj
dz
c_
| | d |d s5| j d | j j xj dz
c_ | |d |j s5| j d | j j xj dz
c_ | | s6| j d | j j xj dz
c_ F| | s6| j d
| j j xj dz
c_ |Y| |j }|6| j d | j j xj dz
c_ |j }t7 |t9 t; j | t? |d t? |d tA | |j |
V ]dS )a
Select all URLs suitable for auto tracing by very basic rules:
- WP site
- suitable throttling
- suitable density
- measure of "slow" URL
ORDER OF RULES MUST NOT BE CHANGED: IT AFFECTS STATISTICS COUNTERS
Nz Skipped by engine: non-wordpresszProcessing URL %srk r r z Skipped by throttled percentzSkipped by densityz Skipped by slowness percentilez"Skipped by max slowness percentilez"Skipped by unavailable domain info
url_slow_reqs)r r" r# r! r$ r% )!r rg r is_a_wordpress_domainra rt urls_numberrf rC r2 r@ r rF non_url_fieldsr r r3 r r r4 r r5 r r6 r r< r r r, r meanr max)rh domain_data domain_owner
skipped_countrH durationsdatar s r/ urls_computationally_filteredz(AutoTracer.urls_computationally_filtered" s. -T[99 J J K L&&{'HII
!!"DEEE !, 7
"))]:))
))]:)) )=T[+Ja(b(b 8
8
$Y$--- !!"5s;;;
))Q.))66txx@T7U7UW[\lWmnn K%%&DEEEJ&00A500 ++D1A,BKDabb K%%&:;;;J&..!3.. 77 BB K%%&FGGGJ&//14// ;;IFF K%%&JKKKJ&33q833 '"&"6"6{7N"O"OK"* ))*NOOO
*44944 '2'7 !$RWY%7%7!8!8$'_(=$>$> "4(8#9 : :!$Y * 6$0 a8
%J J r. stats_collectedc # K | | D ]}|j | j v r;| j d|j | j j xj dz
c_ K| |j r;| j d|j | j j xj
dz
c_
|V dS )z
From selected by computed thresholds URLs take those for which:
- autotracing enabled
- nginx disabled
ORDER OF RULES MUST NOT BE CHANGED: IT AFFECTS STATISTICS COUNTERS
z'Skipped: autotracing is disabled for %srk z Skipped: nginx is enabled for %sN) fill_in_disabled_usersr r% r ra rt rf rC r7 r r8 )rh r rH s r/ rA zAutoTracer.urls_selectedw s
##%%%5577
C"d&999 !!"KSM`aaa
"++q0++$$S%899
!!"DcFYZZZ
"((A-((IIII
r. c # K d}t t }| }| j d| | | _ | }|D ]}| |j r| j
j xj dz
c_ 7||k rv||j
| j k r |dz
}||j
xx dz
cc< |V r| j d|j | j | j
j xj dz
c_ | j d|j | | j
j xj dz
c_ dS )a=
Schedule autotracing by sorted list taking into account the limits:
- no same task for 10 days
- limit per server
- limit per domain
ORDER OF RULES MUST NOT BE CHANGED: IT AFFECTS STATISTICS COUNTERS
Return resulting list of URLs scheduled for auto tracing
r zSorted scheduled list %srk z#Skipped URL %s by domain limit (%s)z#Skipped URL %s by server limit (%s)N)r r, urls_sortedra rt r re r r r rf rC r9 r$ per_domain_limitr; r: )rh general_tasks_countertasks_counter_per_domain sorted_urlsper_server_smart_limitrH s r/ rv zAutoTracer.urls_scheduled s !" #.s#3#3 &&(( 4kBBB 1133!%!C!C!E!E 5 5C((11
",,1,,$'===+CN;d>SSS)Q.),S^<<<A<<< IIII K%%= -
J&33q8333 !!9G*
"//14///9 5 5r. c t t | d d }t |d d S )zd
Sort URLs by total number of requests first
and by average duration second
c | j S N)r us r/ z(AutoTracer.urls_sorted.. s q~ r. T)keyreversec | j S r )r# r s r/ r z(AutoTracer.urls_sorted.. s r. )sortedrd rA )rh first_series r/ r zAutoTracer.urls_sorted sJ
T$"4"4"6"6 7 7=U=U_cddd k'='=tLLLLr. c , t | _ dS )z6
Fill internal list of disabled users
N)r )rh s r/ r z!AutoTracer.fill_in_disabled_users s - . .r. c D | j t t d 5 }t j t | j | ddd dS # 1 swxY w Y dS # t $ r3}| j dt | Y d}~dS d}~ww xY wdS )zI
Save collected statistics for current iteration to file
Nwz*Unable to save iteration stats to file: %s)
rf openr jsondumpr OSErrorra warningr* )rh
stats_filer| s r/ ry z AutoTracer.save_iteration_stats s : !
Z0#66 >*IfTZ 0 0*===> > > > > > > > > > > > > > > > > >
Z
Z
Z ##$PRUVWRXRXYYYYYYYYY
Z " !s: A (A A AA AA
B *(BB c x t t 5 }t j | }t |d |d |d t di |d }ddd n # 1 swxY w Y n6# t t j t f$ r t | j }Y nw xY wt | S )z@
Load statistics for latest iteration from file
r? r@ rA rC Nr- ) r r r loadr> r1 r JSONDecodeErrorKeyErrorr? r )rh r _data stat_datas r/ load_iteration_statsz AutoTracer.load_iteration_stats s
2,--
*-- !/**+/* 445#344
-x8 2 2 2 d 011III 2i s5 A7 A A+ A7 +A//A7 2A/3A7 70B*)B*c | }| t t d t t |S )rL r )r disabled_users_quantity)r updatedictr r r )rh stats_loadeds r/ get_statszAutoTracer.get_stats sS 0022 D SQ_QaQaMbMb c c cdddr. r )rI N)&r& r' r( r) r^ r} staticmethodr r r* r objectr r r r, r rd r r r r r r r r r
r r rA rv r
r r ry r r
__classcell__)ri s @r/ rZ rZ k s# = = = = = =#$ #$ #$ #$ #$J Nt N N N \N 1S 1Xf-= 1 1 1 \1# $ 5s 5 5 5 5
HTN
d
*d t PT I# I4 I I I I DXc] Dt D D D D GXd^ G]a Gfj G G G G 4 Sx} S S S Sj Xd^ xPS} 2.5
.5 .5 .5 .5`MT#Y M M M M/ / / / Z Z Z Z!d ! ! ! !" 4 r. rZ c t j t j t d S )a= Absolute path of the standalone auto-tracer log.
Always resolves inside the fixed, root-owned clos_ssa log directory
(derived from ``autotracing_log``) rather than against the caller's cwd,
so the destination cannot be redirected by the directory the dev entry
point happens to be launched from.
zauto_tracer_standalone.log)ro rP joindirnamer r- r. r/ standalone_log_pathr s) 7<<88:V W WWr. log_pathc t j | t j t j z t j z t j z d }t j |d S )a Open ``log_path`` for appending and return the file object.
Uses ``O_NOFOLLOW`` so a symlink pre-planted at the target is refused
rather than followed: the open raises instead of appending through the
link to an arbitrary root-owned file.
i a)ro r O_WRONLYO_CREATO_APPEND
O_NOFOLLOWfdopen)r log_fds r/ open_standalone_logr sA WXr{RZ 7"+ E
UW\
]
]F
9VS ! !!r. __main__)streamlevel)Br) rc r r_ ro collectionsr dataclassesr r r r r fnmatchr typingr
r r r
r urllib.parser numpyr ssa.dbr ssa.internal.exceptionsr ssa.modules.commonr ssa.modules.storager r ssa.autotracingr r r ssa.configurationr ssa.configuration.schemesr ssa.internal.constantsr r ssa.internal.utilsr rr xray.shared_libraryshared_libraryrn ImportErrorr| r r1 r> rE r* rX rZ r r r& basicConfigDEBUGr r- r. r/ r) s # # # # # # 0 0 0 0 0 0 0 0 0 0 ( ( ( ( ( ( ( ( A A A A A A A A A A A A A A ! ! ! ! ! ! ! ! ! ! ! ! , , , , , , % % % % % % G G G G G G G G K K K K K K K K K K + + + + + + A A A A A A J J J J J J J J * * * * * * ******* HHHHHH
@ @ @ @ @ @ @ @ : &3 &4 & & & & D D D D D D D D N XS X X X X"# " " " " zKMMM G 2 23F3F3H3H I IQXQ^____
AACCCCC s B B"B B"