Mj d Z ddlZddlmZ ddlmZmZmZmZ ddl Z ddl m
Z
m Z m Z ddl
mZmZ e G d d Zd ee fd
Zd Zd eeee f fd Zd
ZdS )zF
Represents storage where ssa data is collected, stored and extracted
N) dataclass)ListIteratorTupleDict)funccastdistinct)
session_scope
RequestResultc D e Zd ZU eed< ee ed< eed< eed< dS )
DomainData domain_namedomain_total_reqsis_a_wordpress_domain urls_numberN)__name__
__module__ __qualname__str__annotations__r intbool J/opt/cloudlinux/venv/lib64/python3.11/site-packages/ssa/modules/storage.pyr r sF Cy r r returnc # z K t | 5 }| t j t j dt j t j t j t j t j
t j t t j
t j t j dt j t j t j dt j }t! j |d }|D ]^\ }}t% | }d}dgdz }|D ]*\ } }
} } }
| |t' |
< t ||
}+t) ||| | V _ ddd dS # 1 swxY w Y dS )z7
Iterates data from database domain-by-domain.
%Hc | d S Nr r items r z#iter_domains_data../ s
VZ[\V] r keyr )r r r r N)r queryr domainr strftime
created_atCountidmax wordpresscountr
pathgroup_byorder_by itertoolsgroupbytupler r )enginedbresults_by_hourresults_by_hour_groupedr groupdomain_results_by_hourr requests_number_by_hour_hour requests_num is_wordpressurlss r iter_domains_datarC s
v !" (( M$
8 9 9 J}' ( ( H], - - Jx
233 4 4
( M$
8 9 9
( M$
8 9 9
#,"3OI]I]"^"^"^ "9 K%*5\\ "K ()cBh #=S
5
594 |T5A'D 2 !+t 4 4 '"9&2 '
#! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! !s FF00F47F4c
# K t | 5 }| t j t j dt j t j t t j t j d t j
t j d t j t t j t j d t j |k t j | t j t j dt j t j t j dt j }d}dgdz dgdz dgdz }}}|D ]q\ } }
} } }
|r/|| k r)|t+ |||| fV dgdz dgdz dgdz }}}| |t- |
< | |t- |
< |
|t- |
< | }r| t+ | ||| fV ddd dS # 1 swxY w Y dS ) z6
Iterates urls data from database url-by-url.
r url_throttled_reqsurl_total_reqs
url_slow_reqsNr r' r1 rE rF rG )r r( r r1 r r* r+ Sumr hitting_limits
sqlalchemyIntegerlabelr, r- is_slow_requestfilterr) in_r2 r3 dictr )r7 r all_pathsr8 urls_data
previous_pathrE rF rG r1 r? url_throttled_req
url_total_req url_slow_reqs r iter_urls_datarX C s
v 1
"HH M$
8 9 9 HT ,j.@
u)** J
e$%% HT -z/AC C
eO$$
& K /
& " "9 - -
(
dM4L M M
(
dM4L M M
% ,
C"HqcBhb -:N KT ! ! FD$)=,
1 $!6!6#T&'9#1"/ & & & C"HqcBhb 5BN" -> s4yy )(5N3t99 %'3M#d)) $ MMD 1)'
Y1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
s I$JJ
Jc # K t | 5 }| t j t j t j |k t j }t j |d }|D ]\ }}|d |D fV ddd dS # 1 swxY w Y dS )zA
Get information about durations of requests url-by-url.
c | d S r! r r" s r r$ z#get_url_durations.. s
DG r c g | ]\ }}|S r r ).0r> durations r
z%get_url_durations.. s :::[Q:::r N)
r r( r r1 r] rO r) r3 r4 r5 )r7 r r8 rS durations_by_pathr& r; s r get_url_durationsr` { s#
v ;"HH "
& K /
(
& - ++ - -+ ; ;JC::E:::: : : : : ;; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ;s BB77B;>B;c
# K t | 5 }| t j t j dt j t j t t j t j d t j
t j d t j t t j t j d t j |k t j t j dt j t j t j dt j }| t j t j t j |k t j }t+ j |d }t+ j |d }t/ |d }t/ |d }|||\ } }
|\ } } | | k rd gd
z }
d gd
z }d gd
z }|
D ]>\ }}}}}||
t1 | < ||t1 | < ||t1 | < ?d | D }| |t3 | |
|| fV t/ |d }t/ |d }n'| | k rt/ |d }nt/ |d }||ddd dS # 1 swxY w Y dS )
u
Stream per-URL data for a single domain.
Yields ``(path, durations, url_data)`` per path, where:
- ``path`` is the URL path (str);
- ``durations`` is a list of raw request durations for that path;
- ``url_data`` is a dict with per-hour aggregates
(``url_throttled_reqs``, ``url_total_reqs``, ``url_slow_reqs``
as 24-element lists) plus ``path``.
Merges two ``ORDER BY path`` streams in lock-step instead of
materialising the per-domain ``{path: [durations...]}`` dict up
front, which caused OOM on large databases (CLPRO-3077). A second
benefit: this drops the redundant ``WHERE path IN (...)`` clause —
both queries already filter by ``domain``, so the path list adds
no selectivity, only memory pressure on the SQL parser.
r rE rF rG c | d S r! r rs r r$ z&iter_domain_url_data..
1 r r% c | d S r! r rc s r r$ z&iter_domain_url_data.. re r Nr r' c g | ]\ }}|S r r )r\ r> ds r r^ z(iter_domain_url_data.. s 5 5 541aQ 5 5 5r rH )r r( r r1 r r* r+ rI r rJ rK rL rM r, r- rN rO r) r2 r3 r] r4 r5 nextr rQ )r7 r r8 agg_query dur_query agg_by_path dur_by_pathagg_itemdur_itemagg_path agg_groupdur_path dur_grouprE rF rG r> r? throttledtotalslow durationss r iter_domain_url_datarx s $
v E3"HH M$
8 9 9 HT ,j.@
u)** J
e$%% HT -z/AC C
eO$$
& K /
( M$
8 9 9
( M$
8 9 9
# , HH "
& K /
(
' ~~FFF ' ~~FFF T** T**"x';"* Hi"* Hi8##&'S2X""#r!"b
7@ 4 43AtYt4=&s4yy105N3t99-/3M#d)),, 5 59 5 5 5 4!'9#1"/ , , , T22 T22H$$
T22 T22? "x';ME3 E3 E3 E3 E3 E3 E3 E3 E3 E3 E3 E3 E3 E3 E3 E3 E3 E3s L"MMM)__doc__r4 dataclassesr typingr r r r rK r r r
ssa.dbr r r rC rX r r r` rx r r r r} s@ ! ! ! ! ! ! . . . . . . . . . . . . + + + + + + + + + + / / / / / / / / %*!5 % % % %N5
5
5
p;d3c
?.C ; ; ; ;*W3 W3 W3 W3 W3r