9Mjg" d Z ddlZddlZddlZddlmZmZ ddlmZ ddlm Z m
Z
m Z ddl m
Z
mZmZ G d d ZdS ) z[
This module contains FaultDetector class, aimed to track throttling
for incoming requests
N)datetime timedelta)RLock)OptionalTupleAny)
drop_after check_periodthrottling_thresholdc Z e Zd ZdZdZdZd Zdedede e
ef fdZ e defd Z
e defd
Ze defd Ze de
fd Zed
edefd ZdedefdZd dededee ddfdZdeddfdZdedede e
ef fdZd dZd dZdefd Z dS )
FaultDetectorz
Fault Detector class
i' i c t j d | _ t | _ t j t | _ t | _ t | _ d| _ d S )Nfault_detectorz$/usr/share/alt-php-xray/flush.latest)
logging getLoggerloggerdictthrottling_stat collections defaultdictint_uid_entry_count
_entry_uidr locktime_marker_fileselfs P/opt/cloudlinux/venv/lib64/python3.11/site-packages/xray/agent/fault_detector.py__init__zFaultDetector.__init__% sZ '(899 #vv !, 7 < < &&GG F t_key cpu_valuereturnc | j 5 | j | \ }}| || \ }}| | n# t $ r d\ }}Y nw xY wddd n # 1 swxY w Y ||fS )z)
Perform fault detection
)Fr N)r r _was_throttled_since _flush_entryKeyError)r r! r" saved_cpu_hitting_limitsthrottled_times r __call__zFaultDetector.__call__0 s Y ) )
) # 3E : 1 261J1J y2* 2*.!!%((((
:
:
:19.
: ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ~--s3 A*A /A* AA*AA**A.1A.c B t t j S )z(
Current Unix timestamp
)r timer s r timestampzFaultDetector.timestamp? s
49;; r c 6 | t S )z8
Drop after value as datetime.timedelta
)timedelta_in_minutesr r s r _drop_afterzFaultDetector._drop_afterF s
((444r c 6 | t S )z:
Check period value as datetime.timedelta
)r1 r
r s r
_check_periodzFaultDetector._check_periodM s
(( 666r c t | j 5 }t | }ddd n # 1 swxY w Y | | | j k S # t t f$ r7 | j 5 |
ddd n # 1 swxY w Y Y dS w xY w)ze
If throttling_stat should be flushed right now
regarding check period value
NF) openr r readstrip
current_deltar4 OSError
ValueErrorr _update_latest_flush)r latest_infolatest_flush_times r should_flushzFaultDetector.should_flushT sS d+,,
D $' (8(8(:(:(@(@(B(B$C$C!
D
D
D
D
D
D
D
D
D
D
D
D
D
D
D%%&7884;MM M$
,
,))+++
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,55 sX A? 4A
A? AA? A A? ?CB8, C8B< <C?B< CCminsc " t | S )zB
Get timedelta object for given number of minutes
)minutes)r )r@ s r r1 z"FaultDetector.timedelta_in_minutesc s
&&&&r
time_stampc ^ dt dt fd} || j || z
S )zQ
Calculate timedelta for given time_stamp from current timestamp
tsr# c * t j | S )z>
Make a datetime object from ts (int)
)r
fromtimestamp)rE s r _castz*FaultDetector.current_delta.._casto s )"-- -r )r r r/ )r rC rH s r r9 zFaultDetector.current_deltaj sG
.c .h . . . . uT^$$uuZ'8'888r Nkeycpuuidc | j }| j 5 || j v}|rU|S| j |d | j k r/| j d| j | ddd dS |rXt | j | j k r;| j dt | j ddd dS | j
d||| ||f| j |< |r!|| j |< || j |xx dz
cc< | j
dt | j ddd dS # 1 swxY w Y dS )a
Save throttling entry for given key.
:param uid: SO_PEERCRED uid of the connecting peer (when known). A
per-UID quota is enforced so one uid's request-init flood
cannot saturate the shared cap and starve other tenants.
Nr zEthrottling_stat per-uid quota (%d) reached for uid=%s, dropping entryz0throttling_stat at capacity (%d), dropping entryz(Adding new throttling entry %s: (%i, %i) Throttling entries total %i) r/ r r r get_MAX_ENTRIES_PER_UIDr warninglen _MAX_ENTRIESdebugr )r rI rJ rK stampis_news r savezFaultDetector.savew s
Y 9 9 44F
#/ -11#q99T=VVV ## %=>B>WY\^ ^ ^ 9 9 9 9 9 9 9 9
#d 233t7HHH ##$V$'(<$=$=? ? ? 9 9 9 9 9 9 9 9
K H !3
/
/
/),e D %
4 (+$?)#...!3... K ; !$"6 7 7
9
9
9/ 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9s AE0A
E
A9EEEc | j 5 | j d| || j v }| j |d |rj| j |d }|M| j |d dk r.| j |xx dz cc< | j | dk r| j |= | j dt | j ddd dS # 1 swxY w Y dS )z6
Remove throttling entry by given key
zAbandon throttling entry %sr NrM rN ) r r rT r popr r rO rR )r rI existedrK s r r& zFaultDetector._flush_entry sX Y 9 9 K ;S A A AT 11G $ $S! , , ,
7o))#t44?t'<'@'@a'H'H1'L'L)#...!3...,S1Q66 1# 6 K ; !$"6 7 7
9
9
9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9s CC..C25C2
initial_value
current_valuec ||z
dz }|t k }| j dt || ||dk r|ndfS )z
Check current throttling time of given lve_id vs given initial_value
Return: tuple(fact of throttling, throttling time in ms)
i@B z"Throttling checked with %i: %s, %ir )r r rT )r r[ r\
spent_timethrottling_facts r r% z"FaultDetector._was_throttled_since s\ $m3?
$';; >.): 7 7 7 j1nn
!CCr c | j r| j 5 | j dt | j | D ]}| | | | j dt | j ddd dS # 1 swxY w Y dS dS )z,
Clear throttling stat dict
z Flush started, total entries %iz Flush finished, total entries %iN) r? r r rT rR r _expired_entriesr& r< )r rI s r flushzFaultDetector.flush s =
=
= !!"C"%d&:";";= = = 0022 + +C%%c****))+++ !!"D"%d&:";";= = =
=
=
=
=
=
=
=
=
=
=
=
=
=
=
=
=
=
= = =s BB77B;>B;c t | j d 5 }| t | j ddd dS # 1 swxY w Y dS # t
$ r | j d Y dS w xY w)z?
Write current timestamp into time marker file
wNz(Failed to save timestamp of latest flush)r6 r writestrr/ r: r error)r r= s r r< z"FaultDetector._update_latest_flush s Jd+S11
7[!!#dn"5"5666
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7 J J J K H I I I I I I Js3 A (A A AA AA $B Bc t }| j D ][\ }}|\ }}| | }|| j k r1| j d|| | | \|S )ze
Collect expired throttling stat entries keys
regarding the drop after value
z%Expired entry detected %i: %s elapsed)listr itemsr9 r2 r rT append)r expiredrI valuer) r/ _deltas r ra z FaultDetector._expired_entries s
&& .4466 $ $JC LAy'' 22F((( !!"I"%v/ / /s###r )N)r# N) __name__
__module__ __qualname____doc__rS rP r r r r boolr, propertyr/ r r2 r4 r? staticmethodr1 r9 r rW r& r% rb r< ri ra r r r
r
s< L G G G
.c
.c
.eD#I6F
.
.
.
. 3 X 5Y 5 5 5 X5 7y 7 7 7 X7 d X '3 '9 ' ' ' \' 9 9 9 9 9 9!9 !9 !9# !9HSM !9T !9 !9 !9 !9F9 9 9 9 9 9$
D,/
D,/
D49$)4D
D
D
D
D = = = = J J J J
$
r r
)rr r r r. r r threadingr typingr r r xray.internal.constantsr r
r r
rv r r rz s ( ( ( ( ( ( ( ( ' ' ' ' ' ' ' ' ' ' R R R R R R R R R RJ J J J J J J J J Jr