Wjn d dl Z d dlZd dlZd dlZd dlZd dlZd dlZd dlmZmZ d dl m
Z
d dl m Z m
Z
mZ d dlmZ d dlZd dlmZ d dlmZmZ d dlmZ d d lmZ d d
lm Z d d l m Z m Z m Z m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z' G d d
Z(dS ) N)datetimetimezone)Path)DictAnyOptional)deepcopy) PKG_VERSION)cpusers userdomains)drop_privileges)ApacheProcessor)DocrootProcessor)
VERSION MAJOR_VER STORAGE_DIR OPT_OUT_FILEDOCROOT_SCAN_TIMEOUTDOCROOT_LIMIT_PER_RUNSENDING_TIMEOUTSENDING_COMPRESSIONMAX_RECORD_SIZEFILE_SIZE_LIMITc v e Zd ZdZd1dej dee dee fdZ de
de
fd Z de
fd
Z d2d Z
d2d Zd
edee fdZd2dZdedefdZdededdfdZd2dZedefd ZdeddfdZdededededede
eef f d Zd e
eef defd Zd e
eef de
eef fd Z d
ed!ed"ed#e
eef d e
eef d$e de
eef fd%Z d&ed'ed(ed)edef
d*Z d&ed'edededededed+edefd,Z ded-e ded&ed'edededededed+edefd.Z! d3dededededed&ed+ed'eddfd0Z"dS )4WebsiteCollectoraB
Website data collection.
Collects and exports various Apache-related data:
- Apache configuration files (.conf) from standard directories
- Apache system information (version, loaded modules)
- .htaccess files from website document roots
- Maintains processing state to avoid duplicate scans
Nlogger storage_dir max_sitesc |t }|| _ || _ | j dt | j | | t
| j | _ t | j | _ |dz | _
|dz | _ di dd | _ t | j | _ | | dS )
a
Initialize website collector for on-demand data collection and sending.
Args:
storage_dir: Directory for storing website collector state
max_sites: Maximum number of sites to process (with hash-based selection)
Nz1[WEBSITE-COLLECTOR] Initialized with max_sites=%s)r )r z website_collector_stats.jsonz
dry-run.jsonlFr )global_apache_configs_sentdocrootscollected_version)r r r debugstr_ensure_directoriesr docroot_processorr apache_processor
stats_file dry_run_filesending_stat_defaultsr
sending_stats_load_sending_statsreset_cache_within_version)selfr r r s ]/opt/cloudlinux/venv/lib64/python3.11/site-packages/cl_website_collector/website_collector.py__init__zWebsiteCollector.__init__5 s %K " MsSWSaObObccc
[ 999 "2!E!E!E /t{ C C C &(FF '/ 9*/!"&
&
"
&d&@ A A
"""
'')))))
docroots_datareturnc | j |S t | j di }t d| j |z
}|dk r#| j d|| j i S t | |k r|S g }| D ]U\ }}t j |
}| |||f V|
d i }|d| D ] \ } }}|||< | j d||t | t | |S )a_
Hash and sort docroots, then return top N based on max_sites limit.
Takes into account already sent sites to respect the global limit.
Args:
docroots_data: Dict with docroot as key and data as value (unprocessed only)
Returns:
Dict with selected docroots (limited and sorted by hash)
Nr! r z5Max sites limit reached: %d already sent, limit is %dc | d S )Nr )xs r/ z:WebsiteCollector._hash_and_sort_docroots.. s
1Q4 r1 )keyzYHash-based selection: %d already sent, %d remaining limit, selected %d from %d candidates)r lenr+ getmaxr r# itemshashlibsha256encode hexdigestappendsort)
r. r2 already_sent_countremaining_limithashed_docrootsdocrootdata docroot_hashselected_docroots_s
r/ _hash_and_sort_docrootsz(WebsiteCollector._hash_and_sort_docroots\ s > ! !!3!7!7
B!G!G H H a 2D!DEE a K U 0$.
B
B
BI } 0 0 * 0 0 2 2 B BMGT">'..*:*:;;EEGGL " "L'4#@ A A A A 000 /0@ 0@ A . . Aw)- g & & ;,o /00#m2D2D F F F
! r1 c d i }d} t }|D ])} t | }i }|D ]\ }}t | } | rut j t
| t j rI| | r|dz
}o||vr ||g d||< || d | d}
|
D ]&\ }} t | t k rd}
n| ||< '|
r n5# t $ r'} | j d|| Y d} ~ #d} ~ ww xY wn2# t $ r%} | j d | Y d} ~ nd} ~ ww xY w| j d
t | | | | S ) z
Get docroots for sending, with optional filtering and limit.
Returns:
Dict of docroots ready for processing (limited and filtered by sent status)
r )rG usernamedomainsrP FTz%Failed to get domains for user %s: %sNz([WEBSITE-COLLECTOR] Panel API failed: %sz>>sA E C0D
E
D> D93E 9D>>E
E1 E,,E1c j | j ret | j dd 5 }t j | | _ | j d ddd dS # 1 swxY w Y dS dS # t $ r&}| j d| Y d}~dS d}~ww xY w)z+Load sending statistics from tracking file.rutf-8encodingzLoaded sending statsNz Failed to load sending stats: %s) r( rQ openjsonloadr+ r r# rV r. fr` s r/ r, z$WebsiteCollector._load_sending_stats s E%%''
>$/3AAA >Q)-1 D&K%%&<===> > > > > > > > > > > > > > > > > >
>
> E E E K @! D D D D D D D D D Es: 0B 4A3& B 3A77B :A7;B
B2 B--B2c | j dd }|t k rdS | j dt t t | t | j | _ dS )z
Sync sending stats cache within the same version to remove entries for docroots
that no longer exist in the system.
r" r Nzj[WEBSITE-COLLECTOR] New version of collector, current version=%s, previously collected=%s re-setting cache) r+ r; r r infor$ r r r* )r. prev_collected_vers r/ r- z+WebsiteCollector.reset_cache_within_version s " / 3 34G K K * * F DEH\\SVWiSjSj l l l &d&@ A Ar1 file_pathc t | }| }|r| d n|}| r@| r,t j t | t
j s | j
d| dS | j t k r5| j
d|| j dS | dd S # t $ r'}| j
d || Y d}~dS d}~ww xY w)
zx
Read any file content on-demand for sending.
Supports both Apache configs and .htaccess files.
F)strictz([WEBSITE-COLLECTOR] Cannot read file: %sNz5[WEBSITE-COLLECTOR] File too big: %s (size: %d bytes)rd replace)rf errorsz-[WEBSITE-COLLECTOR] Error reading file %s: %s)r
is_symlinkresolverQ is_filerR rS r$ rT r rW statst_sizer read_textrV )r. ro prt real_pr` s r/ read_file_contentz"WebsiteCollector.read_file_content sB
YA J0:AQYYeY,,,FMMOO
(8(8
RYs6{{TVT[=\=\
!!"LiXXXt{{}}$66 !!"Y[d"(++--"79 9 9t##WY#GG G K MyZ[ \ \ \44444 s% B,D 0AD D
E( E
Ec t j t j | j d< t | j d< t | j dd 5 }t j
| j |dd d d d n # 1 swxY w Y | j d
d S # t $ r&}| j d | Y d }~d S d }~ww xY w) z)Save sending statistics to tracking file. last_updatedr" wrd re F)indent ensure_asciiNzSaved sending statsz4[WEBSITE-COLLECTOR] Failed to save sending stats: %s)r nowr utc isoformatr+ r rg r( rh dumpr r# rV rW rj s r/ _save_sending_statsz$WebsiteCollector._save_sending_stats s< Y19 hl1K1K1U1U1W1WD ~ .6?D 2 3dosW===
O $,aNNNN
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O K 3 4 4 4 4 4 Y Y Y K TVW X X X X X X X X X Ys< A B1 B
> B1
BB1 B B1 1
C!;C C!rG c ~ | j di |d }t | S )zMCheck if docroot data was fully sent (both htaccess and user apache configs).r! F)r+ r; bool)r. rG
docroot_statss r/ rU z WebsiteCollector.is_docroot_sent s8 *..z2>>BB7ERR
M"""r1 rO c || j d vrd|i| j d |< t j t j | j d | d< dS )zMark docroot data as sent.r! rO last_processedN)r+ r r r r r )r. rG rO s r/ mark_docroot_sentz"WebsiteCollector.mark_docroot_sent
sj $,Z8 8 8H7D z *7 3 EMLQYQ]D^D^DhDhDjDj:&w/0@AAAr1 c d| j d< dS )z#Mark global apache configs as sent.Tr N)r+ )r. s r/ mark_global_apache_sentz(WebsiteCollector.mark_global_apache_sent s ;?7888r1 c ( t j S )z
Check if website collection is disabled via opt-out file.
Returns:
True if collection is disabled, False otherwise
)r rQ r6 r1 r/ is_collection_disabledz'WebsiteCollector.is_collection_disabled s "$$$r1 c 6 | ddd dS )z
Create necessary directory structure.
Args:
storage_dir: Directory for storing website collector state
i T)modeparentsexist_okN)mkdir)r. r s r/ r% z$WebsiteCollector._ensure_directories$ s% udTBBBBBr1 client_id system_idplatformpanelserverc l t t j t t |||||i i d
S )z&Create base record with common fields.)
collection_timestampcollection_versioncollected_package_versionr r r r r rH meta_data)inttimer r
)r. r r r r r s r/ _create_base_recordz$WebsiteCollector._create_base_record- s? %( $4$4"))4""
r1 recordc 0 t j |dd S )z`
Serialize record to JSON.
Returns:
Serialized JSON string
F),:)r
separators)rh dumps)r. r s r/ _serialize_recordz"WebsiteCollector._serialize_record= s z&uLLLLr1 base_recordc B | }i |d< i |d< |S )z
Create new record part with base data but empty data/meta_data sections.
Args:
base_record: Base record to copy from
Returns:
New record with empty data sections
rH r )copy)r. r
new_records r/ _create_new_record_partz(WebsiteCollector._create_new_record_partF s/ !%%''
6"$
; r1 file_contentrt current_recordrecords_to_sendc " | }|d |d< |d |d< ||d |< |r
ddi|d |< | | }t | d } | t k rb|d rZ| | }
| |
| | }||d |< |r
ddi|d |< n|}|S )a
Add file to current record, splitting if size limit exceeded.
Args:
file_path: Path of the file to add
file_content: Content of the file
is_symlink: Whether the file is a symlink
current_record: Current record being built
base_record: Base record template for new parts
records_to_send: List to append completed records to
Returns:
Updated current record (may be new if split occurred)
rH r rt Trd )r r r: r@ r rB r ) r. ro r rt r r r temp_recordtemp_serialized temp_sizecurrent_serializeds r/ _add_file_to_recordsz%WebsiteCollector._add_file_to_recordsU s8 $ %))++ ,V 4 9 9 ; ; F#1+#>#C#C#E#E K *6 FI& G3?2FK $Y / 00==..w7788 & &>&+A & "&!7!7!G!G " "#5 6 6 6 ! 9 9+ F FN 1=N6 "9 -
N:F9M{+I6 )N r1 api_urldry_runserialized_dataremote_conf_verc |r} t | j dd 5 }| |dz ddd n # 1 swxY w Y dS # t $ r&}| j d| Y d}~dS d}~ww xY wd t d
| d }|} t r| d } t j
| d
}
t | d } t |
} d| | z z
dz }
| j d| d| d|
t j |
d } t | }| j d| d|d d}d}d}t! | D ]} | || |d }|j dk r dS |j }||dz
k r=| j d|j |dz | t' j d|z n | j d|j # t $ rv}t+ | }||dz
k r8| j d|dz || t' j d|z n| j d | Y d}~d}~ww xY w|rd | }n
|rd | }nd }| j d || dS )!zJSend record or save to JSONL file for dry run. Returns True if successful.ard re
NTz3[WEBSITE-COLLECTOR] Error saving dry-run record: %sFz
text/plainrK )z Content-TypezX-Website-Collector-Version )
compresslevelrN d z;Applied gzip compression: %s -> %s bytes (%.1f%% reduction)r asciiz Base64 encoded: %s -> %s bytes )rH headerstimeout z"Non-200 response %s, attempt %d/%dr z)Final attempt failed: Non-200 response %sz HTTP error on attempt %d/%d: %sz=[WEBSITE-COLLECTOR] Final HTTP error while sending record: %szHTTP z Exception: z
Unknown errorzF[WEBSITE-COLLECTOR] Failed to send record after %d retries. Reason: %s)rg r) writerV r rW r r r@ gzipcompressr: r# base64 b64encodedecoderangepost status_coder sleepr$ )r. sessionr r r r rk r` r data_to_sendcompressed_dataoriginal_bytescompressed_bytes reduction base64_bytes max_retries
last_errorlast_status_codeattemptresponsereasons r/ _send_or_collect_recordz(WebsiteCollector._send_or_collect_record s I
$+S7CCC 4qGGOd 23334 4 4 4 4 4 4 4 4 4 4 4 4 4 4t
!!"WYZ[[[ uuuuu
!-29/M/MO/M/M G +L"
P . 5 5g > > "&- A"N"N"N!$_%;%;G%D%D!E!E##7 !1N!B Bc I !!"_%3"7"7! [Z Z Z
1<0000 ))*U*2*>@ @ @ ! n n n!$QJ q 00 ))*K*1A+{AG G G
1<0000 ))*iklmmmn
)3!133
)3z33( K f )6
3
3
35sY A
A A
AA
AA
A=A88A=$H$6A-H$$
J$.A+J J$remote_config_versionc
| ||||| }
|
dg dd | |
} g } | j }
|
dg }| j dt | t |d D ]y\ }}| j d|d |
|d }|d }| |d || d
d | |
| } z| j }| d g | d
d< | dd | d
d< |
| }| | t | D ]M\ }}| j d|dz t | | ||||| s d S NdS )z
Send global Apache configs and system info, splitting into multiple records if needed.
Returns:
True if all records sent successfully, False otherwise
rO rP
document_root config_pathsz#Found %d global Apache config filesrN z!Processing global config file: %sro N# ERROR: File unreadablert Floaded_modulesrH modulesapache_versionversionz"Sending global config record %d/%dT)r updater r' collect_global_config_pathsr; r r# r: enumerater| r get_apache_system_infor rB r )r. r r r r r r r r r r r r global_configsr i path_infor system_infor r s r/ _send_global_configsz%WebsiteCollector._send_global_configs s< " ..y)XuV\]]
55kBB .JJLL%))."== ?\ARARSSS %lA 6 6
LAy K A9[CY Z Z Z 11)K2HIIL# 9 ! 6 6+& immLRW6X6X _ NN +BBDD ,7OO%>*;7BTBXBXYeglBmBm& _& &NN
,OOQYPZ[[
L , ,^ < <
**
!-n!= I#'#9#9)K:P#Q#QL#+'A %)%>%>!+. immLZ_>`>`& _& &NN
& ! 7 7!%!7!7!G!G " "#5 6 6 6 #,O"<"< A K ?!a%QTUdQeQe f f f//'?\qrr
uu
ts& A C??DD A F77F;>F;Fc t j } |st j nd}
| j dd s| j d|rdnd | |
|||||||| rD|sB| | | j d |
} | D ]\ } }
t j | z
t k r | j
d n| j d | |
d
| | |
d |
d
|
|||||||| rM|sK| | |
d
| | j d | t j | z
}|r"| j d
| j | n| j d| n7# t$ $ r*}| j
d|rdnd| Y d}~nd}~ww xY w|
r|
dS dS # |
r|
w w xY w)a
Send data directly or save to JSONL file in dry-run mode.
Read htaccess and apache configs on-demand and track sending status.
Args:
dry_run: If True, saves data to dry-run.jsonl file instead of sending to API
Nr Fz%s global Apache configs
CollectingSendingz'Global Apache configs sent successfullyz=[WEBSITE-COLLECTOR] Timeout reached during docroot processingz!Processing docroot %s for user %srO rP z Docroot %s sent successfullyz.DRY RUN completed: saved to %s in %.1f secondsz*Sequential sending completed: %.1f secondsz([WEBSITE-COLLECTOR] Failed during %s: %szdry runzsequential sending)r requestsSessionr+ r; r r# r r r ra r= r rW r r rm r) rV close)r. r r r r r r r r
start_timer unprocessed_docrootsrG domain_dataexecution_timer` s r/ send_datazWebsiteCollector.send_dataS s " Y[[
,3=("$$$; %))*FNN
U !!"dll[deee,,W-4-4-6-6-5-2-3-BD D U # U 44666 00222 ))*STTT $(#A#A#C#C (<(B(B(D(D
S
S$9;;+o==K%%&efffE !!"EwP[\fPghhh**7+6y+A+6z+B+2+2+2+4+4+3+0+1+@ B B S # S ..w J8OPPP 00222 ))*H'RRR !Y[[: 5N
` !QSWSdftuuuu !!"NP^___ ! ! ! K H+2 Lii8L
!
!
!
!
!
!
!
! !
w
s0 G4H! I2 !
I+ I I2 II2 2J )NN)r3 N)F)#__name__
__module__ __qualname____doc__loggingLoggerr r r r0 r rL ra r, r- r$ r| r r rU r r staticmethodr r% r r r r listr r r r r r6 r1 r/ r r * s %* %*w~ %*HTN %*^fgj^k %* %* %* %*N0!T 0!d 0! 0! 0! 0!d:?4 :? :? :? :?xE E E E
B
B
B
B 3 8C= 2
Y
Y
Y
Y#s #t # # # #
k k k k k k k@ @ @ @ %D % % % \%Ct C C C C C
S
S
C
X[
eh
mq Hn
MS#X M3 M M M M
4S>
d3PS8n
. c . . RV . -1#s(^. JNsTWx.. .2. 7;CH~. . . . `P),P *.P 25 P
25 P
:> P P P Pd;&); '+; ), ;
), ; (+
; %(; &); 58; =A; ; ; ;z=# = = =_b =$(=58=EH=%(=14=>A=Z]=bf= = = =N #(P P P P P P
P
P *-P P -1P P P P P P r1 r ))rh r rR r r r r> r r pathlibr typingr r r r r r lve_utilsr
clcommon.cpapir r clcommon.clpwdr
%cl_website_collector.apache_processorr &cl_website_collector.docroot_processorr cl_website_collector.constantsr r r r r r r r r r r r6 r1 r/ r s
' ' ' ' ' ' ' ' & & & & & & & & & & ! ! ! ! ! ! / / / / / / / / * * * * * * A A A A A A C C C C C C y y y y y y y y y y r1