Control Web Panel
WebPanel => Updates => Topic started by: Daan on July 01, 2026, 06:00:25 AM
-
Since this night, I'm missing my extra DB users on multiple servers.
Not all, only the ones with a different name than the DB itself.
Is that related to the last update?
-
same thing, after that night missing /root/.ssh/ at all my panels
-
Hi. I've run into the same problems. All additional database users were deleted, and the `/root/.ssh` directory was removed as well.
Is there any information available about this incident? Could it be caused by an update error?
-
Hi,
The same update (0.9.8.1239) pushed a new script today at /usr/local/cwpsrv/htdocs/resources/scripts/temp_hacker_check.
Be careful if you are on AWS or other cloud providers: the script contains a hardcoded list of users to lock down, and centos is included.
Even if your server is completely clean, the script automatically:
Deletes the entire /home/centos/.ssh folder (wiping legitimate keys).
Sets the shell to /sbin/nologin.
-
Hi,
The same update (0.9.8.1239) pushed a new script today at /usr/local/cwpsrv/htdocs/resources/scripts/temp_hacker_check.
Be careful if you are on AWS or other cloud providers: the script contains a hardcoded list of users to lock down, and centos is included.
Even if your server is completely clean, the script automatically:
Deletes the entire /home/centos/.ssh folder (wiping legitimate keys).
Sets the shell to /sbin/nologin.
Thanks for the information! Could you clarify whether this means the script detected malicious code or backdoors in the SSH keys?
Is there a risk that it will delete the users and keys again after I re-add them?
The logs only contain an entry about a successful cleanup.
-
Hi,
To answer your questions based on my logs and the script's code analysis:
Did it detect actual malicious code on my server?
No. The script did not detect any actual backdoors or malicious keys on my instances. In the script, the HACKER_KEY detection block logs specifically to /var/log/hacker_cleanup.log with phrases like "Detected malicious access for user...". In my case, that log only has the Starting and Completed timestamps with absolutely nothing in between (it took less than a second to run). It was a false positive/collateral damage.
Why did it delete the keys and lock the user then?
Because at the end of the script, there is a hardcoded list of USERS (which includes centos, mysql, postfix, etc.) that gets forcefully locked down regardless of whether they are infected or not. The script runs chsh -s /sbin/nologin and rm -rf "$homedir/.ssh" on every single user in that list as a preventive "aggressive cleanup" measure.
Is there a risk it will delete them again?
Yes, absolutely. Since this script runs automatically via the nightly CWP cronjob, it will check the USERS list every single night. If you manually re-add the keys or re-enable the centos user, the script will wipe it again tonight at 03:00+ AM.
The only permanent workaround right now is to leave the centos user locked and create a completely custom administrative user (e.g., my_admin) that is not included in CWP's hardcoded system users list.
The script /usr/local/cwpsrv/htdocs/resources/scripts/temp_hacker_check
#!/bin/bash
# Define the malicious key fragment and log file
HACKER_KEY="IPCsi58xDKuXuq8CMnlIFQHoqiGkyziMQpAks2t0EBa0"
LOG_FILE="/var/log/hacker_cleanup.log"
echo "--- Starting security cleanup: $(date) ---" >> "$LOG_FILE"
# Iterate through all directories in /home
for home_dir in /home/*; do
# Ensure it's a directory
if [ -d "$home_dir" ]; then
user=$(basename "$home_dir")
ssh_dir="$home_dir/.ssh"
auth_keys="$ssh_dir/authorized_keys"
# Check if the malicious key exists in the authorized_keys file
if [ -f "$auth_keys" ] && grep -q "$HACKER_KEY" "$auth_keys"; then
echo "Detected malicious access for user: $user" >> "$LOG_FILE"
# 1. Remove the entire .ssh directory to ensure all backdoors are gone
chattr -i -R "$ssh_dir"
rm -rf "$ssh_dir"
echo " [OK] Removed .ssh directory for $user." >> "$LOG_FILE"
# 2. Set shell to nologin to prevent future SSH or su access
usermod -s /sbin/nologin "$user"
echo " [OK] User $user set to nologin." >> "$LOG_FILE"
# 3. Lock the user account (password locking)
usermod -L "$user"
echo " [OK] User $user account locked." >> "$LOG_FILE"
fi
fi
done
echo "--- Cleanup completed: $(date) ---" >> "$LOG_FILE"
# Manual fix
chattr -i /root/.ssh/authorized_keys
sed -i '/IPCsi58xDKuXuq8CMnlIFQHoqiGkyziMQpAks2t0EBa0/d' /root/.ssh/authorized_keys
chown root:root /root/.ssh/authorized_keys
chmod 600 /root/.ssh/authorized_keys
usermod -L operator
usermod -L mysql
chsh -s /sbin/nologin operator
chsh -s /sbin/nologin mysql
if [ -f "/var/lib/mysql/.ssh" ];then
chattr -i -R /var/lib/mysql/.ssh
rm -Rf /var/lib/mysql/.ssh
fi
# Hacker host
grep -q "mya.cloudsyndication.org" /etc/hosts || echo "127.0.0.1 mya.cloudsyndication.org" >> /etc/hosts
grep -q "gsocket.io" /etc/hosts || echo "127.0.0.1 gsocket.io" >> /etc/hosts
# List of all system users to inspect and secure
USERS=(
bin daemon adm lp sync shutdown halt mail operator games ftp nobody
systemd-network dbus polkitd rpc rpcuser nfsnobody sshd postfix
chrony centos ntp mysql saslauth dovecot dovenull tss named
cwpsrv cwpsvc login clamupdate amavis clamscan vmail vacation
opendkim postgres
)
echo "=== Starting Aggressive System User & SSH Key Cleanup ==="
for u in "${USERS[@]}"; do
# Check if the user actually exists on this system
if id "$u" &>/dev/null; then
# 1. Lock the password
passwd -l "$u" &>/dev/null
# 2. Disable login shell, EXCEPT for sync, shutdown, and halt
if [[ "$u" != "sync" && "$u" != "shutdown" && "$u" != "halt" ]]; then
chsh -s /sbin/nologin "$u" &>/dev/null
fi
# 3. Locate the home directory dynamically
homedir=$(eval echo "~$u")
if [ -d "$homedir/.ssh" ]; then
echo "[TARGETFOUND] Found backdoor .ssh directory for user '$u' at: $homedir/.ssh"
# --- THE FIX FOR 'OPERATION NOT PERMITTED' ---
# Recursively strip immutable (+i) and append-only (+a) flags from the folder and everything inside
chattr -R -i -a "$homedir/.ssh" 2>/dev/null
chattr -i -a "$homedir/.ssh/authorized_keys" 2>/dev/null
# Now perform the aggressive removal
rm -rf "$homedir/.ssh"
# Double check if the removal was successful
if [ -d "$homedir/.ssh" ]; then
echo "[CRITICAL FAILURE] Could not remove $homedir/.ssh even after stripping attributes!"
else
echo "[SUCCESS_CLEANED] Successfully eradicated backdoor keys for user: $u"
fi
fi
fi
done
echo "=== Aggressive cleanup complete ==="
# Define the target directory for custom sudoers rules
SUDO_DIR="/etc/sudoers.d"
# List of known malicious files created by the hacker matching the system users
MALICIOUS_FILES=(
adm amavis bin clamscan clamupdate cwpsrv cwpsvc daemon dbus
dovecot dovenull ftp games halt login lp mail mysql named
nginx nobody opendkim operator polkitd postfix redis saslauth
shutdown sshd systemd-network tss vacation vmail
)
echo "Starting cleanup of /etc/sudoers.d/..."
for file in "${MALICIOUS_FILES[@]}"; do
TARGET_PATH="$SUDO_DIR/$file"
# Check if the malicious file actually exists
if [ -f "$TARGET_PATH" ]; then
# Double check the content to make sure it contains the dangerous NOPASSWD string
if grep -q "NOPASSWD" "$TARGET_PATH"; then
rm -f "$TARGET_PATH"
echo "[REMOVED] Malicious sudoers file deleted: $TARGET_PATH"
else
echo "[WARNING] File $file exists but content does not match pattern. Skipped for safety."
fi
fi
done
echo "Sudoers backup/backdoor cleanup complete."
echo "=== Starting GSocket (gs-netcat) Neutralization ==="
# STEP 1: Immediately block gsocket infrastructure via /etc/hosts
# This kills their connection even if a hidden binary attempts to run.
echo "Blocking GSocket network domains..."
for domain in gsocket.io www.gsocket.io; do
if ! grep -q "$domain" /etc/hosts; then
echo "127.0.0.1 $domain" >> /etc/hosts
echo "[BLOCKED] Added $domain to /etc/hosts"
fi
done
# STEP 2: Kill all active hacker processes related to gsocket
# We target 'gs-netcat', 'gsocket', and connections pointing to gsocket IPs.
echo "Hunting and killing active GSocket processes..."
pkill -f gs-netcat
pkill -f gsocket
pkill -f "gsocket.io"
# STEP 3: Scan and clean persistence points (Cron jobs)
# The attacker likely injects this script into crontabs to restart it every minute.
echo "Scanning crontabs for gsocket persistence..."
for user in $(cut -d: -f1 /etc/passwd); do
if crontab -u "$user" -l 2>/dev/null | grep -qE "gsocket|gs-netcat"; then
echo "[CRON MALWARE] Found in user: $user. Cleaning..."
# Strip lines containing the malware keywords
crontab -u "$user" -l | grep -vE "gsocket|gs-netcat" | crontab -u "$user" -
fi
done
# System-wide cron directories cleanup
echo "Scanning system cron directories..."
find /etc/cron* /var/spool/cron -type f 2>/dev/null | while read -r cronfile; do
if grep -qE "gsocket|gs-netcat" "$cronfile"; then
echo "[SYSTEM CRON] Removing malware lines from: $cronfile"
sed -i '/gsocket/d' "$cronfile"
sed -i '/gs-netcat/d' "$cronfile"
fi
done
# STEP 4: Remove files injected into user profiles (.bashrc, .profile)
# Check all users' login scripts for the '# gsocket - start' block you found.
echo "Scanning user profiles for bash-injected hooks..."
find /home /root -maxdepth 3 -type f -name ".*rc" -o -name ".profile" 2>/dev/null | while read -r profile; do
if grep -q "gsocket" "$profile"; then
echo "[PROFILE HOOK] Found in $profile. Stripping block..."
# This removes everything between the start and end comments of gsocket
sed -i '/# gsocket - start/,/# gsocket - end/d' "$profile"
# Backup safety delete if comments were stripped but keyword remains
sed -i '/gsocket/d' "$profile"
sed -i '/gs-netcat/d' "$profile"
fi
done
echo "=== GSocket remediation complete ==="
echo "=== STARTING COMPREHENSIVE MYSQL SECURITY CLEANUP ==="
# ==========================================
# PHASE 1: EXPLOSIVE DROPS & DYNAMIC HOST FIXES
# ==========================================
# Define explicit whitelist of accounts allowed to hold Grant_priv
ALLOWED_ADMINS="('root', 'mariadb.sys', 'debian-sys-maint')"
echo "[1/4] Hunting for obvious wildcard backdoor accounts..."
mysql -B -N -e "SELECT User, Host FROM mysql.user WHERE Host='%' AND User IN ('dbadmin', 'sys', 'mariadb', 'admin');" | while read -r user host; do
if [ -n "$user" ]; then
echo "[DELETE] Dropping blatant backdoor user: '$user'@'$host'"
mysql -e "DROP USER '$user'@'$host';" 2>/dev/null
fi
done
echo "[2/4] Eradicating wildcard (%) accounts to enforce local-only access..."
# Since localhost versions already exist, we MUST DROP the '%' versions to avoid primary key conflicts.
mysql -B -N -e "SELECT User, Host FROM mysql.user WHERE Host='%' AND User NOT IN $ALLOWED_ADMINS;" | while read -r user host; do
if [ -n "$user" ]; then
echo "[ERADICATE WILDCARD] Physically dropping remote account: '$user'@'$host'"
mysql -e "DROP USER '$user'@'$host';" 2>/dev/null
fi
done
# Apply the drops immediately
mysql -e "FLUSH PRIVILEGES;"
# ==========================================
# PHASE 2: FORENSIC HASH-BASED CLEANUP
# ==========================================
echo "[3/4] Scanning for leftover wildcard accounts with EMPTY passwords..."
mysql -B -N -e "SELECT User, Host FROM mysql.user WHERE Host='%' AND (Password='' OR authentication_string='');" | while read -r user host; do
if [ -n "$user" ]; then
echo "[SECURITY ALERT] User '$user'@'$host' has NO PASSWORD! Generating secure random password..."
RANDOM_PW=$(head /dev/urandom | tr -dc A-Za-z0-9 | head -c 32)
mysql -e "ALTER USER '$user'@'$host' IDENTIFIED BY '$RANDOM_PW';" 2>/dev/null
fi
done
echo "[4/4] Analyzing database for shared hacker signature password hashes..."
HACKER_HASHES=$(mysql -B -N -e "SELECT Password FROM mysql.user WHERE User IN ('dbadmin', 'mariadb', 'sys') AND Host='%' AND Password != '' UNION SELECT authentication_string FROM mysql.user WHERE User IN ('dbadmin', 'mariadb', 'sys') AND Host='%' AND authentication_string != '';" | sort -u)
if [ -n "$HACKER_HASHES" ]; then
for hash in $HACKER_HASHES; do
echo "[FOUND HACKER SIGNATURE] Backdoor hash pattern detected: $hash"
mysql -B -N -e "SELECT User, Host FROM mysql.user;" | while read -r user host; do
user_hash=$(mysql -B -N -e "SELECT Password FROM mysql.user WHERE User='$user' AND Host='$host' UNION SELECT authentication_string FROM mysql.user WHERE User='$user' AND Host='$host';" | grep -v '^$' | head -n 1)
if [ "$user_hash" == "$hash" ]; then
if [ "$user" != "root" ]; then
echo "[ERADICATE] Dropping user '$user'@'$host' due to hacker password hash match."
mysql -e "DROP USER '$user'@'$host';" 2>/dev/null
else
echo "[CRITICAL WARNING] Root is using the hacker password! CHANGE ROOT PASSWORD IMMEDIATELY!"
fi
fi
done
done
fi
# Apply all corrections permanently
echo "Flushing privileges to apply changes permanently..."
mysql -e "FLUSH PRIVILEGES;"
echo "=== COMPREHENSIVE MYSQL CLEANUP COMPLETE ==="
# Roudcube Cleaner
chmod 755 /usr/local/cwpsrv/var/services/roundcube/temp
chmod 755 /usr/local/cwpsrv/var/services/roundcube/logs
RC_DIR="/usr/local/cwpsrv/var/services/roundcube"
TEMP_DIR="$RC_DIR/temp"
echo "=== Commencing Roundcube Directory Security Hardening ==="
if [ -d "$RC_DIR" ]; then
# STEP 1: Fix broad permissions across the Roundcube installation
# Change 777 permissions to secure 755 (owner can write, others can only read/execute)
echo "Hardening broad directory permissions from 777 to 755..."
find "$RC_DIR" -type d -perm 777 -exec chmod 755 {} \;
# STEP 2: Wipe all contents inside the temp folder (including hidden malicious files)
echo "Purging all existing hacker implants from temp directory..."
find "$TEMP_DIR" -mindepth 1 -delete
# STEP 3: Enforce strict file ownership and secure permissions
# We ensure cwpsvc owns it, but completely block world-writable access (755)
echo "Enforcing strict owner permissions on the temp directory..."
chown -R cwpsvc:cwpsvc "$RC_DIR"
chmod 755 "$TEMP_DIR"
# STEP 4: Inject a protective web server rule directly inside Roundcube's directory
# Even if Nginx conf is bypassed, an internal .htaccess drop acts as an immediate fallback barrier
echo "Creating PHP execution block (.htaccess) inside the temp directory..."
cat << 'EOF' > "$TEMP_DIR/.htaccess"
<Files *>
SetHandler default-handler
</Files>
RemoveHandler .php .phtml .php3 .php4 .php5 .php6 .php7 .php8
php_flag engine off
# deny webserver access to this directory
<ifModule mod_authz_core.c>
Require all denied
</ifModule>
<ifModule !mod_authz_core.c>
Deny from all
</ifModule>
EOF
# Secure the .htaccess file itself so it cannot be modified by the web server easily
chmod 644 "$TEMP_DIR/.htaccess"
chown root:root "$TEMP_DIR/.htaccess" 2>/dev/null || chown cwpsvc:cwpsvc "$TEMP_DIR/.htaccess"
echo "[SUCCESS] Roundcube temp folder sterilized and locked down."
else
echo "[ERROR] Roundcube directory not found at $RC_DIR"
fi
# Fix roundcube access to temp and logs
cat << 'EOF' > /usr/local/cwpsrv/conf/cwp_services.conf
location /pma {
root /usr/local/cwpsrv/var/services;
index index.html index.htm index.php;
location ~ \.php$ {
try_files $uri =404;
fastcgi_split_path_info ^(.+\.php)(/.+)$;
fastcgi_read_timeout 600;
fastcgi_pass unix:/usr/local/cwp/php71/var/sockets/cwpsvc.sock;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_param SCRIPT_NAME $fastcgi_script_name;
fastcgi_param PHP_ADMIN_VALUE "open_basedir = /usr/local/cwpsrv/var/services/pma/:/tmp/";
include fastcgi_params;
}
location ~* \.(js|css|png|jpg|jpeg|gif|ico)$ {
access_log off;
log_not_found off;
expires 1M;
}
}
location /roundcube {
root /usr/local/cwpsrv/var/services;
index index.html index.htm index.php;
# BLOCK HACKER ACCESS TO TEMP AND LOGS DIRECTORIES
location ~ ^/roundcube/(temp|logs)/ {
deny all;
return 403;
}
location ~ \.php$ {
try_files $uri =404;
fastcgi_split_path_info ^(.+\.php)(/.+)$;
fastcgi_read_timeout 600;
fastcgi_pass unix:/usr/local/cwp/php71/var/sockets/cwpsvc.sock;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_param SCRIPT_NAME $fastcgi_script_name;
fastcgi_param PHP_ADMIN_VALUE "open_basedir = /usr/local/cwpsrv/var/services/roundcube/:/tmp/:/usr/local/cwp/php71/lib/";
include fastcgi_params;
}
location ~* \.(js|css|png|jpg|jpeg|gif|ico)$ {
access_log off;
log_not_found off;
expires 1M;
}
}
location /phpPgAdmin {
root /usr/local/cwpsrv/var/services;
index index.html index.htm index.php;
location ~ \.php$ {
try_files $uri =404;
fastcgi_split_path_info ^(.+\.php)(/.+)\$;
fastcgi_read_timeout 600;
fastcgi_pass unix:/usr/local/cwp/php71/var/sockets/cwpsvc.sock;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_param SCRIPT_NAME $fastcgi_script_name;
fastcgi_param PHP_ADMIN_VALUE "open_basedir = /usr/local/cwpsrv/var/services/phpPgAdmin/:/tmp/";
include fastcgi_params;
}
location ~* \.(js|css|png|jpg|jpeg|gif|ico)$ {
access_log off;
log_not_found off;
expires 1M;
}
}
EOF
-
While analyzing the script, I couldn't find any reason for it to delete the entire `/root/.ssh` folder. The script uses `sed` to remove a specific key, so this behavior looks strange.
Part of the script appears to be overly aggressive:
SELECT User, Host FROM mysql.user WHERE Host='%' AND User NOT IN ('root', 'mariadb.sys', 'debian-sys-maint');
It removes all users where `HOST = %`.
In my setup, all users have `HOST = %`, because database access is controlled by a whitelist at the provider level.
Will this script be fixed?
-
Us too. We have many users that are % but the firewall is stopping access.
Now we have to re add all these users across multiple servers. All while customers are screaming.
Totally unacceptable.
-
Feels like an AI programming (a dumb one)...
From temp_hacker_check:
# Since localhost versions already exist, we MUST DROP the '%' versions to avoid primary key conflicts.
wtf? Primary key conflicts into mysql users table? Smart... /not
So, the only way to keep our servers working as expected is to run a .sql file restoring the db users at 03:01AM?
I don't know if the cwp staff even reads this forum...
-
I believe its an attemp of the programers to close cwpro forever, this cannot happend, we are paying for this licence.
Come on , wtf are you doing? breaking everything people worked for.
OMG so unreliable i have like 25 servers, ALL BROKEN. i just want to die.
-
Does anyone knows if there is a way to BLOCK CWP updates? and install them when we want?
-
What if I block the script until better times? How will the system react to this during an update?
chmod 400
chown root:root
chattr +i
-
I believe its an attemp of the programers to close cwpro forever, this cannot happend, we are paying for this licence.
Come on , wtf are you doing? breaking everything people worked for.
OMG so unreliable i have like 25 servers, ALL BROKEN. i just want to die.
That is odd.
All of our servers are still intact with all the user and system accounts.
But seeing this thread did make my heart skip a beat.
I am curious, what OS are you guys that lost users running?
-
To disable a bash script (such as the hacker check), simply insert exit 0 after the first shebang line:
#!/bin/bash
exit 0For a PHP script such as cron.php, insert the exit function at the start of the script:
exit()
-
I am curious, what OS are you guys that lost users running?
Alma 8
-
Some of my clients having the same issue. The privilege users are gone and need to be readd.
-
Mmm can confirm strange happening with this update.
No SQL user problems for me but it did make some sort of change. I use certs instead of user/password to access SFTP & all my logins were denied this morning. I had to allow password auth and then match users with IPs to secure as a work around.
-
question: And who is the hacker right now?
-
Hi,
This update has caused serious problems on my servers. For the last two days, MySQL users have been deleted repeatedly.
As a temporary workaround, I added an `exit` at the beginning of the `temp_hacker_check` script to prevent it from running again, because it was causing damage to my configuration.
Another problem is that the cleanup log is extremely basic. It only shows when the script starts and finishes, but it does not clearly log what users were deleted, what changes were applied, or why those actions were taken. This makes it very difficult to audit the damage or understand exactly what happened.
In my opinion, this script is not at the level of quality and safety expected from CWP. A security cleanup script should not silently delete MySQL users or modify server access without clear logging, confirmation, backups, or at least a safer detection method.
Please review this update and the `temp_hacker_check` script urgently, because it can break production servers and remove legitimate database users.
Thank you.
-
Just a little recommendation: in addition to running daily backups, also consider running AutoMySQLBackup to dump you DB daily:
https://github.com/sixhop/AutoMySQLBackup
-
I believe its an attemp of the programers to close cwpro forever, this cannot happend, we are paying for this licence.
Come on , wtf are you doing? breaking everything people worked for.
OMG so unreliable i have like 25 servers, ALL BROKEN. i just want to die.
That is odd.
All of our servers are still intact with all the user and system accounts.
But seeing this thread did make my heart skip a beat.
I am curious, what OS are you guys that lost users running?
Alma 8.
All the servers had users with @% cause i need to, everything went to shit.
-
So, what should we do now regarding `temp_hacker_check`?
I have temporarily added an `exit` at the beginning of the script to prevent it from running. Should I also remove its write/execute permissions?
Honestly, I do not feel comfortable having this script running on my servers. Everything was working fine before, and what I need most is stability. I can take care of security on my side.
In some services, I do have MySQL users with host `%`. Is that really considered so abnormal or dangerous that they should be removed without prior notice or confirmation?
In my opinion, updates like this should not be pushed to the Stable branch, especially if Stable is supposed to mean “at least two weeks tested.”
I would like to know what you plan to do with this script, so I can prepare accordingly and also have an official recommendation on how to proceed. For the past two days, I have been waking up to complaints from clients because they are getting errors when trying to connect to their databases.
I hope that, for now, adding an `exit` at the beginning of the script is enough to prevent further issues, but I do not think this is a proper long-term solution.
I really hope you can review whether something went wrong with this update and adjust it properly. From my side, I want to reiterate that I would prefer to be able to disable this script completely, as I do not need it.
Thank you!
-
This things keep being deleted everyday. Lol
-
And here we have the issues that happen when we have no changelog of new updates... We never know what have changed.