Control Web Panel
WebPanel => Information => Topic started by: blue-yu on March 22, 2023, 06:09:11 PM
-
Hello,
Today I was informed by our national domain registry that couple of my ip addresses are doing some malicious activities. Also they suspected that it is "Ebury Linux SSH rootkit/backdoor trojan".
And really after malware scan of all my servers (5 servers), everyone was having this trojan. Any hint on how can those trojans get in to the system, and how to prevent future problems?
-
Yep got a notification to that this was the case. Would love to know whats going on here.
-
What are you using for your malware scan? We got the same notice but our scans are coming up clean.
-
I found trojan on all servers. I went to Security/Security Center/Malware Scan and then selected Custom scan from / (root) folder. It took a while but Trojan was found. Other things was also found in emails but it's normal :) Unix.Dropper.Ebury-9906999-0
was found in usr/lib64/libkeystats.so. I have no idea how it got there. All of my server have different password generated at random.org (for example dr3Zd^VQnyy2q^w3). SSH port is not default.
-
My server admin said they saw rumblings about other CWP servers having this issue starting on March 17th -20th. Maybe the March 20th update was a patch?
Im running the scans now, will probably take a while. What OS were you running on the infected machines? One of mine was Centos 8 and the other was AlmaLinux 8.5. Also were you on a fully updated CWP?
Are you planning on re-building from scratch? From what I read thats the only definite solution.
-
The same thing happened to me. 2 servers with CentOS 7 and CWP 7 were infected. I did a quick analysis and it turned out that the infection happened more than 11 months ago for sure. Another thing I found out is that the server was not infected 3 years ago. I learned this from the backups I have for these 2 servers. So, the infection did not happen on March 17-20... Keep this in mind.
-
just did a complete scan. Took 108 minutes scanning almost half a million files. I dont have it
-
Ok, I'am infected too...
I am scaning now with clam, will clam clean it?
Thx!
-
Once the system is compromised, it is unknown which backdoors are open.
-
so only way to get rid it is clean os install?
-
Maybe. The first thing you can do is change the SSH port and restrict access to SSH login for all users on the system to trusted IP addresses. Change the passwords for absolutely all users. This does not solve the problem since CWP is compromised and requests can be executed as root from there, but somehow it ensures that the server is not used for botnets - DDoS, email spam, etc. As I said, the infection through CWP was long ago. Personally, I think one of my servers was infected minutes before 03.09.2021, 03:46:34, because the logs before that are missing, and it has been online since 2020. I also restored backups and the infection existed 2-3 years ago. Even if the server is cleaned, as long as the vulnerability in CWP exists, it is still under threat. Personally, I will wait for the CWP bug to be fixed and then reinstall the server with the new CWP panel.
-
By the way, which hosting provider do you use? My servers are with hetzner.com.
-
By the way, which hosting provider do you use? My servers are with hetzner.com.
Hetzner as well!
-
Maybe. The first thing you can do is change the SSH port and restrict access to SSH login for all users on the system to trusted IP addresses. Change the passwords for absolutely all users. This does not solve the problem since CWP is compromised and requests can be executed as root from there, but somehow it ensures that the server is not used for botnets - DDoS, email spam, etc. As I said, the infection through CWP was long ago. Personally, I think one of my servers was infected minutes before 03.09.2021, 03:46:34, because the logs before that are missing, and it has been online since 2020. I also restored backups and the infection existed 2-3 years ago. Even if the server is cleaned, as long as the vulnerability in CWP exists, it is still under threat. Personally, I will wait for the CWP bug to be fixed and then reinstall the server with the new CWP panel.
thank you
but, can you please tell me which way I can be 100% sure that malware exists?
I'm asking this because many tess found on internet shows that my system is not infected.
Your test only shows that it is. And if I run it on other server (which is not connected to my original in any way), there too it shows positive
-
thank you
but, can you please tell me which way I can be 100% sure that malware exists?
I'm asking this because many tess found on internet shows that my system is not infected.
Your test only shows that it is. And if I run it on other server (which is not connected to my original in any way), there too it shows positive
Check if you have /usr/lib64/libkeystats.so file in your system. If you do you're infected. I would say it's safe to bet that the majority of CWP users are infected and don't know it.
As top20 said most likely the vulnerability with CWP is still open so cleaning out the server, re-installing the OS and then putting back CWP will probably just end up with the same issue until it's patched.
-
thank you
but, can you please tell me which way I can be 100% sure that malware exists?
I'm asking this because many tess found on internet shows that my system is not infected.
Your test only shows that it is. And if I run it on other server (which is not connected to my original in any way), there too it shows positive
Check if you have /usr/lib64/libkeystats.so file in your system. If you do you're infected. I would say it's safe to bet that the majority of CWP users are infected and don't know it.
As top20 said most likely the vulnerability with CWP is still open so cleaning out the server, re-installing the OS and then putting back CWP will probably just end up with the same issue until it's patched.
I don't have that file
-
Maybe. The first thing you can do is change the SSH port and restrict access to SSH login for all users on the system to trusted IP addresses. Change the passwords for absolutely all users. This does not solve the problem since CWP is compromised and requests can be executed as root from there, but somehow it ensures that the server is not used for botnets - DDoS, email spam, etc. As I said, the infection through CWP was long ago. Personally, I think one of my servers was infected minutes before 03.09.2021, 03:46:34, because the logs before that are missing, and it has been online since 2020. I also restored backups and the infection existed 2-3 years ago. Even if the server is cleaned, as long as the vulnerability in CWP exists, it is still under threat. Personally, I will wait for the CWP bug to be fixed and then reinstall the server with the new CWP panel.
thank you
but, can you please tell me which way I can be 100% sure that malware exists?
I'm asking this because many tess found on internet shows that my system is not infected.
Your test only shows that it is. And if I run it on other server (which is not connected to my original in any way), there too it shows positive
If my instructions indicate that you are infected, it is 99.99% certain, especially if both tests are positive. There has clearly been a mass infection through CWP. Which hosting provider are you using?
-
I don't have that file
Are you on Centos 8 or Almalinux? If so the file won't be there, it's only there on Centos 7. My Centos 8 and Almalinux servers were exploited also on the 19th with the same notice of ebury from my host, still trying to figure out exactly how. My server admin believes it's just a vulnerability in CWP and we have to wait for a fix. Once again maybe the update on the 20th patched something? Who knows.
-
But regarding ssh -G:
It should be noted that people using an OpenSSH version released after October 2014 will get a false positive with the ESET test, since there is now a legitimate -G switch in the SSH binary. See e.g. the SSH man page on OpenBSD.org or the Github mirror of the actual commit adding this switch. –
Daniel Andersson
Feb 14, 2016 at 19:42
From:
https://stackoverflow.com/questions/22526214/ssh-g-21-grep-e-illegal-e-unknown-dev-null-echo-system-clean
I am in Croatia, and have local hosting provider
-
I don't have that file
Are you on Centos 8 or Almalinux? If so the file won't be there, it's only there on Centos 7. My Centos 8 and Almalinux servers were exploited also on the 19th with the same notice of ebury from my host, still trying to figure out exactly how. My server admin believes it's just a vulnerability in CWP and we have to wait for a fix. Once again maybe the update on the 20th patched something? Who knows.
I'm on CentOS 7.9.2009
-
Here is some more info for Ebury 2
https://www.welivesecurity.com/2017/10/30/windigo-ebury-update-2/
-
check the server by the instructions just to be sure...but this looks like a false alert.
https://srvfail.com/check-clean-ebury-ssh-rootkit/
-
check the server by the instructions just to be sure...but this looks like a false alert.
https://srvfail.com/check-clean-ebury-ssh-rootkit/
yes, I've checked with tests in that link, and it seems that my server is not infected
-
Thanks for highlighting this! No Ebury Trojans here on any of my 3 CWP servers; just one case of Win.Trojan.Hide-1 under one WordPress install, which was promptly exorcised:
/home/account/public_html/wp-admin/zSROyV.php
Win.Trojan.Hide-1
-
You can quickly check if you are infected with Ebury by checking if the file /usr/lib64/libkeystats.so exists or by running the following command through the console - ssh -G 2>&1 | grep -e illegal -e unknown > /dev/null && echo "System clean" || echo "System infected"
Definitely, this command to check can get a false positive.
I have several servers, I'm checking these, and just one have the file '/usr/lib64/libkeystats.so', but all my servers are being pointed as "System infected" through this command.
The file 'libkeystats.so' can just be a legitimate file from the package 'keyutils-libs-1.5.8-3.el7.x86_64', if not infected.
In Centos 7, the check can be made through the following command:
rpm -qf /lib64/libkeyutils.so.1.5
Checking the server containing the file '/usr/lib64/libkeystats.so', with the instructions of the above security sites, it's pointing the file is not infected.
The packages using it can be listed by:
rpm -q --whatrequires keyutils-libs
Regards,
Netino
-
You can quickly check if you are infected with Ebury by checking if the file /usr/lib64/libkeystats.so exists or by running the following command through the console - ssh -G 2>&1 | grep -e illegal -e unknown > /dev/null && echo "System clean" || echo "System infected"
I'm sorry, but I don't think your information is accurate here -- on either count. The -G switch for SSH is now legitimate, and the existence of /usr/lib64/libkeystats.so does not prove an Ebury rootkit infection.
just did a complete scan. Took 108 minutes scanning almost half a million files. I dont have it
Have you confirmed that the malware security scanner accurately identifies Ebury? I did that scan but then a manual check -- and note that checking via SSH does not necessarily prove a clean bill of health. You need to check via a local console or else Ebury could alter the results shown via SSH.
-
Ok, there seems to be a possible false positive story going on here...
Step by step:
First:
ssh -G 2>&1 | grep -e illegal -e unknown > /dev/null && echo "System clean" || echo "System infected"This code's only purpose is to find out whether or not the command "ssh -G" returns the message "illegal option" in the first line. If it doesnt return this line, then it somehow means that the system is infected.
Second:
Searching for those files:
[root@pmail ~] ls -all /lib64/libkeyutils.so*
lrwxrwxrwx 1 root root 27 Jun 19 2021 /lib64/libkeyutils.so -> /usr/lib64/libkeyutils.so.1
lrwxrwxrwx. 1 root root 18 Jun 19 2021 /lib64/libkeyutils.so.1 -> libkeyutils.so.1.6
-rwxr-xr-x. 1 root root 16192 Jun 19 2021 /lib64/libkeyutils.so.1.6
Take a look at the date.
Next step is to check if those files are installed from a repo:
[root@pmail ~]# rpm -q --whatprovides /lib64/libkeyutils.so*
keyutils-libs-devel-1.5.10-9.el8.x86_64
keyutils-libs-1.5.10-9.el8.x86_64
keyutils-libs-1.5.10-9.el8.x86_64
Next step is to check when was this installed:
[root@pmail ~]# dnf history list keyutils-libs
ID | Command line | Date and time | Action(s) | Altered
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
1 | | 2021-12-26 14:06 | Install | 362 EETake a look at the date and what line number is this. It says line number 1. Which means these files are installed on the first ever install
[root@pmail ~]# dnf history info 1 | grep 'keyutils-libs'
Install keyutils-libs-1.5.10-9.el8.x86_64 @baseos
This install was BEFORE CWP was installed. So, this seems like some mass hysteria going on with these files. It's a false positive.
Third:
netstat -plan | grep atdIt returns a result, but nothing is using it.
Probably because I have all ports closed except those needed?
Now, does a result from this last command indicate that my system is infected?
Here is an example of another article saying that a result like this means you're infected:
unix 2 [ ACC ] STREAM LISTENING 103713 8119/atd @/tmp/dbus-ZP7tFO4xsL
The red part is what indicates infection. I don't have the red part on my output.
-
I am scaning now with clam, will clam clean it?
-
You should run all the scanners under the Security tab of CWP, as well as the specific checks mentioned in this thread.
-
https://arstechnica.com/security/2024/05/ssh-backdoor-has-infected-400000-linux-servers-over-15-years-and-keeps-on-spreading/ (https://arstechnica.com/security/2024/05/ssh-backdoor-has-infected-400000-linux-servers-over-15-years-and-keeps-on-spreading/)
This article about Ebury is a good read, and specifically mentions the weakness in CWP that Ebury exploited.
-
uf..great...
was that ever addresed by CWP team, or?
-
uf..great...
was that ever addresed by CWP team, or?
This most likely won't be something addressed by CWP, at least, the removal of trojan itself. I do hope CWP team has identified and fixed the exploit which allowed this trojan to be installed. Most likely Ebury was injected into CWP hosts via a CWP vulnerability over the years. Come to find out my system has Ebury installed, and most likely has been like that for years undetected.
Malicious DLLs were found in the following locations,
- /usr/lib64/libkeyutils.so.1.5
- /usr/lib64/libkeystats.so
With a (duplicated) running process of,
- /usr/lib/systemd/systemd-udevd
With an open UNIX socket at,