Control Web Panel

WebPanel => Information => Topic started by: blue-yu on March 22, 2023, 06:09:11 PM

Title: Ebury trojan on all of my CWP servers
Post by: blue-yu on March 22, 2023, 06:09:11 PM
Hello,

Today I was informed by our national domain registry that couple of my ip addresses are doing some malicious activities. Also they suspected that it is "Ebury Linux SSH rootkit/backdoor trojan".

And really after malware scan of all my servers (5 servers), everyone was having this trojan. Any hint on how can those trojans get in to the system, and how to prevent future problems?
Title: Re: Ebury trojan on all of my CWP servers
Post by: Rob P on March 22, 2023, 08:15:16 PM
Yep got a notification to that this was the case. Would love to know whats going on here.
Title: Re: Ebury trojan on all of my CWP servers
Post by: Rob P on March 22, 2023, 08:30:09 PM
What are you using for your malware scan? We got the same notice but our scans are coming up clean.
Title: Re: Ebury trojan on all of my CWP servers
Post by: blue-yu on March 23, 2023, 08:24:59 AM
I found trojan on all servers. I went to Security/Security Center/Malware Scan and then selected Custom scan from / (root) folder. It took a while but Trojan was found. Other things was also found in emails but it's normal :) Unix.Dropper.Ebury-9906999-0
 was found in usr/lib64/libkeystats.so. I have no idea how it got there. All of my server have different password generated at random.org (for example dr3Zd^VQnyy2q^w3). SSH port is not default.
Title: Re: Ebury trojan on all of my CWP servers
Post by: Rob P on March 23, 2023, 11:39:24 AM
My server admin said they saw rumblings about other CWP servers having this issue starting on March 17th -20th. Maybe the March 20th update was a patch?

Im running the scans now, will probably take a while. What OS were you running on the infected machines? One of mine was Centos 8 and the other was AlmaLinux 8.5. Also were you on a fully updated CWP?

Are you planning on re-building from scratch? From what I read thats the only definite solution.
Title: Re: Ebury trojan on all of my CWP servers
Post by: top20 on March 23, 2023, 02:29:49 PM
The same thing happened to me. 2 servers with CentOS 7 and CWP 7 were infected. I did a quick analysis and it turned out that the infection happened more than 11 months ago for sure. Another thing I found out is that the server was not infected 3 years ago. I learned this from the backups I have for these 2 servers. So, the infection did not happen on March 17-20... Keep this in mind.
Title: Re: Ebury trojan on all of my CWP servers
Post by: iraqiboy90 on March 23, 2023, 05:52:48 PM
just did a complete scan. Took 108 minutes scanning almost half a million files. I dont have it
Title: Re: Ebury trojan on all of my CWP servers
Post by: dp41646 on March 23, 2023, 06:34:42 PM
Ok, I'am infected too...
I am scaning now with clam, will clam clean it?

Thx!
Title: Re: Ebury trojan on all of my CWP servers
Post by: top20 on March 23, 2023, 07:01:35 PM
Once the system is compromised, it is unknown which backdoors are open.
Title: Re: Ebury trojan on all of my CWP servers
Post by: dp41646 on March 23, 2023, 07:03:04 PM
so only way to get rid it is clean os install?
Title: Re: Ebury trojan on all of my CWP servers
Post by: top20 on March 23, 2023, 07:38:37 PM
Maybe. The first thing you can do is change the SSH port and restrict access to SSH login for all users on the system to trusted IP addresses. Change the passwords for absolutely all users. This does not solve the problem since CWP is compromised and requests can be executed as root from there, but somehow it ensures that the server is not used for botnets - DDoS, email spam, etc. As I said, the infection through CWP was long ago. Personally, I think one of my servers was infected minutes before 03.09.2021, 03:46:34, because the logs before that are missing, and it has been online since 2020. I also restored backups and the infection existed 2-3 years ago. Even if the server is cleaned, as long as the vulnerability in CWP exists, it is still under threat. Personally, I will wait for the CWP bug to be fixed and then reinstall the server with the new CWP panel.
Title: Re: Ebury trojan on all of my CWP servers
Post by: top20 on March 23, 2023, 08:02:24 PM
By the way, which hosting provider do you use? My servers are with hetzner.com.
Title: Re: Ebury trojan on all of my CWP servers
Post by: Rob P on March 23, 2023, 08:02:58 PM
By the way, which hosting provider do you use? My servers are with hetzner.com.

Hetzner as well!
Title: Re: Ebury trojan on all of my CWP servers
Post by: dp41646 on March 23, 2023, 08:16:57 PM
Maybe. The first thing you can do is change the SSH port and restrict access to SSH login for all users on the system to trusted IP addresses. Change the passwords for absolutely all users. This does not solve the problem since CWP is compromised and requests can be executed as root from there, but somehow it ensures that the server is not used for botnets - DDoS, email spam, etc. As I said, the infection through CWP was long ago. Personally, I think one of my servers was infected minutes before 03.09.2021, 03:46:34, because the logs before that are missing, and it has been online since 2020. I also restored backups and the infection existed 2-3 years ago. Even if the server is cleaned, as long as the vulnerability in CWP exists, it is still under threat. Personally, I will wait for the CWP bug to be fixed and then reinstall the server with the new CWP panel.
thank you
but, can you please tell me which way I can be 100% sure that malware exists?
I'm asking this because many tess found on internet shows that my system is not infected.
Your test only shows that it is. And if I run it on other server (which is not connected to my original in any way), there too it shows positive
Title: Re: Ebury trojan on all of my CWP servers
Post by: Rob P on March 23, 2023, 08:25:20 PM
thank you
but, can you please tell me which way I can be 100% sure that malware exists?
I'm asking this because many tess found on internet shows that my system is not infected.
Your test only shows that it is. And if I run it on other server (which is not connected to my original in any way), there too it shows positive

Check if you have /usr/lib64/libkeystats.so file in your system. If you do you're infected. I would say it's safe to bet that the majority of CWP users are infected and don't know it.

As top20 said most likely the vulnerability with CWP is still open so cleaning out the server, re-installing the OS and then putting back CWP will probably just end up with the same issue until it's patched.

Title: Re: Ebury trojan on all of my CWP servers
Post by: dp41646 on March 23, 2023, 08:26:38 PM
thank you
but, can you please tell me which way I can be 100% sure that malware exists?
I'm asking this because many tess found on internet shows that my system is not infected.
Your test only shows that it is. And if I run it on other server (which is not connected to my original in any way), there too it shows positive

Check if you have /usr/lib64/libkeystats.so file in your system. If you do you're infected. I would say it's safe to bet that the majority of CWP users are infected and don't know it.

As top20 said most likely the vulnerability with CWP is still open so cleaning out the server, re-installing the OS and then putting back CWP will probably just end up with the same issue until it's patched.

I don't have that file
Title: Re: Ebury trojan on all of my CWP servers
Post by: top20 on March 23, 2023, 08:32:33 PM
Maybe. The first thing you can do is change the SSH port and restrict access to SSH login for all users on the system to trusted IP addresses. Change the passwords for absolutely all users. This does not solve the problem since CWP is compromised and requests can be executed as root from there, but somehow it ensures that the server is not used for botnets - DDoS, email spam, etc. As I said, the infection through CWP was long ago. Personally, I think one of my servers was infected minutes before 03.09.2021, 03:46:34, because the logs before that are missing, and it has been online since 2020. I also restored backups and the infection existed 2-3 years ago. Even if the server is cleaned, as long as the vulnerability in CWP exists, it is still under threat. Personally, I will wait for the CWP bug to be fixed and then reinstall the server with the new CWP panel.
thank you
but, can you please tell me which way I can be 100% sure that malware exists?
I'm asking this because many tess found on internet shows that my system is not infected.
Your test only shows that it is. And if I run it on other server (which is not connected to my original in any way), there too it shows positive

If my instructions indicate that you are infected, it is 99.99% certain, especially if both tests are positive. There has clearly been a mass infection through CWP. Which hosting provider are you using?
Title: Re: Ebury trojan on all of my CWP servers
Post by: Rob P on March 23, 2023, 08:33:50 PM
I don't have that file

Are you on Centos 8 or Almalinux? If so the file won't be there, it's only there on Centos 7. My Centos 8 and Almalinux servers were exploited also on the 19th with the same notice of ebury from my host, still trying to figure out exactly how. My server admin believes it's just a vulnerability in CWP and we have to wait for a fix. Once again maybe the update on the 20th patched something? Who knows.
Title: Re: Ebury trojan on all of my CWP servers
Post by: dp41646 on March 23, 2023, 08:34:30 PM
But regarding ssh -G:
Code: [Select]
It should be noted that people using an OpenSSH version released after October 2014 will get a false positive with the ESET test, since there is now a legitimate -G switch in the SSH binary. See e.g. the SSH man page on OpenBSD.org or the Github mirror of the actual commit adding this switch. –
Daniel Andersson
 Feb 14, 2016 at 19:42

From:
https://stackoverflow.com/questions/22526214/ssh-g-21-grep-e-illegal-e-unknown-dev-null-echo-system-clean

I am in Croatia, and have local hosting provider
Title: Re: Ebury trojan on all of my CWP servers
Post by: dp41646 on March 23, 2023, 08:35:20 PM
I don't have that file

Are you on Centos 8 or Almalinux? If so the file won't be there, it's only there on Centos 7. My Centos 8 and Almalinux servers were exploited also on the 19th with the same notice of ebury from my host, still trying to figure out exactly how. My server admin believes it's just a vulnerability in CWP and we have to wait for a fix. Once again maybe the update on the 20th patched something? Who knows.

I'm on CentOS 7.9.2009
Title: Re: Ebury trojan on all of my CWP servers
Post by: dp41646 on March 23, 2023, 09:09:48 PM
Here is some more info for Ebury 2
https://www.welivesecurity.com/2017/10/30/windigo-ebury-update-2/
Title: Re: Ebury trojan on all of my CWP servers
Post by: studio4host on March 23, 2023, 09:26:24 PM
check the server by the instructions just to be sure...but this looks like a false alert.
https://srvfail.com/check-clean-ebury-ssh-rootkit/
Title: Re: Ebury trojan on all of my CWP servers
Post by: dp41646 on March 23, 2023, 09:44:42 PM
check the server by the instructions just to be sure...but this looks like a false alert.
https://srvfail.com/check-clean-ebury-ssh-rootkit/

yes, I've checked with tests in that link, and it seems that my server is not infected
Title: Re: Ebury trojan on all of my CWP servers
Post by: overseer on March 24, 2023, 03:02:39 AM
Thanks for highlighting this! No Ebury Trojans here on any of my 3 CWP servers; just one case of Win.Trojan.Hide-1 under one WordPress install, which was promptly exorcised:
Quote
/home/account/public_html/wp-admin/zSROyV.php
Win.Trojan.Hide-1
Title: Re: Ebury trojan on all of my CWP servers
Post by: Netino on March 25, 2023, 03:05:25 AM
You can quickly check if you are infected with Ebury by checking if the file /usr/lib64/libkeystats.so exists or by running the following command through the console -
Code: [Select]
ssh -G 2>&1 | grep -e illegal -e unknown > /dev/null && echo "System clean" || echo "System infected"

Definitely, this command to check can get a false positive.
I have several servers, I'm checking these, and just one have the file '/usr/lib64/libkeystats.so', but all my servers are being pointed as "System infected" through this command.

The file 'libkeystats.so' can just be a legitimate file from the package 'keyutils-libs-1.5.8-3.el7.x86_64', if not infected.
In Centos 7, the check can be made through the following command:
Code: [Select]
rpm -qf /lib64/libkeyutils.so.1.5
Checking the server containing the file '/usr/lib64/libkeystats.so', with the instructions of the above security sites, it's pointing the file is not infected.

The packages using it can be listed by:
Code: [Select]
rpm -q --whatrequires keyutils-libs
Regards,
Netino
Title: Re: Ebury trojan on all of my CWP servers
Post by: overseer on March 25, 2023, 09:18:28 AM
You can quickly check if you are infected with Ebury by checking if the file /usr/lib64/libkeystats.so exists or by running the following command through the console -
Code: [Select]
ssh -G 2>&1 | grep -e illegal -e unknown > /dev/null && echo "System clean" || echo "System infected"
I'm sorry, but I don't think your information is accurate here -- on either count. The -G switch for SSH is now legitimate, and the existence of /usr/lib64/libkeystats.so does not prove an Ebury rootkit infection.
just did a complete scan. Took 108 minutes scanning almost half a million files. I dont have it
Have you confirmed that the malware security scanner accurately identifies Ebury? I did that scan but then a manual check -- and note that checking via SSH does not necessarily prove a clean bill of health. You need to check via a local console or else Ebury could alter the results shown via SSH.
Title: Re: Ebury trojan on all of my CWP servers
Post by: iraqiboy90 on April 09, 2023, 07:30:39 PM
Ok, there seems to be a possible false positive story going on here...

Step by step:
First:
Code: [Select]
ssh -G 2>&1 | grep -e illegal -e unknown > /dev/null && echo "System clean" || echo "System infected"This code's only purpose is to find out whether or not the command "ssh -G" returns the message "illegal option" in the first line. If it doesnt return this line, then it somehow means that the system is infected.

Second:
Searching for those files:
Code: [Select]
[root@pmail ~] ls -all /lib64/libkeyutils.so*
lrwxrwxrwx  1 root root    27 Jun 19  2021 /lib64/libkeyutils.so -> /usr/lib64/libkeyutils.so.1
lrwxrwxrwx. 1 root root    18 Jun 19  2021 /lib64/libkeyutils.so.1 -> libkeyutils.so.1.6
-rwxr-xr-x. 1 root root 16192 Jun 19  2021 /lib64/libkeyutils.so.1.6
Take a look at the date.

Next step is to check if those files are installed from a repo:
Code: [Select]
[root@pmail ~]# rpm -q --whatprovides /lib64/libkeyutils.so*
keyutils-libs-devel-1.5.10-9.el8.x86_64
keyutils-libs-1.5.10-9.el8.x86_64
keyutils-libs-1.5.10-9.el8.x86_64

Next step is to check when was this installed:
Code: [Select]
[root@pmail ~]# dnf history list keyutils-libs
ID     | Command line                                                                                                                                                                           | Date and time    | Action(s)      | Altered
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
     1 |                                                                                                                                                                                        | 2021-12-26 14:06 | Install        |  362 EE
Take a look at the date and what line number is this. It says line number 1. Which means these files are installed on the first ever install
Code: [Select]
[root@pmail ~]# dnf history info 1 | grep 'keyutils-libs'
    Install keyutils-libs-1.5.10-9.el8.x86_64                         @baseos
This install was BEFORE CWP was installed. So, this seems like some mass hysteria going on with these files. It's a false positive.

Third:
Code: [Select]
netstat -plan | grep atdIt returns a result, but nothing is using it.

Probably because I have all ports closed except those needed?
Now, does a result from this last command indicate that my system is infected?

Here is an example of another article saying that a result like this means you're infected:
Quote
unix 2 [ ACC ] STREAM LISTENING 103713 8119/atd @/tmp/dbus-ZP7tFO4xsL
The red part is what indicates infection. I don't have the red part on my output.
Title: Re: Ebury trojan on all of my CWP servers
Post by: atulgodzs on February 12, 2024, 12:20:43 PM
I am scaning now with clam, will clam clean it?
Title: Re: Ebury trojan on all of my CWP servers
Post by: overseer on February 12, 2024, 12:57:45 PM
You should run all the scanners under the Security tab of CWP, as well as the specific checks mentioned in this thread.