How to add *.domain.com to CSF Firewall

I need to add *.ahrefs.net to CSF Firewall but all I can find in the CWP > CSF Firewall area is to add an ip address. How can I allow all bots coming from */ahreafs.net to the allow using the CWP UI.

You can add FQDN hostnames to /etc/csf/csf.dyndns
Then restart the firewall.

Best way to stop that damn crawler and other like it is via ModSecurity.

I use a custom rule that block out that site, along wit 124 other like it.

If you are running ModSecurity along with the OWASP Ruleset on the directory structure I posted, send me a PM with your email, and I can send you the .conf & .txt file to put in the /after/ directory.

You can add FQDN hostnames to /etc/csf/csf.dyndns
Then restart the firewall.

@overseer
I thought that csf.dyndns whitelisted domain names?

Kinda like how csf.ignore will ignore IP's and not send any notifications.

He said "allow" in the original post, so I gave him that answer -- didn't seem that he wanted to block that domain.

I would take you up on your script offer if you want to PM me the d/l links.

To make ahrefbot unable to access your server you need to block the IPs of the hosts used by the ahrefbot. You can obtain the IPs here:
https://help.ahrefs.com/en/articles/78658-what-is-the-list-of-your-ip-ranges
Just put the IPs into:
/etc/csf/csf.deny
and then restart CSF.

Please pay attention the IPs (networks) used by the bot can be changed from time to time. So it is better to obtain and update the IPs in CSF using the script from the following link:
https://api.ahrefs.com/v3/public/crawler-ip-ranges
and then restart csf.

Here is a simple solution to block access to the server from the networks used by AhrefBot:

1.Open the file:
/etc/csf/csf.deny
2. Put the following line into the file:

Code: [Select]

Include /etc/csf/csf.ahrefbot.deny3. Create and open the file /etc/cron.daily/ahrefbot_blocker using your favorite text editor (vi,ee,nano, etc).
4. Put the following content into the file:

Code: [Select]

#!/bin/bash

wget -O /tmp/crawler-ip-ranges https://api.ahrefs.com/v3/public/crawler-ip-ranges > /dev/null 2>&1
cat /tmp/crawler-ip-ranges |grep "/" | awk -F\" '{print $4}' > /tmp/ahrefbot.ips
diff  /etc/csf/csf.ahrefbot.deny /tmp/ahrefbot.ips > /dev/null 2>&1
error=$?
if [ $error -ne 0 ]
then
   echo "files are different or some file absents"
   cat /tmp/ahrefbot.ips > /etc/csf/csf.ahrefbot.deny
   rm -f /tmp/ahrefbot.ips /tmp/crawler-ip-ranges
   csf -r
fi

5. Make the file executable:

Code: [Select]

chmod 755 /etc/cron.daily/ahrefbot_blocker
What does it do ?

The script /etc/cron.daily/ahrefbot_blocker will be executed by the server every day. The script downloads the list of IPs used by the ahrefbot and compares it with the list of networks/ips in the file /etc/csf/csf.ahrefbot.deny. If the files have some difference then content of /tmp/ahrefbot.ips is copited into /etc/csf/csf.ahrefbot.deny and csf is reloaded. If no difference then nothing happens.

@cyberspace

Do you have an way to do the same for csf.allow?

Include /etc/csf/csf.hetrixtools.allow?

HetrixTool has kinda a messy way at: https://docs.hetrixtools.com/how-to-whitelist-our-uptime-monitoring-nodes-in-csf/

So they do have a updated IP list at https://hetrixtools.com/resources/uptime-monitor-ips.txt

Just minor changes:

1.Open the file:
/etc/csf/csf.allow
2. Put the following line into the file:

Code: [Select]

Include /etc/csf/csf.hetrixtools.allow3. Create and open the file /etc/cron.daily/hetrixtools_whitelist using your favorite text editor (vi,ee,nano, etc).
4. Put the following content into the file:

Code: [Select]

#!/bin/bash

wget -O /tmp/uptime-monitor-ips.txt https://hetrixtools.com/resources/uptime-monitor-ips.txt > /dev/null 2>&1
if [ $? -ne 0 ]
then
        echo "Can't download the file"
        exit 1
fi
cat /tmp/uptime-monitor-ips.txt | awk '{print $2 ,"#", $1}' > /tmp/hetrixtools.ips
diff  /etc/csf/csf.hetrixtools.allow /tmp/hetrixtools.ips > /dev/null 2>&1
error=$?
if [ $error -ne 0 ]
then
   echo "files are different or some file absents"
   cat /tmp/hetrixtools.ips > /etc/csf/csf.hetrixtools.allow
   rm -f /tmp/hetrixtools.ips /tmp/uptime-monitor-ips.txt
   csf -r
fi
5. Make the file executable:

Code: [Select]

chmod 755 /etc/cron.daily/hetrixtools_whitelist
The idea is the same. Additionally the script checks the exit code of "wget" to avoid strange behavior if the file can't be downloaded.

Thanks, will give it a try