Control Web Panel
Other => Other => Topic started by: episani on June 04, 2019, 03:51:39 AM
-
Hi,
Is there a way to prevent search engine crawlers such as BING to kill my server?
It has happened twice this week.
It took me a while to figure out what was going on, but when BING crawls one on my sites, the load average of the server goes to the roof. It is the same as having a denial of service attack as it crawls hundreds of pages at the same time.
Then the server goes back to normal but the bind service doesn't recover and remains down.
I have to connect to the server and start bind (named) manually.
I will install monit in the server to solve this problem but the crawler error will persist.
Any ideas in how to deal with it?
Thanks in advance.
-
they are killing websevers also, many issues recently because of that, maybe block ip ranges with csf
check connections
sh /scripts/net_show_connections
then block ip ranges, this ip is only example:
csf -d 54.104.23.0/24
-
The best way to check number of spiders/crawlers to your server is this command:
cat /usr/local/apache/domlogs/*.log | awk -F\" '{print $6}' | sort | uniq -c | sort -n
or
cat /usr/local/apache/domlogs/specific-domain.log | awk -F\" '{print $6}' | sort | uniq -c | sort -n
Also you can firstly clean all of your logs and get "new/fresh" result:
Beware, this command truncate all logs to empty:
find /usr/local/apache/domlogs/ -type f -name "*.log" -exec truncate --size 0 {} \;
With this command you can check which logs will be deleted first:
find /usr/local/apache/domlogs/ -type f -name "*.log" -exec ls -if 0 {} \;