error 403 Forbidden

Hi,
I have researched this forum under the above and found more than a dozen forum posts about it.

Not one of them offers any kind of solution and not one of them has ever been marked as resolved. This leads me to think that every one of these people has dumped this panel and gone somewhere else due to not solution. Its time that stopped and a working list of possible methods of resolving this issue is documented here!

So with the above in mind FFS dont answer if you havent got a reasonable suggestion on how to problem solve this.

For those who come back with the usual "check the logs"...

1. the logs i have checked are empty of anything related to the above error
2. I am running nginx, which logs should i search for errors relating to the above?

None of the websites on a brand new server work...al of them throw this exact same error.

I have disabled and uninstalled mod security,
I have disabled all firewalls
I have changed web servers from nginx to apache...doesnt do anything.
I have hard reset the VPS, no change.
selinux is disabled

There is something in a central location that is independant of nginx and apache that is doing this...but i for the life of me cannot figure out what it is?

I am not sure what else i can do?

Hi, seems like you forgot to say what exactly problem xD
Logs should be checked in /usr/local/apache/domlogs/

The question Subject is the question Igor 

logs you suggested i check are as follows for two of the domains where this isnt working (only have two on the server)

Please note...all i am typing into the web browser is http://mydomain.com.au (where "mydomain" is the name of the registered domain name)

Also, all port scans run from mxtoolbox seem to be ok.

first domain error log

2019/04/19 11:11:53 [error] 19078#19078: *36 openat() "/home/adamj/public_html/robots.txt" failed (2: No such file or directory), client: 66.249.75.17, server: mydomain.com.au, request: "GET /robots.txt HTTP/1.1", host: "mydomain.com.au"

There is nothing else in the above log!

second domain error log

2019/04/19 05:41:00 [error] 25424#25424: *1606 openat() "/home/mydomain2/public_html/manual/fr/mod/mod_proxy_express.html" failed (2: No such file or directory), client: 216.244.66.239, server: mydomain2.com.au, request: "GET /manual/fr/mod/mod_proxy_express.html HTTP/1.1", host: "mydomain2.com.au"
2019/04/19 05:41:02 [error] 25424#25424: *1607 openat() "/home/mydomain2/public_html/manual/zh-cn/rewrite/intro.html" failed (2: No such file or directory), client: 216.244.66.239, server: mydomain2.com.au, request: "GET /manual/zh-cn/rewrite/intro.html HTTP/1.1", host: "mydomain2.com.au"
2019/04/19 05:44:29 [error] 25424#25424: *1611 openat() "/home/mydomain2/public_html/manual/fr/mod/mod_log_forensic.html" failed (2: No such file or directory), client: 216.244.66.239, server: mydomain2.com.au, request: "GET /manual/fr/mod/mod_log_forensic.html HTTP/1.1", host: "mydomain2.com.au"
2019/04/19 05:46:35 [error] 25424#25424: *1613 openat() "/home/mydomain2/public_html/manual/es/programs/apxs.html" failed (2: No such file or directory), client: 180.76.15.149, server: mydomain2.com.au, request: "GET /manual/es/programs/apxs.html HTTP/1.1", host: "mydomain2.com.au"
2019/04/19 05:49:06 [error] 25424#25424: *1616 openat() "/home/mydomain2/public_html/robots.txt" failed (2: No such file or directory), client: 216.244.66.239, server: mydomain2.com.au, request: "GET /robots.txt HTTP/1.1", host: "mydomain2.com.au"
2019/04/19 06:24:25 [error] 25424#25424: *1638 openat() "/home/mydomain2/public_html/manual/zh-cn/mod/mod_cgid.html" failed (2: No such file or directory), client: 216.244.66.239, server: mydomain2.com.au, request: "GET /manual/zh-cn/mod/mod_cgid.html HTTP/1.1", host: "mydomain2.com.au"
2019/04/19 06:45:51 [error] 25424#25424: *1663 openat() "/home/mydomain2/public_html/robots.txt" failed (2: No such file or directory), client: 216.244.66.239, server: mydomain2.com.au, request: "GET /robots.txt HTTP/1.1", host: "mydomain2.com.au"
2019/04/19 06:48:44 [error] 25424#25424: *1671 openat() "/home/mydomain2/public_html/robots.txt" failed (2: No such file or directory), client: 144.76.137.254, server: mydomain2.com.au, request: "GET /robots.txt HTTP/1.1", host: "mydomain2.com.au"
2019/04/19 06:48:46 [error] 25424#25424: *1671 openat() "/home/mydomain2/public_html/manual/ko/programs/dbmmanage.html" failed (2: No such file or directory), client: 144.76.137.254, server: mydomain2.com.au, request: "GET /manual/ko/programs/dbmmanage.html HTTP/1.1", host: "mydomain2.com.au"
2019/04/19 06:48:48 [error] 25424#25424: *1671 openat() "/home/mydomain2/public_html/manual/pt-br/programs/split-logfile.html" failed (2: No such file or directory), client: 144.76.137.254, server: mydomain2.com.au, request: "GET /manual/pt-br/programs/split-logfile.html HTTP/1.1", host: "mydomain2.com.au"
2019/04/19 07:15:45 [error] 25424#25424: *1693 openat() "/home/mydomain2/public_html/manual/es/programs/logresolve.html" failed (2: No such file or directory), client: 180.76.15.14, server: mydomain2.com.au, request: "GET /manual/es/programs/logresolve.html HTTP/1.1", host: "mydomain2.com.au"
2019/04/19 07:22:15 [error] 25424#25424: *1698 openat() "/home/mydomain2/public_html/manual/zh-cn/mod/mod_authn_socache.html" failed (2: No such file or directory), client: 216.244.66.239, server: mydomain2.com.au, request: "GET /manual/zh-cn/mod/mod_authn_socache.html HTTP/1.1", host: "mydomain2.com.au"
2019/04/19 07:33:20 [error] 25424#25424: *1707 openat() "/home/mydomain2/public_html/robots.txt" failed (2: No such file or directory), client: 216.244.66.239, server: mydomain2.com.au, request: "GET /robots.txt HTTP/1.1", host: "mydomain2.com.au"
2019/04/19 07:49:05 [error] 25424#25424: *1717 openat() "/home/mydomain2/public_html/manual/es/misc/perf-tuning.html" failed (2: No such file or directory), client: 180.76.15.135, server: mydomain2.com.au, request: "GET /manual/es/misc/perf-tuning.html HTTP/1.1", host: "mydomain2.com.au"
2019/04/19 08:20:50 [error] 25424#25424: *1738 openat() "/home/mydomain2/public_html/robots.txt" failed (2: No such file or directory), client: 216.244.66.239, server: mydomain2.com.au, request: "GET /robots.txt HTTP/1.1", host: "mydomain2.com.au"
2019/04/19 09:08:13 [error] 25424#25424: *1812 openat() "/home/mydomain2/public_html/robots.txt" failed (2: No such file or directory), client: 216.244.66.239, server: mydomain2.com.au, request: "GET /robots.txt HTTP/1.1", host: "mydomain2.com.au"
2019/04/19 09:41:46 [error] 25424#25424: *1839 directory index of "/home/mydomain2/public_html/" is forbidden, client: 149.202.86.127, server: mydomain2.com.au, request: "GET / HTTP/1.1", host: "mydomain2.com.au"
2019/04/19 09:46:26 [error] 25424#25424: *1847 openat() "/home/mydomain2/public_html/manual/da/mod/event.html" failed (2: No such file or directory), client: 216.244.66.239, server: mydomain2.com.au, request: "GET /manual/da/mod/event.html HTTP/1.1", host: "mydomain2.com.au"
2019/04/19 09:56:17 [error] 30217#30217: *2 openat() "/home/mydomain2/public_html/robots.txt" failed (2: No such file or directory), client: 216.244.66.239, server: mydomain2.com.au, request: "GET /robots.txt HTTP/1.1", host: "mydomain2.com.au"
2019/04/19 10:40:24 [error] 19078#19078: *11 openat() "/home/mydomain2/public_html/manual/da/mod/mod_substitute.html" failed (2: No such file or directory), client: 216.244.66.239, server: mydomain2.com.au, request: "GET /manual/da/mod/mod_substitute.html HTTP/1.1", host: "mydomain2.com.au"
2019/04/19 10:43:21 [error] 19078#19078: *14 openat() "/home/mydomain2/public_html/robots.txt" failed (2: No such file or directory), client: 216.244.66.239, server: mydomain2.com.au, request: "GET /robots.txt HTTP/1.1", host: "mydomain2.com.au"
2019/04/19 11:04:46 [error] 19078#19078: *28 openat() "/home/mydomain2/public_html/robots.txt" failed (2: No such file or directory), client: 37.9.87.223, server: mydomain2.com.au, request: "GET /robots.txt HTTP/1.1", host: "mydomain2.com.au"
2019/04/19 11:04:50 [error] 19078#19078: *28 openat() "/home/mydomain2/public_html/manual/en/faq/index.html" failed (2: No such file or directory), client: 37.9.87.223, server: mydomain2.com.au, request: "GET /manual/en/faq/index.html HTTP/1.1", host: "mydomain2.com.au"
2019/04/19 11:07:36 [error] 19078#19078: *31 openat() "/home/mydomain2/public_html/manual/en/mod/mod_proxy.html" failed (2: No such file or directory), client: 180.76.15.22, server: mydomain2.com.au, request: "GET /manual/en/mod/mod_proxy.html HTTP/1.1", host: "www.mydomain2.com.au"
2019/04/19 11:30:55 [error] 19078#19078: *63 openat() "/home/mydomain2/public_html/robots.txt" failed (2: No such file or directory), client: 216.244.66.239, server: mydomain2.com.au, request: "GET /robots.txt HTTP/1.1", host: "mydomain2.com.au"

can i ask...should public_html for all domains be owned as follows?

Owner = domain user
Group = nobody

i dont normally use centos, my other server is debian, however i dont recall it having group=nobody

Error 403 is an webserver level error, just make sense to search in webserver logs.
Might be you looking at the wrong logs.
It may be that a wrong configuration can send requests to other areas, thus logging into different file logs.

If in apache virtual hosts are not being logged events, the first bet is to search in main server error log, at '/usr/local/apache/logs/error_log'.

As you are mentioning to use nginx, you must to check if the events are being logged in nginx logs.
You must check where in your configuration, normally in '/etc/nginx/nginx.conf', and the error log in '/var/log/nginx/error.log'.
Might be too your configuration is not logging events in the severity is wish.
Check the directive "error_log" in the configuration file.

Regards,
Netino

As an update...

I now find the following:
desktop pc google chrome - 403 error (even after clearing browser cache and cookies for all time)
same desktop pc Microsoft Edge and firefox - both of these two broswers work without any problems displaying the webpage
mobile device google chrome or samsung phone browser - both work no problems
Imac desktop pc - website works no problems
microsoft surface tablet IE and Chrome - works no problems

All of the above devices are using the same static ip address to access the internet.

So for some reason, this issue is restricted only to the desktop pc google chrome browser that i use to manage centos web panel and clearing cookies and browser cache makes no difference.

Chrome redirect automatically from 'http' to 'https'.
So, your site *must* be responsive too in 'https'.
This is the case..??

I have removed and reinstalled google chrome on this pc, makes no difference. The websites on CWP still are throwing 403 errors just for google chrome on just this computer!

hmm i found this...

[Wed Apr 17 05:13:30.659622 2019] [:error] [pid 1439:tid 140368417879808] [client 120.146.145.157:63170] [client 120.146.145.157] ModSecurity: Access denied with code 403 (phase 2). Pattern match "(?i:(?:\\\\b(??:s(?:ys\\\\.(?:user_(??:t(?:ab(?:_column|le)|rigger)|object|view)s|c(?:onstraints|atalog))|all_tables|tab)|elect\\\\b.{0,40}\\\\b(?:substring|users?|ascii))|m(?:sys(??:queri|ac)e|relationship|column|object)s|ysql\\\\.(db|user))|c(?:onstraint ..." at ARGS_NAMES:SELECT * FROM mysql.users. [file "/usr/local/apache/modsecurity-owasp-old/base_rules/modsecurity_crs_41_sql_injection_attacks.conf"] [line "116"] [id "950007"] [rev "2"] [msg "Blind SQL Injection Attack"] [data "Matched Data: SELECT * FROM mysql.users found within ARGS_NAMES:SELECT * FROM mysql.users: SELECT * FROM mysql.users"] [severity "CRITICAL"] [ver "OWASP_CRS/2.2.9"] [maturity "9"] [accuracy "8"] [tag "OWASP_CRS/WEB_ATTACK/SQL_INJECTION"] [tag "WASCTC/WASC-19"] [tag "OWASP_TOP_10/A1"] [tag "OWASP_AppSensor/CIE1"] [tag "PCI/6.5.2"] [hostname "149.28.179.232"] [uri "/index.php"] [unique_id "XLa1@iKjtZH4edW30V0rDQAAAMI"]

So clearly one line of investigation into 403 forbidden error is to check your home/office up address in mod security to ensure it has not been blacklisted (mod security has a tab where you can perform IP check).

In my case, it is not specifically the IP address that has been banned, because I have other systems on this IP that are working no problem...it is the combination of IP, desktop PC, and chrome browser!

So I am not sure exactly how mod security has distinguished one browser and computer from others on my network, however clearly it does...and in fact even goes so far as to block a specific browser on a specific computer amongst a group of machines all on the same network.

Perhaps it obtains internal hostname or IP address on my office network?

In any case, I now know the problem and need to remove this blacklisting.

The key to my finding this was doing an ipaddress lookup for my office network static IP address. I suggest this is a far better option than searching through logs.

As it turned out, the logs I should have been checking were the mod_security ones (not nginx or apache). And that explains why my apache/nginx logs had nothing about this error!

I think this should serve as a learning experience for this forum and hopefully provide at least one avenue of investigation to this largely unanswered question on this forum!

First port of call...check mod_security logs to ensure your home/office ip hasn't been blacklisted!