Author Topic: Google indexing link to CWP  (Read 1241 times)

0 Members and 1 Guest are viewing this topic.

Offline
*
Google indexing link to CWP
« on: September 13, 2023, 10:07:32 AM »
Hi all,

I got something strange...  A couple of weeks ago i placed a new domain on my sever, and now Google has indexed it, but when i do a search in Google to that domain name, Google has found it (domain.com) but is only giving a link to a CWP login not the website itself:

https://cpanel.domain.com/login/

On all other sites i don't have this problem, and can't find what the issue causes.

Maybe someone knows how this can happen?

Thnx!

Offline
****
Re: Google indexing link to CWP
« Reply #1 on: September 13, 2023, 02:35:07 PM »
You can use robots.txt to control what gets indexed and what doesn't. You can also ask Google to delete admin areas of your site from their index. They are usually prompt about those kind of takedown requests.

Offline
*
Re: Google indexing link to CWP
« Reply #2 on: September 13, 2023, 06:32:46 PM »
I suppose that the robot.txt has to be in the root of the website? i placed it there.

User-agent: Googlebot
Disallow: /cpanel.domain.com/

Offline
****
Re: Google indexing link to CWP
« Reply #3 on: September 14, 2023, 02:25:35 AM »
Sorry, but that's not correct syntax. You can't specify a domain in the robots.txt -- it just has to be at the site root for whatever domain you are wanting to control.
Code: [Select]
User-agent: Googlebot
Disallow: /

Offline
*
Re: Google indexing link to CWP
« Reply #4 on: September 14, 2023, 08:17:20 PM »
Thank you Overseer,

But that means that Google is indexing nothing (the whole site)?

Offline
****
Re: Google indexing link to CWP
« Reply #5 on: September 14, 2023, 08:33:29 PM »
Perhaps look at the site root for CWP:
Code: [Select]
/usr/local/cwpsrv/htdocs/admin
/usr/local/cwpsrv/htdocs/admin/admin
And I would think you would want to disallow ALL indexing of those locations:
Code: [Select]
User-agent: *
Disallow: /

Offline
*
Re: Google indexing link to CWP
« Reply #6 on: September 15, 2023, 08:19:53 PM »
I made that robots.txt

User-agent: *
Disallow: /

Tried to upload to the 2 locations, but on SFTP i get "permission denied", the CWP filemanager gives me a green upload, but i don't see the robots.txt as a file in the location

Offline
****
Re: Google indexing link to CWP
« Reply #7 on: September 15, 2023, 10:20:46 PM »
You may have to create the files on the server via a root shell using the nano text editor. Then you would also want to make them immutable, otherwise the next CWP update would likely delete them.

Code: [Select]
chattr +i /usr/local/cwpsrv/htdocs/admin/robots.txt
chattr +i /usr/local/cwpsrv/htdocs/admin/admin/robots.txt

Offline
*
Re: Google indexing link to CWP
« Reply #8 on: September 16, 2023, 11:09:10 AM »
Hi Overseer,

Thnx again for your reply, that makes sense to me, i am not really a "server nerd" so your help is much appreciated.

I made the robots.txt file trough SSH in Nano, but when i want to save it i also get the reply "permission denied".


Offline
****
Re: Google indexing link to CWP
« Reply #9 on: September 16, 2023, 06:41:31 PM »
Sorry, I should've checked for you, but those CWP directories themselves are immutable already. So start by removing the immutable bit on the directories:
Code: [Select]
chattr -i /usr/local/cwpsrv/htdocs/admin/admin
chattr -i /usr/local/cwpsrv/htdocs/admin

Then create the robots.txt files:
Code: [Select]
nano /usr/local/cwpsrv/htdocs/admin/admin/robots.txt
nano /usr/local/cwpsrv/htdocs/admin/robots.txt

Set the immutable bit on the robots.txt files:
Code: [Select]
chattr +i /usr/local/cwpsrv/htdocs/admin/admin/robots.txt
chattr +i  /usr/local/cwpsrv/htdocs/admin/robots.txt

Finally, reset the immutable bit on the CWP directories:
Code: [Select]
chattr +i /usr/local/cwpsrv/htdocs/admin/admin
chattr +i /usr/local/cwpsrv/htdocs/admin

Offline
*
Re: Google indexing link to CWP
« Reply #10 on: September 18, 2023, 11:14:21 AM »
Hi Overseer,

Thank you very much for the help, the last worked perfect!