Google indexing link to CWP

[Xadus]

Hi all,

I got something strange...  A couple of weeks ago i placed a new domain on my sever, and now Google has indexed it, but when i do a search in Google to that domain name, Google has found it (domain.com) but is only giving a link to a CWP login not the website itself:

https://cpanel.domain.com/login/

On all other sites i don't have this problem, and can't find what the issue causes.

Maybe someone knows how this can happen?

Thnx!

[overseer]

You can use robots.txt to control what gets indexed and what doesn't. You can also ask Google to delete admin areas of your site from their index. They are usually prompt about those kind of takedown requests.

[Xadus]

I suppose that the robot.txt has to be in the root of the website? i placed it there.

User-agent: Googlebot
Disallow: /cpanel.domain.com/

[overseer]

Sorry, but that's not correct syntax. You can't specify a domain in the robots.txt -- it just has to be at the site root for whatever domain you are wanting to control.

Code Select Expand

User-agent: Googlebot
Disallow: /

[Xadus]

Thank you Overseer,

But that means that Google is indexing nothing (the whole site)?

[overseer]

Perhaps look at the site root for CWP:

Code Select Expand

/usr/local/cwpsrv/htdocs/admin
/usr/local/cwpsrv/htdocs/admin/admin
And I would think you would want to disallow ALL indexing of those locations:

Code Select Expand

User-agent: *
Disallow: /

[Xadus]

I made that robots.txt

User-agent: *
Disallow: /

Tried to upload to the 2 locations, but on SFTP i get "permission denied", the CWP filemanager gives me a green upload, but i don't see the robots.txt as a file in the location

[overseer]

You may have to create the files on the server via a root shell using the nano text editor. Then you would also want to make them immutable, otherwise the next CWP update would likely delete them.

Code Select Expand

chattr +i /usr/local/cwpsrv/htdocs/admin/robots.txt
chattr +i /usr/local/cwpsrv/htdocs/admin/admin/robots.txt

[Xadus]

Hi Overseer,

Thnx again for your reply, that makes sense to me, i am not really a "server nerd" so your help is much appreciated.

I made the robots.txt file trough SSH in Nano, but when i want to save it i also get the reply "permission denied".

[overseer]

Sorry, I should've checked for you, but those CWP directories themselves are immutable already. So start by removing the immutable bit on the directories:

Code Select Expand

chattr -i /usr/local/cwpsrv/htdocs/admin/admin
chattr -i /usr/local/cwpsrv/htdocs/admin

Then create the robots.txt files:

Code Select Expand

nano /usr/local/cwpsrv/htdocs/admin/admin/robots.txt
nano /usr/local/cwpsrv/htdocs/admin/robots.txt

Set the immutable bit on the robots.txt files:

Code Select Expand

chattr +i /usr/local/cwpsrv/htdocs/admin/admin/robots.txt
chattr +i  /usr/local/cwpsrv/htdocs/admin/robots.txt

Finally, reset the immutable bit on the CWP directories:

Code Select Expand

chattr +i /usr/local/cwpsrv/htdocs/admin/admin
chattr +i /usr/local/cwpsrv/htdocs/admin

[Xadus]

Hi Overseer,

Thank you very much for the help, the last worked perfect!

[Xadus]

Above worked perfect, but now Google is indexing it again...

Someone has a new solution for this?

[overseer]

Did a CWP update blow away the robots.txt files?

[Xadus]

Nope, the robots are still there.

There are more sites on that server and it's only on 1 site (I don't want to expose this site in publice here, otherwise you can see it if you search the domain name on Google). some other sites are normally indexed by Google, others that where indexed are gone.

Google is a big mess the last time...

[Xadus]

Anyone...?