Google indexing link to CWP

Hi all,

I got something strange...  A couple of weeks ago i placed a new domain on my sever, and now Google has indexed it, but when i do a search in Google to that domain name, Google has found it (domain.com) but is only giving a link to a CWP login not the website itself:

https://cpanel.domain.com/login/

On all other sites i don't have this problem, and can't find what the issue causes.

Maybe someone knows how this can happen?

Thnx!

You can use robots.txt to control what gets indexed and what doesn't. You can also ask Google to delete admin areas of your site from their index. They are usually prompt about those kind of takedown requests.

I suppose that the robot.txt has to be in the root of the website? i placed it there.

User-agent: Googlebot
Disallow: /cpanel.domain.com/

Sorry, but that's not correct syntax. You can't specify a domain in the robots.txt -- it just has to be at the site root for whatever domain you are wanting to control.

Code: [Select]

User-agent: Googlebot
Disallow: /

Thank you Overseer,

But that means that Google is indexing nothing (the whole site)?

Perhaps look at the site root for CWP:

Code: [Select]

/usr/local/cwpsrv/htdocs/admin
/usr/local/cwpsrv/htdocs/admin/adminAnd I would think you would want to disallow ALL indexing of those locations:

Code: [Select]

User-agent: *
Disallow: /

I made that robots.txt

User-agent: *
Disallow: /

Tried to upload to the 2 locations, but on SFTP i get "permission denied", the CWP filemanager gives me a green upload, but i don't see the robots.txt as a file in the location

You may have to create the files on the server via a root shell using the nano text editor. Then you would also want to make them immutable, otherwise the next CWP update would likely delete them.

Code: [Select]

chattr +i /usr/local/cwpsrv/htdocs/admin/robots.txt
chattr +i /usr/local/cwpsrv/htdocs/admin/admin/robots.txt

Hi Overseer,

Thnx again for your reply, that makes sense to me, i am not really a "server nerd" so your help is much appreciated.

I made the robots.txt file trough SSH in Nano, but when i want to save it i also get the reply "permission denied".

Sorry, I should've checked for you, but those CWP directories themselves are immutable already. So start by removing the immutable bit on the directories:

Code: [Select]

chattr -i /usr/local/cwpsrv/htdocs/admin/admin
chattr -i /usr/local/cwpsrv/htdocs/admin
Then create the robots.txt files:

Code: [Select]

nano /usr/local/cwpsrv/htdocs/admin/admin/robots.txt
nano /usr/local/cwpsrv/htdocs/admin/robots.txt
Set the immutable bit on the robots.txt files:

Code: [Select]

chattr +i /usr/local/cwpsrv/htdocs/admin/admin/robots.txt
chattr +i  /usr/local/cwpsrv/htdocs/admin/robots.txt
Finally, reset the immutable bit on the CWP directories:

Code: [Select]

chattr +i /usr/local/cwpsrv/htdocs/admin/admin
chattr +i /usr/local/cwpsrv/htdocs/admin

Hi Overseer,

Thank you very much for the help, the last worked perfect!

Above worked perfect, but now Google is indexing it again...

Someone has a new solution for this?

Did a CWP update blow away the robots.txt files?

Nope, the robots are still there.

There are more sites on that server and it's only on 1 site (I don't want to expose this site in publice here, otherwise you can see it if you search the domain name on Google). some other sites are normally indexed by Google, others that where indexed are gone.

Google is a big mess the last time...

Anyone...?