Control Web Panel
WebPanel => CentOS 7 Problems => Topic started by: Xadus on September 13, 2023, 10:07:32 AM
-
Hi all,
I got something strange... A couple of weeks ago i placed a new domain on my sever, and now Google has indexed it, but when i do a search in Google to that domain name, Google has found it (domain.com) but is only giving a link to a CWP login not the website itself:
https://cpanel.domain.com/login/
On all other sites i don't have this problem, and can't find what the issue causes.
Maybe someone knows how this can happen?
Thnx!
-
You can use robots.txt to control what gets indexed and what doesn't. You can also ask Google to delete admin areas of your site from their index. They are usually prompt about those kind of takedown requests.
-
I suppose that the robot.txt has to be in the root of the website? i placed it there.
User-agent: Googlebot
Disallow: /cpanel.domain.com/
-
Sorry, but that's not correct syntax. You can't specify a domain in the robots.txt -- it just has to be at the site root for whatever domain you are wanting to control.
User-agent: Googlebot
Disallow: /
-
Thank you Overseer,
But that means that Google is indexing nothing (the whole site)?
-
Perhaps look at the site root for CWP:
/usr/local/cwpsrv/htdocs/admin
/usr/local/cwpsrv/htdocs/admin/adminAnd I would think you would want to disallow ALL indexing of those locations:
User-agent: *
Disallow: /
-
I made that robots.txt
User-agent: *
Disallow: /
Tried to upload to the 2 locations, but on SFTP i get "permission denied", the CWP filemanager gives me a green upload, but i don't see the robots.txt as a file in the location
-
You may have to create the files on the server via a root shell using the nano text editor. Then you would also want to make them immutable, otherwise the next CWP update would likely delete them.
chattr +i /usr/local/cwpsrv/htdocs/admin/robots.txt
chattr +i /usr/local/cwpsrv/htdocs/admin/admin/robots.txt
-
Hi Overseer,
Thnx again for your reply, that makes sense to me, i am not really a "server nerd" so your help is much appreciated.
I made the robots.txt file trough SSH in Nano, but when i want to save it i also get the reply "permission denied".
(https://i.imgur.com/wOZq3UY.jpg)
-
Sorry, I should've checked for you, but those CWP directories themselves are immutable already. So start by removing the immutable bit on the directories:
chattr -i /usr/local/cwpsrv/htdocs/admin/admin
chattr -i /usr/local/cwpsrv/htdocs/admin
Then create the robots.txt files:
nano /usr/local/cwpsrv/htdocs/admin/admin/robots.txt
nano /usr/local/cwpsrv/htdocs/admin/robots.txt
Set the immutable bit on the robots.txt files:
chattr +i /usr/local/cwpsrv/htdocs/admin/admin/robots.txt
chattr +i /usr/local/cwpsrv/htdocs/admin/robots.txt
Finally, reset the immutable bit on the CWP directories:
chattr +i /usr/local/cwpsrv/htdocs/admin/admin
chattr +i /usr/local/cwpsrv/htdocs/admin
-
Hi Overseer,
Thank you very much for the help, the last worked perfect!
-
Above worked perfect, but now Google is indexing it again...
Someone has a new solution for this?
-
Did a CWP update blow away the robots.txt files?
-
Nope, the robots are still there.
There are more sites on that server and it's only on 1 site (I don't want to expose this site in publice here, otherwise you can see it if you search the domain name on Google). some other sites are normally indexed by Google, others that where indexed are gone.
Google is a big mess the last time...
-
Anyone...?
-
Do you have appropriate robots.txt in all your site roots? I know in the long ago past, I had a private site that got indexed by Google, even though nothing links to it. My suspicion was that someone who used the site used the Google toolbar to browse to it. I asked Google to take down the listing and put up a proper robots.txt file and they complied right away. Nowadays it's probably Chrome phoning home to the Big G, so your secrets are hardly safe anymore.
-
On that particulair site i have a standard robots.txt
I have no clue how to prevent Google indexing "https://cpanel.domain.com/login/" in the robots.txt from the root of the website.
-
Do you actually use the "cpanel" service record? That's one I don't support for my servers; I require my clients to append :2083, which is familiar to many people coming from cPanel servers
-
Do you actually use the "cpanel" service record? That's one I don't support for my servers; I require my clients to append :2083, which is familiar to many people coming from cPanel servers
I have no clients on my server, i use him only for myself.
Further i found something else, when i search the domain name on Bing or DuckDuckGo then i don't get the links to the controlpanel, just the normal link to the website. Could it have something to do with Google?
By the way, there are now 2 domains with the same problem...
-
Bump....