Level 54
You may need to ask google to remove it from their index if it’s already been included.
Test with https://support.google.com/webmasters/answer/6062598?hl=en
Nov 15, 2021
5
Level 10
Robots.txt file not blocking indexing
Hi All, I have a Laravel project live on the web but I want to block search engines from grabbing it. I read online to do this you create a robots.txt file with
User-agent: *
Disallow:
and put this in your folder. I put a copy of this file in my app root folder and a copy in the public folder. However, when I check on the google search console it says URL is on Google. Am I doing this wrong for a Laravel project?
Level 51
Level 10
@webrobert I tried your change and when I tested it I got this https://en.ryte.com/free-tools/robots-txt/?refresh=1&url=https%3A%2F%2Fwww.secretchickens.com%2F&useragent=Googlebot&submit=Evaluate
Level 51
@chrisgrim great it worked.
Crawling https://www.secretchickens.com/ with Googlebot NOT allowed!
Level 122
public folder only. Your project root should be inaccessible.
1 like
Please or to participate in this conversation.