Block search engines from crawling the entire Forge web server.

Ye olde robots.txt?

Thanks, but where do I put the robots.txt file?

inside the public folder (DUH!)

I think people don't get my question... I'm not trying to block a specific site in my Forge server. I want to block the ENTIRE FORGE SERVER this way I don't need to worry about putting a robots.txt per site in their public folder.

We really need to know what type of blocking do you need... at a network level or just website access?

If it's just website, you could do browser checks to make sure it's an actual browser and not a crawl bot. Or gather IPs for the bots and block them at a hardware (firewall) level. Can't know every search engine's IPs though...

Hi,

Without going too deep in this conversation (my knowledge is basic), I just need to block Google to craw the different sites I create in my Forge server, nothing else... so answering Bashy - "network level or just website access?" I guess just website access.

Thank you.

If it's just google you want to block then your robots.txt would be :

User-agent: Googlebot
Disallow: /

Yup, robots.txt will "tell" the User-agent to not crawl your site. The search engine doesn't have to abide by it though.