Be part of JetBrains PHPverse 2026 on June 9 – a free online event bringing PHP devs worldwide together.

MohammadAbusaleh's avatar

Forge + DigitalOcean + LetsEncrypt Blocking Crawlers

Good Day,

I'm using Forge + DigitalOcean with LetsEncrypt SSL and whenever I test crawling with ScreamingFrog I get connection time out, so I assume something is blocking crawling on the website. What I know:

  • I don't have a no robot meta tag.
  • The crawler crawls 4 pages only.
  • I have redirects from non-https to https, and from non-www to www
  • I added asset caching to nginx configuration and that's the only thing that I changed
  • It's the same on all my websites that use Forge + DO and LetsEncrypt.
  • It's not the case for websites without LetsEncrypt

EDIT: tested the website without the LetsEncrypt certificate and it crawled fine, with LetsEncrypt it blocked crawling

Thanks in advance

0 likes
0 replies

Please or to participate in this conversation.