Feb 20, 2018
0
Level 4
Forge + DigitalOcean + LetsEncrypt Blocking Crawlers
Good Day,
I'm using Forge + DigitalOcean with LetsEncrypt SSL and whenever I test crawling with ScreamingFrog I get connection time out, so I assume something is blocking crawling on the website. What I know:
- I don't have a no robot meta tag.
- The crawler crawls 4 pages only.
- I have redirects from non-https to https, and from non-www to www
- I added asset caching to nginx configuration and that's the only thing that I changed
- It's the same on all my websites that use Forge + DO and LetsEncrypt.
- It's not the case for websites without LetsEncrypt
EDIT: tested the website without the LetsEncrypt certificate and it crawled fine, with LetsEncrypt it blocked crawling
Thanks in advance
Please or to participate in this conversation.