GET request to external resources returns forbidden on server, but not locally

I've built a small scraper for a client, which (legitimately!) scrapes information from another website. The strange thing is that when I test it from the local server, everything is fine, and the website returns the body, but as soon as I deploy it to the server (Laravel Vapor), I'm getting a 403 Forbidden code as a response. What could trigger this behaviour? Is the IP address of the Vapor instance the cause, or is there a setting that can be updated?

martinbean

4 years ago

Level 80

Is the IP address of the Vapor instance the cause

Possibly. Vapour runs on Lambda which will execute within AWS’s cloud infrastructure, which will most likely have a static IP address range. So if the website being scraped has blocked traffic from AWS’s network, then that will give you 403 errors.

You’ll need to test that this is actually the case. Maybe spin up an EC2 instance and try doing to a cURL request from inside the EC2 instance and see if you get a 403 there as well.

1 like

Please or to participate in this conversation.