I run a crawler on my website and there are some links from Amazon that always return 503 code on production, but they return 200 in my local machine. Does anyone have any ideas on this issue? Is the problem from my server IP or something?
https://www.amazon.com/dp/B00SZB0BXC https://www.amazon.com/dp/B01BFKIJ06 https://www.amazon.com/dp/B005H8C4ZE
Here is the code of crawler
/** * @var array */ protected static array $clientOptions = [ RequestOptions::COOKIES => true, RequestOptions::CONNECT_TIMEOUT => 10, RequestOptions::TIMEOUT => 10, RequestOptions::ALLOW_REDIRECTS => true, RequestOptions::HEADERS => [ 'User-Agent' => '*', ], ]; Crawler::create(static::$clientOptions) ->setCrawlProfile(new CustomProfile($this->website)) ->setCrawlObserver(new LinkCrawler($this->website)) ->setParseableMimeTypes(['text/html', 'text/plain']) ->setMaximumResponseSize(1024 * 1024 * 3) ->startCrawling($this->website->url);