kakallatt
1 month ago
152
2
Laravel

Spatie Crawler: Some links always return 503 Error on production

Posted 1 month ago by kakallatt

I run a crawler on my website and there are some links from Amazon that always return 503 code on production, but they return 200 in my local machine. Does anyone have any ideas on this issue? Is the problem from my server IP or something?

Some links:

https://www.amazon.com/dp/B00SZB0BXC
https://www.amazon.com/dp/B01BFKIJ06
https://www.amazon.com/dp/B005H8C4ZE

Here is the code of crawler

/**
 * @var array
 */
protected static array $clientOptions = [
    RequestOptions::COOKIES         => true,
    RequestOptions::CONNECT_TIMEOUT => 10,
    RequestOptions::TIMEOUT         => 10,
    RequestOptions::ALLOW_REDIRECTS => true,
    RequestOptions::HEADERS         => [
        'User-Agent' => '*',
    ],
];


Crawler::create(static::$clientOptions)
            ->setCrawlProfile(new CustomProfile($this->website))
            ->setCrawlObserver(new LinkCrawler($this->website))
            ->setParseableMimeTypes(['text/html', 'text/plain'])
            ->setMaximumResponseSize(1024 * 1024 * 3)
            ->startCrawling($this->website->url);

Please sign in or create an account to participate in this conversation.