Level 102
Maybe see if you can run several in parallel?
https://github.com/spatie/async
Or dispatching a job for each batch (maybe 100?)
2 likes
Sep 3, 2021
6
Level 2
Check Website Status
Hi All,
I have 1000k website URL's into my DB, Using curl Library I am checking each website status, e.g is website response is active or not. Curl works fine for every single record. I wrote down a crown job query the script will execute on everyday start. But when the script executes it will take too much to check 1000k website URLs and return me active or inactive status. I need to do this step very quickly is there any better method to do this, please.
Level 54
Just checking your numbers though ? Are we taking 1000k to be 1 million urls?
1 like
Level 2
Our number of URLs is huge and up to millions.
Level 54
So if you have a million records, and each curl takes 1 second you’ll be looking at 277 hours to check all your urls.
That’s a lot to do in 1 day.
Is this app for an uptime checker or looking for domains which have expired? If it’s either there maybe a better way to do it
1 like
Level 7
is there answer for this question?
Level 56
@eskiesirius bumping one year old threads usually is not the best way on getting answers.
Please open a new thread. And to be frank, if you actually read the responses on this thread you will get the answer for the question you made.
Please or to participate in this conversation.