Making sure job completes
What do you guys think would be the best way to make sure a single PHP job completes. My current use-case will need to do the following:
- Connect a server based on details for an account
- Check if there are any "updates" (new files)
- If there are new files -> download them, parse it and save them to specific location (file storage + the metadata in database).
As there could be thousands of accounts running this job multiple times (4) a day for each account it is pretty much critical that this job runs stable. Currently I'm using the database queue driver to put in jobs for each account and afterwards running a few workers.
Though I'm not sure how I can make sure my job gets processed and any possible error gets handled (think of disasters like power of server drops, internet connection got disabled, ...). What would be the best way to survive these things?
Please or to participate in this conversation.