I'm not sure where to post this, and not quite sure what to do about it.. The bottom line is that Laravel takes data from the queues and handles it as though those queue messages are valid, correctly structured, contain job objects that can be unserialized without errors, and basically does a poor job of handling any kind of invalid data that may end up on the queue.
I am using Azure queues, and occasionally Azure will put its own message onto the queue, or it looks that way. I suspect it is the Microsoft library that acts like a message has been put onto the queue when connections time out, but the end result is that the MS Azure Queue Storage library tells Laravel, "here is a message". That message contains some timeout error details and a bit of XML. Laravel cannot cope with that, and ends up throwing all sorts of exceptions, instead of just failing the job and throwing it onto the failed heap.
I had a job put onto the queue for a job class that does not exist (it was another Laravel instance that put it onto the wrong queue). Again, Laravel tries unserializing that job payload, which throws an exception that is not handled in any sensible way. The job exits, but does not call its failed() method, so the queue worker does not know it has ended, the queue message remains locked in Azure for the half an hour that Azure will allow a process to lock it, then the message gets dumped back onto the queue by Azure and the process starts again - an error every half an hour, no completed jobs in that time, no failed jobs registered either, and messages checked out of the Azure queue and so not even visible through other means for debugging.
Then I thought, what if I just throw "xxx" onto the queue. Surely that will be handled correctly? On no, it throws Laravel into an endless loop, trying to process the "xxx" job about 50 times a second, failing each time, and retrying immediately. Again, it is not handled as a failed job, and so no retry count is kept.
So, IMO Laravel queue handling needs some serious fixes here. It is just the points at which it assumes the job details it has been given by an outside source (Azure, memory-based queues, Aamazon SQS etc.) are correctly formatted, valid, safe, and complete. Just one byte out and a production system can go into an overdrive endless loop, or just lose jobs completely.
Is this a common problem? Are there any issues that anyone here is aware of that covers these types of problems that I can add to? This is out-of-the-box Laravel 5.6 behaviour, and it's leaving me a bit nervous. When queues work, they are great, but when something goes wrong, they seem to be able to bring the system down.