Handling queues in Azure with Lumen micro services
Just as a pretext, the microservices approach is fairly new to me, so there are some assumptions in what I state.
Okay, I have a number of Lumen applications that handle incoming requests. Those requests involve some quick validation checks and then a fast response to the requester. The request details are then dispatched to a queue for further processing.
The standard way for Laravel to handle queues, is for the process wanting to dispatch a job, to choose the job, give it some data (normally a model instance) then dispatch it. Laravel then has queue listeners to listen to those queues, open the message, read the metadata to determine the job to run, look up the model instance data, then run the job. That's kind of fine - it all runs from the command line continuously listening, and can handle some large jobs.
Now, my queues are Azure queues, and my listener would be a microservice. So two problems here:
- Firstly Azure queue connections will time out very quickly if you just sit there listening, and the listeners need to be regularly restarted to be a fully robust system.
- Secondly, a (at least one) listener process needs to be running constantly, doing nothing most of the time. That's not very cloud-friendly.
So my though was that Laravel could dispatch jobs as it normally does, then Azure would monitor the event of an entry being added and send it to the appropriate service end point. So Azure does the listening and fires off the job through HTTP.
So my questions:
- Firstly, does that approach make any sense whatsoever?
- Would I continue to dispatch the jobs in the normal way? Would Azure being able to send the full payload to the Lumen service endpoint, and would the endpoint be able to unwrap the metadata and data to make sense of what needs to be run? Is that kind of built into Laravel?
- Or should I perhaps just throw the model data (name and ID would probably be enough) onto the queues with no reference to what job is needed, and just let the endpoint that Azure chooses (by configuration) send that must simpler data structure?
We have another monolithic system running successfully in a VM in Azure, so we know it can talk to Azure resources (queues, filesystems, databases etc.) but we now need an approach that is a little less tightly bound, more robust, and a lot faster.
I am, of course, trying all these things out, but any observations from experience in these things would be most welcome! Maybe Azure simply does not do this kind of thing, and I still need an application-level queue listener to keep watching the queues (the Azure docs seem to imply that polling is needed, which I guess is okay, since once the data gets into the queues, it is no longer super-time critical, but it does seem like it could be wastefully eating up processing).
Please or to participate in this conversation.