I am embarking on a new project that feels like it would revolve around a workflow engine. It has data coming in from various sources, going out to various destinations, and working through the system, taking on various states, linking to other data etc. On top of that will be admin pages to monitor what is going on inside the system.
I realise that is pretty vague, but the whole thing screams "workflow engine", and will likely need queues to handle the flow of data around the system. At the highest level, we are joining orders in e-commerce stores to suppliers who will be handling the product supply and shipping.
So, are there any recommendations on what I should watch here first, to get an idea of how to take this kind of approach in designing and developing the system? There is no mention of workflow engines or queue handling or scheduled jobs to push things around, so far as I can see. So either I'm searching for the wrong things, or Laracasts is perhaps more geared towards CRUD-oriented applications?
Any tips?
@consil Without knowing the specifics, it sounds like you want to look at a state machine implementation that tracks the workflow of entities, and also handles the transitions of one state to another.
It also seems like an event-driven approach would work well here, rather than the direct editing of “resources”. So you would raise events that modify the data, i.e. “product was dispatched”, “product was returned” etc.
Just a small example of a part of this system, to hopefully illustrate where I think my Laravel app is heading.
One part of the system will regularly visit shops and check for new orders having been placed. New orders are then scanned to see if they contain products that are relevant to my system. If there are any, then the order details are downloaded and pushed into queue to be processed further.
So this part looks like it needs a scheduler, and presumably command-line commands to run through this, and a queue, as well as storage so it can keep track of orders it has seen. Admin pages must see what it has looked at, and what has been passed on to the next stage, and where things have got stuck (failed to contact the shop, failed to pull order details, etc.). It may also be useful for Admin pages to be able to set scheduler timings and other options.
What I am having difficulty with, is a design framework to put all these requirements into, with reference to what Laravel and its popular third-party packages supports.
@martinbean events - that's a good point. So conceptually, something happens, such as a record being added to a model, and that fires a registered event. That event then, perhaps, adds a message to a queue to flag up that this record needs to be processed by something else (something that may involve contacting another site, hence the queue to detach/loosely couple those processes). All I am concerned about at the first instance, is getting that record into storage through Eloquent, and the cascade of other off-line things in relation to that record "just happens" as a result. Is that kind of how events would work?
I think there will be a number of state machines, each handling different areas and operating on different models. I don't know if there are Laracast tutorials on state machines, and the best way to implement them in Laravel, or whether I just put something together by hand in the appropriate controllers.
I suppose what I am looking for, is a good framework approach to build this system in, because it could turn out complex further down the road, and taking my own ad-hoc approach at this first stage is likely to make later additions a lot harder to manage.
@consil With an events-based approach, you would fire events, and the events would have handlers that actually manipulate the entities in your application. So an “order was shipped” event handler may update an Order model’s status to “shipped”.
So it’s kind of the inverse of the “traditional” CRUD approach: instead of your models raising events, you raise events whose corresponding handlers update your models based on what occurred in that event.
Thanks, I'll do some reading up on that. Video collection here:
@martinbean [edit: sorry, that should have been @consil ;)] Can you perhaps elaborate on the approach you took? I have a similar use case and had a quick look at https://github.com/lexik/LexikWorkflowBundle and https://github.com/yohang/Finite. Did you consider either of those?
Anyone else have experience and tips around this issue?
Please or to participate in this conversation.