Distributed work queues with Beanstalk

I have a rather large data import process developed using Beanstalk Queues and Lumen. In order to speed up the process, I was going to have a single BeanstalkD queue and have multiple devices running BeanstalkD workers to process the BeanstalkD queues. I have multiple devices talking to the Beanstalkd queue, but I am finding that when the second device attempts to process a job on the queue list, it has already been processed by the first device. I thought the purpose of beanstalkd was to allow the work to be distributed to multiple worker devices to offload the processing from a single device. Am I wrong on this?

vvvphpdev

9 years ago

Level 5

You could add InteractsWithQueue in your handler

and call

$this->job->getJobId();

The rest is up to you to handle those id's, something like checking if ID is already stored, if exists then ignore it, else do the job.

In your handler, let's use caching
- check if the id is already in the job by calling Cache::has(...) thing.
  - if exists, then check if the job is in the failing table.
- if the id not found, then call the Cache::put(...) thing and insert the job id.
  - then do the logic

Please or to participate in this conversation.