ZetecVan's avatar

Queues - Redis/Horizon vs Beanstalkd

My project requires me to parse an xml file, and store the data in a couple of mysql tables. While developing, I've been using Beanstalkd to fire a job to store each xml 'record' and then once all the xml has been parsed, I've read that table, firing off a job to update other records/tables based on that record.

It runs quite smoothly, although can take a while because of the number of records.

Tonight I thought I'd give Horizon and Redis a go. I switched it out and triggered the job to parse. That went through quicker than Beanstalkd. Then it automatically triggered the rest of the processing. It processed the first record, and then just sat there for about five minutes. I was checking everything to see why when the next job kicked in.

So instead of each job taking just a second, they're taking a random time from a second to a minute or so. All the job is doing is reading the parsed record and updating a couple of existing records or creating a couple of new ones.

Has anyone else experienced this sort of behaviour?

0 likes
3 replies
fintable's avatar

3 years later: Yes, I've been having the same problem with Horizon. I have no visibility into why pending jobs are not picked up by Horizon workers. They're just sitting there in the pending queue. Gonna try beanstalkd and see what happens.

r0ny's avatar

@fintable

Were you able to find a solution? This still seems like an issue. Pending jobs are not getting picked up with horizon randomly.

Please or to participate in this conversation.