ZetecVan
1 year ago

Queues - Redis/Horizon vs Beanstalkd

Posted 1 year ago by ZetecVan

My project requires me to parse an xml file, and store the data in a couple of mysql tables. While developing, I've been using Beanstalkd to fire a job to store each xml 'record' and then once all the xml has been parsed, I've read that table, firing off a job to update other records/tables based on that record.

It runs quite smoothly, although can take a while because of the number of records.

Tonight I thought I'd give Horizon and Redis a go. I switched it out and triggered the job to parse. That went through quicker than Beanstalkd. Then it automatically triggered the rest of the processing. It processed the first record, and then just sat there for about five minutes. I was checking everything to see why when the next job kicked in.

So instead of each job taking just a second, they're taking a random time from a second to a minute or so. All the job is doing is reading the parsed record and updating a couple of existing records or creating a couple of new ones.

Has anyone else experienced this sort of behaviour?

Please sign in or create an account to participate in this conversation.