Laravel Redis Queues Across Servers

@lionheartblue The worker will have to execute on the queued jobs, so ideally, they should be on the same server. Think about it, even if you broke queueing out to a separate server, what difference would it make if the worker had to pull those tasks in anyway? It would just add a layer of complexity and processing that is unnecessary.

That is what I thought, but when using the AWS SQS service, this is broken out to another server, so I assumed any queue service could work this way.

However, I do agree that it makes no sense if it has to pull the tasks in.

Okay, I will make this change and hope it finally all works :)

Can it be a separate application on the same server? Or has to be within the same APP?

So everything I have read about this suggests I can do what I am now doing. Yet I cannot get it to work.

So I have Server One, which currently hosts a copy of the application and a second queue application running Horizon.

I have a second server Server Two which hosts my application and is Web facing.

I want Server Two to push jobs to the Redis queue on Server One...then the application on Server One will process the queue jobs.

I have amended my config/datbase.php on all servers and applications to point to Server One's IP. I have amended my config/queue.php on all servers and applications to use redis.

But I can only queue directly from the queue application hosting Horizon.

@lionheartblue I'm not sure. Thinking just a separate db connection aimed at the other server?

So the Queue application is a plan Laravel App just running Horizon and Telescope.

It's pointing to the same Mysql DB as my workers application and the web application.

Testing with php artisan tinker within the Queue application with a simple Queue::push($job, 'high'); returns an incremental integer. In this case 5.

Testing within either of the other applications, I seem to get a return value of a string, such as this: "1pUnbWu2gvn2XNVGpN25jgI5Smsqfk5E" which to me suggests it is not Queueing to the same place!

Oh, and also I have tested telnet to the Redis IP and port, and I can access it from all servers.

@lionheartblue I think you just need to specify the queue connection...

https://laravel.com/docs/6.x/queues#connections-vs-queues

We were both thinking of this as a database thing... but queues have their own connection config. Use that. Sorry, I'm barely starting to understand a lot of this stuff myself. Only set up queues in one app I use so far.

As far as I can tell, I have this all set up correctly:

The config/queue.php from my workers server:

'redis' => [
        'driver' => 'redis',
        'connection' => 'default',
        'queue' => env('REDIS_QUEUE', 'whatjobs.low'),
        'retry_after' => 90,
         'block_for' => null,
],

The config/database.php from my workers server. (So this server ALSO hosts the redis queues on a diff application, but the same server)

 'redis' => [
      'cluster' => false,
                'default' => [
                    'url' => env('REDIS_URL'),
                    'host' => env('REDIS_HOST', '127.0.0.1'),
                    'password' => env('REDIS_PASSWORD', null),
                    'port' => env('REDIS_PORT', 6379),
                    'database' => env('REDIS_DB', 0),
               ],
 ],

So assuming the above is correct, where is this application queueing its jobs? Because I don't get any errors when I Queue jobs, they just don't appear in the Queues.

@lionheartblue So far as my understanding, they would queue to Redis. When you say they don't appear in the queues... what do you mean by that? Where are you looking? Do you use Horizon?

Yes, I am using Horizon. And all my servers and applications are deployed using Forge.

So, when I tinker a Queue on the application running Horizon, I see a job hit the Queue. I can see it in Horizon.

If I do a tinker on my main application (on the same server) and push a job to the same Queue, it appears to complete without error, but does not show up on Horizon.

@lionheartblue Double-check your environments in config/horizon.php

Make sure that is referencing the proper connection and queue.

Then do php artisan horizon:terminate, and php artisan horizon to restart.

Also the docs suggest using supervisor to make sure Horizon and Queue workers stay alive. Not sure if you have that, but that exists in the docs in case these fail. Which they can.

So my config/horizon.php:

    'environments' => [
        'production' => [
            'supervisor-1' => [
                'connection' => 'redis',
                'queue' => ['high'],
                'balance' => 'auto',
                'minProcesses' => 1,
                'maxProcesses' => 20,
                'tries' => 3,
            ],
            'supervisor-2' => [
                'connection' => 'redis',
                'queue' => ['normal'],
                'balance' => 'auto',
                'minProcesses' => 1,
                'maxProcesses' => 10,
                'tries' => 3,
            ],
            'supervisor-3' => [
                'connection' => 'redis',
                'queue' => ['low'],
                'balance' => 'auto',
                'minProcesses' => 1,
                'maxProcesses' => 5,
                'tries' => 3,
            ],
        ],
...

@lionheartblue I think your 'queue => ['whatever'] should have default as a value somewhere.

    'environments' => [
        'production' => [
            'supervisor-1' => [
                'connection' => 'redis',
                'queue' => ['default', 'high'],  <---- something like this.
                'balance' => 'auto',
                'minProcesses' => 1,
                'maxProcesses' => 20,
                'tries' => 3,
            ],
        ],

I believe that setting specifies which queue to watch.

Ummm, okay. I thought that default was just the name of a queue, but all my queues are called high, normal and low.

@lionheartblue I might be off base here. I'm not super versed at this stuff either, I'm really just trying to help think through what the root issue could be. Hmmm...

This thread is what lead me to think that file might be off...

https://github.com/laravel/horizon/issues/244

It seems like most of the configs there have 'default' in that spot along side the queue name.

I apologize for not being very scientific here. Just pointing out a place to look.

¯_(ツ)_/¯

@fylzero I really appreciate your help on this. I will read the article and have a play. Thanks

So @fylzero, I have gone through and made changes suggested in the article you found. Sadly, this has not helped.

Although digging further into the logs, I found that Queue'd Jobs were still trying to send to the old AWS SQS Service I was using, despite this being removed from the .ENV file.

No clear command helped resolve this, so I ended up deleting everything and re-deploying from my repo. This resolved the SQS issue, but still no joy with my Redis Queue!

And it really makes no sense. No errors now in the logs. It reports that a job has been Queued, I get a returned string ID of the Queued Job, as mentioned before, which still looks like an SQS one, rather than one generated from Redis.

Maybe I am being rather ridiculous, but as a test, I have just installed another new Laravel Application and set up Queueing via this same Redis Server.

This works without issue. So, now I am wondering am I being stupid by assuming that my original Laravel 5.4 application should be able to push jobs to my new Laravel 6.2 Redis Queue Application?

Do they need to be the same version?

@lionheartblue Honestly, I'm not really good with cross-version Laravel questions... Laravel documents their upgrade paths so well and it is so easy to do that I always keep on the latest version.

@fylzero Yes I know, usually I do too, but I have inherited this application to maintain and it is so far behind, and using dependencies that won't work in newer versions. We are currently re-developing the main application but until then I have to get this stable.

So, I downgraded my Queue application to Laravel 5.4 and my new test application to 5.4. This then proved to me that 5.4 is part of my problem, as my new test app could no longer Queue jobs to this Redis server.

So, I upgraded them both, and my main application all to 5.5, which is the first version that supports Horizon.

My test application can queue jobs. Horizon shows the jobs, and the tinker app shows a return value of "5" which increments by 1 on each new Queue call.

But my main application still does NOT queue on this Redis Server. And the tinker app returns a unique string as mentioned before. This seems really odd to me.

Edit: Therefore my ONLY solution was to switch to Beanstalkd for the Queues. And dispose of Horizon.

@lionheartblue Yeah, unfortunately having the versions behind leaves you in somewhat uncharted territory. Keep in mind that you can usually work with package creators through Github to get them to update your dependancies for you. Usually. I've helped 3 or so developers get their packages over the v6 wall so I could update my apps. Most of the time they just aren't aware that they have a limitation set in their composer file or need to update the str helper package or something simple. Anyway, good luck!