Application blocks when queue worker is running

Hello

PHP 8.0.7, Laravel 8, Servers on Laravel Forge

I have 2 servers: backend and worker

My problem is that when I'm running a queue worker on worker server my backend server freezes for few seconds every time I'm dispatching more jobs in queue.

Generally this are very small jobs, using for dispatching broadcasts event and code of jobs generally should not be a problem.

I have made 250$/month servers (for redis mysql and both servers) on digital ocean to be sure not hitting any limits.

I'm creating chat app.

When I'm sending messages very fast for backend server like 20 messages in 3 seconds (manually by keyboard), I'm getting response time, like for first 10 requests about 100ms, and next is coming around 5second or more.

Disabling queue worker on worker server resolves problem and all responses are very fast.

I was also using horizion instead of normal worker even with 20 processes but results was more or less the same.

I was using redis and database as QUEUE_CONNECTION with same results.

Here is my config:

       'database' => [
            'driver' => 'database',
            'table' => 'jobs',
            'queue' => 'default',
            'retry_after' => 90,
        ],


        'redis' => [
            'driver' => 'redis',
            'connection' => 'default',
            'queue' => env('REDIS_QUEUE', 'default'),
            'retry_after' => 90,
            'block_for' => 0,
        ],

Totaly not understand what's going on. Am I missing some configuration?

Poulos221

4 years ago

Level 1

It's prone to deadlock because all the threads in the pool may die before the thing you put in the queue is visible. Mitigate this by setting a reasonable keep alive time.

The task is not wrapped the way your Executor may expect. Lots of executor implementations wrap their tasks in some sort of tracking object before execution. Look at the source of yours.

Adding via getQueue() is strongly discouraged by the API, and may be prohibited at some point. A almost-always-better strategy is to install ThreadPoolExecutor .CallerRunsPolicy which will throttle your app by running the task on the thread which is calling execute().

However, sometimes a blocking strategy, with all its inherent risks, is really what you want. I'd say under these conditions

You only have one thread calling execute() You have to (or want to) have a very small queue length You absolutely need to limit the number of threads running this work (usually for external reasons), and a caller-runs strategy would break that.

Your tasks are of unpredictable size, so caller-runs could introduce starvation if the pool was momentarily busy with 4 short tasks and your one thread calling execute got stuck with a big one. So, as I say. It's rarely needed and can be dangerous, but there you go.

https://www.myccpay.bid/

1 like

piotrzatorski

4 years ago

Level 2

Really thanks for answer.

Problem was caused by Digitalocean Redis database.

When i migrated from Digitalocean (is something misconfigured there?) to Forge Redis, now everything work super fast :)

Please or to participate in this conversation.