Push Jobs from NodeJs backend

Hi, I'm experimenting a way to push a Job into a redis Queue to be consumed by a Laravel backend using Horizon to run workers.

I navigated into the Laravel Queue system code to understand the structure of the payload that a job should be to be recognized and executed by the workers.

This is the Pyaload that should be sent to Redis:

let id = uuidv4();

let payload = {
        uuid: id,
        id: id,
        attempts: 0,
        displayName: 'App\Jobs\Converter',
        job: 'Illuminate\Queue\CallQueuedHandler@call',
        data: {
            commandName: 'App\Jobs\Converter',
            command: Serialize.serialize(
                new Converter(params),
                {
                    'App\Jobs\Converter': Converter
                }
            )
        }
    }

// Push he job in the Laravel queue.
        redis.rpush('queues:default', JSON.stringify(payload), (err, replay) => {
            console.log("Queued");
        });

This simple process works, but pushing the job from NodeJs nothing is showed in "Pending Jobs" and "Completed Jobs" sections in the Horizon dashboard.

The Converter job is correctly gathered from the queue and executed, but it isn't reported as "Pending" and "Completed".

I tried to debug the internal code of the Laravel Queue System, but I have not found any difference in the payload from a job posted from Node or from Laravel itself.

What kind of steps are required to allow Horizon to identify the Job as "Pending" first and "Completed" later?

rodrigo.pedra

5 years ago

Level 56

The data.command contents on the payload should be serialized using PHP serialization. I am guessing you are aware of that and have this custom Serialize.serialize() call doing that.

Also be sure you are connecting to the same redis database as the Laravel app. Not just the same redis server, but the same redis database. By default a Laravel app uses the 0 database for queues, but it might be configured to use another one, check if the REDIS_DB env variable is set to another database.

Also Laravel prefixes keys on redis to avoid collision when several apps are running on the same server, check your Laravel app's ./config/database.php under redis.options.prefix to figure what prefix it is using.

One last thing: Wouldn't be easier to set an endpoint you can call from node and have that endpoint dispatch the job within Laravel itself?

Hope any of these helps.

Valeri0

5 years ago

Level 4

Hi Rodrigo, thank you for your contribution.

Yes I use "php-serialize" npm module to create a valid PHP serilized object from Node.

I'm trying to avoid Laravel to treat the http requests that schedule this job. This endpoint receives more than 5 million requests per day, and node show me a 10x better performance to do this simple task as http handler.

As I mentioned above it's quite strange because the job is correctly executed in the Laravel side so technical configuration should be right. Node script and Laravel are communicating right through redis.

The issue is that the jobs scheduled from Nodejs aren't showed in the Horizon "Completed Jobs" and "Pending Jobs" views.

If instead I schedule the job using the Controller inside Laravel, they appears in the Horizon views.

Horizon allow to navigate the workload in a simple way, It could be really helpful to allow the jobs to be correctly tracked.

I thought it could be really useful for others.

When the job is pushed outside of Laravel, Horizon does not track them as completed.

There are some additional steps to do in redis to allow Horizon to mark the jobs as completed?

rodrigo.pedra

5 years ago

Level 56

Now I get what you meant.

Horizon extends the Redis queue shipped with Laravel and dispatches custom events when a job is pushed:

https://github.com/laravel/horizon/blob/f69fd50e204d46b1464daca0f47397f148952887/src/RedisQueue.php#L49-L66

So I guess you will have to dispatch that event somehow within your Laravel application so it can track events from outside.

Maybe instead of calling and endpoint you can push a Redis job from your node app that its sole purpose is to dispatch the real job from Laravel itself, so at least you get Horizon working on the heavy duty job.

You can have these jobs on different queues so the one coming from node job is processed faster.

Please or to participate in this conversation.