Be part of JetBrains PHPverse 2026 on June 9 – a free online event bringing PHP devs worldwide together.

nulele's avatar

Best strategy for multi-database scheduled tasks (jobs)

Hello,

I have developed a multi-database laravel 9 project where every client has its own database. I use two connections:

  1. static connection to a "base" database called "base"
  2. dynamic connection to clients databases called "multi"

I read every client's database connection data from a table in the "base" database.

Everything works fine but I still haven't found a way to run scheduled task (jobs actually) separately for every client, so that client A can run Job A and B, client B can run Job A and C, client C can run Job D...

I'd like a solution where every client's database has it own jobs/failed_jobs tables and without the hassle of creating hard-coded config/queue.php or supervisor configuration for every new client.

Maybe I need something to dynamically create a queue for every client in a Service Provider but despite reading laravel's docs, tons of stackoverflow, laracasts and other forums articles, didn't find a way to undestand how to do that.

Any suggestion? Thanks

0 likes
6 replies
nulele's avatar

Well, I reached a compromise between what I wanted and what I am able to do.

Now I have moved jobs and failed_jobs in the "base" database. So my queue connection now points to "base" as well. Also the failed_jobs configuration is set on "base" connection (see below).

I created one supervisor configuration for each client. Important here is the --queue option where I set the client code, for instance:

command=php /path/to/my/app/artisan queue:work database --queue=team123 --tries=2 --sleep=3

In Console/Kernel.php I set the queue name for every job:

$schedule->job(new TestJob(), 'team123')
  ->dailyAt('09:00');

I then created a job middleware to dynamically initialize the right client's database connection based on the client code as above (team123).

    public function handle($job, $next)
    {
        // get client's connection data from "base" table connections
        // $job->queue contains the value from the supervisor configuration
        $connection = Connection::query()
            ->where('code', $job->queue)
            ->first();

        Config::set('database.connections.multi.database', $connection->dbname);
        Config::set("database.connections.multi.username", $connection->username);
        Config::set("database.connections.multi.password", $connection->password);

        DB::purge('multi');
        DB::reconnect('multi');

        return $next($job);
    }

Everything works!!! buuuuuut... if I have errors in a job, Laravel search for failed_jobs table in the client's database and not in the "base" database.

As said I already set the failed_jobs configuration to "base" connection in config/queue.php:

'failed' => [
        'driver' =>  'database',
        'database' => 'base',
        'table' => 'failed_jobs',
    ],

Any idea how to force failed_jobs to be searched on the "base" connection?

Thanks

nulele's avatar
nulele
OP
Best Answer
Level 1

Me stupid, it works!!! The problem was a couple of typos in this piece of config/queue.php file. The final version is:

'failed' => [
        'driver' =>  'database-uuids',
        'database' => 'base',
        'table' => 'failed_jobs',
    ],
Tray2's avatar

I'm sorry but I don't see the need for a database per client, if they are using the same application, they should use the same database. Your approach will give you a shitload of headaches further on.

I suggest rethinking the design and just use one database, and make the distinction with a client_id in the necessary tables.

If you would have been using Oracle databases, you could have used database links so it would seem that you were in the same database all along, but investing in an oracle database migth not be the best thing.

1 like
nulele's avatar

@Tray2 thanks but that's not the point of my question. The multi-database approach, it's a specification of the project; it preserves privacy and consistency of data; it favors database operations performance as customers with a lot of data do not slow down those with few... what I have created so far, works very well excepts for this little failed_jobs problem :)

Tray2's avatar

@nulele I know that, I just wanted to give my two cents on the design. As for privacy, consitency and performance.

  • Privacy is handled with the client_id preferably in a query scope.
  • Consistency isn't a problem with the proper database model.
  • Performance can be solved by using partitioned tables, one partition per client.

However if the solution you got works for you, go with it.

Please or to participate in this conversation.