Jobs issue with switching redundant databases

Level 61

It almost looks like they are still trying to connect to the old database instance.

Yes, cause the framework is "loaded" only once when the worker is started and it is still targeting the old database.

Do not run it as a Daemon. It will be slower since it will have to boot whole framework on each job, but it will not have issues as you've described.

Level 2

Thanks, isn’t there another solution?

The IP for the database stays the same, it’s just pointed to the backup instance.

I would have expected the workers to automatically connect to the new instance.

Level 61

And what exactly changes in the DB_* configuration from .env file?

You can create a scheduled command which will check if the correction is successful and if not it will purge DB connection and establish a new one based on some new configuration.

Level 2

Nothing changes in the config/env files.

Level 61

So you are saying that with the same credentials and same IP address and same database name it is not able to access the database which by all configuration points is identical?

Level 2

Yes, I don’t change anything in the code, I just have to restart the workers to make sure they look at the new instance.

Best Answer

Level 61

Then just decorate jobs

// You can put this somewhere in the AppServiceProvider
\Illuminate\Support\Facades\Queue::before(function (\Illuminate\Queue\Events\JobProcessing $event) {
	// check if there are connections
	// then just return (defensive programming)
	// if not execute something like below
        /** @var \Illuminate\Database\Illuminate\Database\DatabaseManager $databaseManager */
        $databaseManager->purge();
        $databaseManager->reconnect();
});

Level 2

Thanks! Could I also do that the other way around? So after a job is processed or failed close the connection? That way all new jobs should reconnect right?

Level 61

Could I also do that the other way around?

Do you need a connection after a job?

Level 2

Do you need a connection after a job?

Not for that job anymore.

A bit of side information, the workers run on 5-15 different servers (depending on how busy it is). Each server is responsible for their own queue(s) and there are usually more than 100K jobs queued and hundreds of jobs processing at the same time.