Deployments and migrations

Hi, I work on a large laravel app that needs to maintain zero down time. 90% of the time a new release involves changes in the db (new columns/tables, or worse) The new code in the release relies on the new db structure. Meaning if I run the deployment in these steps:

git pull
php artisan migrate

After I run the first commend, the app will crash in some places until the migration will complete.

Another problem I have with running migrations is that once I execute that command, I feel things are out of my hands, if something breaks or gets stuck, or there's a mysql error, I don't have control to easily and quickly abort/revert/fix or even understand if something is now critically wrong with the db.

What I've done in the past with another framework was to run manually all the db changes, if it was a table-locking change, we'd run it in the background, and only after that we git pulled the code. I see migrations useful only for dev envs and staging/qa servers, I don't understand how this can be used in production without crashing the appliaction.

Strangely I feel like these 2 issues should be on everybody's mind, but I couldn't find this addressed anywhere. So am I missing something? is there a best practice for deployments with db changes? any recommended workflow?

Cronix

7 years ago

Level 67

It's impossible to have "zero downtime" with db updates. Most people put the site in maintenance mode (after prewarning the users in advance the site will be unavailable for some time), do the updates, then put the site back up.

I think the only way to do a true zero downtime deployment is to whip up a new server, install everything fresh on that, run the db changes. Once it's completed, kill the old server and switch to the new. That way the people on the live server are left unaffected as nothing is actually changing until you switch servers.

A very large company I used to contract with did it that way (they had about 10 app servers). They used ansible and just threw away the old servers and built new ones with the latest os updates/code/db, etc. They didn't "upgrade" the existing servers. They just built up new ones, and deleted the old ones once everything was switched over. It was all automated, and pretty sweet. They were a lot more masterful at devops than I am though, but it seems to fit the scenario you are wanting.

1 like

Please or to participate in this conversation.