I just deployed some changes to our live site (using Envoyer).
It included two migrations. The first one passed, and the second one failed. Since it failed, the new code was not going to be used. So we kept using the old code, but the first migration had already altered the database.
So the site went down because the old code couldn't use the new database structure (from the first migration).
We couldn't redeploy an old commit, because we were already using old code. And we couldn't go forward because that second migration would keep failing.
We don't like being in such a situation. It gives us anxiety. We receive alerts that the site is down. Our users yell at us. We're stuck. It's not recommended for our mental health. :(
Is there anything we can do about this?
It occurred to us that if we wrapped all migrations in a single transaction, even the first one would "roll back" if the second one failed. If we could do this, everything would continue to function, and we would be very very happy.
Any comments?
Sounds to me like you didn't run this properly in the UAT environment.
You didn't specify what the failure was, or any error messages, but it should have failed in dev, and test as well.
What I would do, is either manually roll back the changes made by the first migration, or manually deploy the changes that failed.
Yes, this wouldn't have happened if we would have tested with live data.
Yes, we fixed it by adjusting the failing migration so it would pass.
But wouldn't you also agree that if all migrations were wrapped in a single transaction, this would have saved us a lot of heartache?
@magmatic Migrations have a down method for this purpose. If it was a destructive migration (e.g. not possible to revert because the migration results in deleting data, etc) then it’s time for a hotfix, and then to look at why the migration failed, learn the lesson, and put in place provisions to prevent that situation from happening again.
There are two rules to always do if messing with migrations on a live site.
There was another user a while back that lost around 6 months of company data by not following these rules. What's ironic is the user had no current backups.
One reason not to use Migrations for one-time data transforms. Use a command for this and run it manually.
Make sure such data transform commands are idempotent - that is, you can run them many times and the result will be the same.
Get used to looking at the migrations table. Sometimes you can only get out of a hole by altering the table (adding or removing a row).
But you can always revert to the backup... ? You have backups? Especially the one you took before transforming data?
You had the site in maintenance mode when you did this so that the data was not being changed whilst you worked?
I find the One Time Operations a nice package for handling these "one-offs" as they work like migrations, can run them async of synchronously out of the box and means you keep the migrations for what they're designed for, schema definitions.
Please or to participate in this conversation.