What I did to fix it was to first create a new access token on my personal account. Somehow this was deleted, and the only token was on the client's team account. So now with two accounts I tested it that moving forward I can use the new token, all good.
To move the servers over, I did the following.
- Took a snapshot of each
- Transferred ownership of the snapshot to myself
- On my account I accepted ownership
- I created a new droplet from each snapshot
- Had to log into the console to update the root pass (just has to be done so forge can login)
- Updated the meta for the Forge servers. I just updated the IP addresses.
- I will delete the droplet from the team (but not until the DNS is updated)
The caveat being I have to update the IP address for each site in the DNS settings which I don't have access to (for some, yes). But with this method at least there wont be any downtime. I just need to wait for the client's to get back to me.
Additionally, I will have to repay this client who carried these servers for the 4 months! Surprised they didn't notice, and thankful they didn't just delete them.
I have a theory as to how to reproduce the issue, but not sure I even want to try. What also is strange is, when I attempt to delete the token from Forge, it wont let me and states that it's still being used by one of the servers. Well, it shouldn't be since I updated them all. I'll have to triple check before deleting the original droplets.