Ansible, Zabbix, probably Netbox for inventory
Best way to handle dozens of instances of my app?
I'm in charge of deploying dozens, maybe over a hundred of instances of my app which handles sensitive data and I'm at a loss. I've managed Linux servers in the past no problem, however this is ... A lot.
What are my choices for handling them all? Updates, watching their status, nightly backups, ideally text or email notifications when one or more go down.
Could you use a multi tenant setup or do they need to actually be individual?
Would a SaaS work.
But if they are basically the same, I see no problem with regular updates. I am guessing you are getting paid for such updates and maintenance of the apps.
The problem will possible come if some users have various (different) packages installed.
You could have a maintenance database so you can keep to keep track on what each user has.
@jarcas Think we need more information before we’d be able to suggest the most appropriate solution for your use case.
First question would be: why are you deploying multiple instances? Can it not just be a single codebase like @jlrdw mentions?
What are you going to do about updates? If you need to fix a bug, then how are you intending to propagate that bug fix to each running instance of your app if there are dozens or hundreds or them?
Thanks for the replies!
@jlrdw they need to be individual, separated VPSs or servers - I'm leaning towards SaaS, as the customers aren't techies. Yeah there will be a monthly or yearly fee. Everything else will be the same - all the same packages, same code, etc. just different data in the db and storage.
@martinbean Sorry about that! It's highly sensitive information that needs to be isolated from other instances, so it's the same codebase but separate VPS/servers. Sorry for the confusion.
Updates - ideally push to a release branch after testing and pull on the clients.
It's highly sensitive information that needs to be isolated from other instances, so it's the same codebase but separate VPS/servers.
I recommend you really think this through. There are a lot of SaaS apps that hold sensitive information, yet they don't have an entirely separate enviroment for each tenant. Multi-tenant apps are safe as long as you implement isolation correctly.
Maybe using separated or isolated database/storage (servers) and have one version of your codebase?
Sadly this project started off as just for one company to use, but with feature creep... A lot of feature creep... It eventually became something other companies were interested in, it'd take months to change it to safely share an environment.
Avoid any differences in hardware, OS, installed software, updates - make your hosts as identical as can be.
Don't let yourself do anything by hand on individual host - all actions and changes are allowed only using configuration management system (my choice is Ansible), in batch mode, after proper testing on a test node.
In these conditions administering 100 servers is not much harder than 1, but still challenging, of course.
The same applies to the code of your app: keep codebase and database structure identical on all hosts. Once you let differences spread - you failed, having uncontrollable mess...
Monitoring your infrastructure is very important and points to many problems before they get critical.
Reliable and tested backups are critical. Although I wish you never need them, you certainly will, sooner or later.
Please or to participate in this conversation.