I'm working on a prototype at the moment, where I'm slowly getting closer and closer, to throw everything online and test on concurrent users. I'm looking for a bit of advice here.
It's basically a small task app, that will have a server for the App/API, DB and NodeJS (Redis Pub/Sub) for web sockets and then another "worker" server, for the actual tasks, which consists of lots of cURL download/uploads and a bit of crawling.
This service will have peak hours, where a lot of users will be placing tasks/jobs in the queue (around 200 jobs/hour), so right now I'm trying to plan a little a head of which service size I will have to choose, how many workers it's possible to have and even if beanstalkd is the right way to go.
I'm hoping that the users won't have to wait more than a couple of minutes before their task is completed. In my tests locally, the biggest tasks takes around 30 seconds to complete, using Queue:listen on Homestead. As far as I know beanstalkd on Homestead only uses 1 worker?
I don't have that much experience with beanstalkd and multiple workers and would like to know if there is a limit to how many workers you can have?
In my research, on setting up multiple workers through supervisord, I often see people mentioning 10 workers on various forums, but none of them mentions their server specs or even if they are using a VPS at all. I'm hoping for a bit higher number of workers, but I don't know if that is completely crazy?
I'm planning on deploying the worker part on a 4GB, 2 Core server size on DO, would it be possible to have 50 workers on this size?