Scaling Laravel - required infrasturcture for a PaaS application

I have an application that runs on Laravel and is managed by Forge laravel.

My application sends sms messages campaigns, with specific promotions.

each campaign is scheduled as a cronjob and is executed as queued job that sends 1Ks of messages per minute each time via third party sms providers.

each sms is logged with the third party message_id and is tracked for engagement performance. this means:

a message is created-> sent to RESTful API RestfulAPi ->returns message status etc.

each status change returns a push notification via pusher to a dashboard in realtime. that is: it refreshes an Ajax dashboard per pusher return event.

 I have reached the stage were I need to scale horizontally .

I have looked at htop and other monitoring tools but can't understand how to derive from that how many requests are actually running and what is the required php-fpm setting , pm.max_children , nginx configuration.

I am getting 502 errors when stress testing on a digitalocean 2 core 2GB droplet. 2000 requests per minute fail after 18 seconds with 502.

I have watched https://serversforhackers.com/scaling-laravel/forge/forge-recipes

but am not sure if the structure @fideloper showed is the correct one for this use case.

most of the stress is backend not many users, it is more of a managed automated system, not sure I need a load balancer between the apps. and I'm not using cache for my execution.

from what i have seen using blackfire most of my logic runs fast but mainly uses like 3-5 mb of memory.

your tips and advise appreciated.

fideloper

7 years ago

Level 11

You may want to look into APM monitoring on something like data dog or new relic. Application Monitoring should give you a better idea of things like requests over time to see if you're hitting issues when getting spikes in HTTP requests or what your other bottlenecks are (vs blackfire which may show a more limited view of just what's going on in code and not necessarily how that affects overall servesr).

There's a few resources in the PHP topic in servers for hackers that explains how you might want to decide what to set your PHP-FPM max_processes (etc) to. Specifically the top two video series linked there talk about it.

There's also my Scaling Laravel course which can give you pointers on how to scale out (altho what your bottlenecks are isn't specifically covered - generally you'll need some monitoring to inform you of that).

1 like

Ace

7 years ago

Level 7

thanks @fideloper I will look into your suggestions.

1.Do you think this app structure requires a horizontal structure with an app load-balancer or one droplet for the app itself and separate droplets for the other resources( DB, Cache,Queue,Cron), as most of the work is in receiving/sending the http requests and processing backend queries and queues.

2.Once I get the correct metrics and benchmark my application, it seems that the preferred structure for my system needs to be scaled on demand,

can this be done with Forge/DigitalOcean droplets or do I have to use Aws for this kind of scaling?

alex_time

6 years ago

Level 18

@FIDELOPER - your course is REALLY interesting @fideloper ! Anyway I am wondering...having a unique load balancer as "gate" wouldn't possibly be a single point of failure? The only alternative I can think of is a round robin DNS on top of two (or more) infrastructures like the one you show in the serie...am I missing something or are there any other possible solutions?

kobear

6 years ago

Level 4

@ACE - DataDog and/or New Relic are your friend in this case. There are a couple of other options in this space, such as App Dynamics (now Cisco), and Dynatrace. If you want to stick with open solutions, you could look at using The Open Tracing Project ( https://opentracing.io/ )

re: App structure if you want to make sure you can scale it all:

load balance the app. (That is just because I prefer to have failover for user interface)
load balance your cache layer.
load balance your queue workers. (This is where you will get the most gains).

DB's are best left on a single node, unless you want to have Active/Passive fault failover.

Cron does not load balance - it is a local Unix utility. You could use a system like DKron ( https://github.com/victorcoder/dkron ), but that seems overkill.

So, your cron server could just kick off the job that seeds the queue workers to send the messages out, then when SMS messages are returned (via webhook presumably), other workers could process those. That way the load and magic happens on the load balanced queue worker layer.

Good luck!

Please or to participate in this conversation.