SSL setup in Forge with loadbalancer

Ok, I've kept working at it and finally figured out how to do this well. So for anyone who doesn't currently know how to install a LetsEncrypt SSL on a load-balanced application in Forge, here's how you do it...

1.) Install a lets-encrypt SSL on your load-balancer, as you normally would.

2.) At the time of this post, Forge does not allow you to create cron-jobs on load-balancers through the interface. No worries, you can add the CURL command (for renewing your SSL) to one of the web-servers behind the load-balancer.

3.) (Optional) For optimal security, delete your firewall rules on the servers behind the load-balancer. The only one you'll need for the load-balancer to access, is port 80. So create a new rule -> Name: HTTP, Port:80, and IP Address should be set to the internal IP Address of your load-balancer (It's the one in parenthesis on the top right-ish of Forge). Basically, this rule says only the load-balancer can access these servers and only on port 80.

4.) At this point your server goodness is all set up. However, your internal network is using HTTP. This is fine for security purposes, but causes Laravel generated links to be HTTP instead of HTTPS. To fix this, install and setup "fideloper/proxy". It will tell Laravel to take advantage of the proxy headers your Forge load-balancer passes to it, causing your links to turn back to HTTPS.

Further Reading: This setup gets a user from their browser to your load-balancer using HTTPS, but from the load balancer to your web servers uses HTTP. The network between your Forge load-balancer and it's web-servers is considered a private network, and HTTPS is not needed. This is a common practice called "ssl termination", and when done correctly is PCI Compliant. It allows your load-balancer to handle the SSL junk and simplifies the process of maintaining and updating your SSL certificate by keeping it all in one central location.

For the https issue I set Laravel to always use https versus your set up and works fine for me. Otherwise great information.

@jekinney Thanks!

@dahall4 did you get any errors while installing the cert on the load balancer. I am getting the following error:

ERROR: Challenge is invalid! (returned: invalid) (result: {"type":"http-01","status":"invalid","error":{"type":"urn:acme:error:unauthorized","detail":"Invalid response from http://www.wrmd.org/.well-known/acme-challenge/6veBlZgeFngt0u7KqbVl1ToEQ6XSt7kuGseH_RzFccw [104.131.158.216]: 404"},"uri":"https://acme-v01.api.letsencrypt.org/acme/challenge/xKyIWeBdiwN1YCCI74U7Ntk1tiUZTPbwLGWTLFbop0o/28056177","token":"6veBlZgeFngt0u7KqbVl1ToEQ6XSt7kuGseH_RzFccw","keyAuthorization":"6veBlZgeFngt0u7KqbVl1ToEQ6XSt7kuGseH_RzFccw.m5K47GzB4XA5a5xo_-eLcLm7SXeOhcLCXHFXEDwhg3M","validationRecord":[{"url":"http://www.wrmd.org/.well-known/acme-challenge/6veBlZgeFngt0u7KqbVl1ToEQ6XSt7kuGseH_RzFccw","hostname":"www.wrmd.org","port":"80","addressesResolved":["104.131.158.216"],"addressUsed":"104.131.158.216"},{"url":"https://www.wrmd.org/.well-known/acme-challenge/6veBlZgeFngt0u7KqbVl1ToEQ6XSt7kuGseH_RzFccw","hostname":"www.wrmd.org","port":"443","addressesResolved":["104.131.158.216"],"addressUsed":"104.131.158.216"}]})

and I haven't found a way around this yet.

@devinfd I didn't get any errors installing the cert onto the load-balancer. I'm guessing it's related to something else. I have gotten errors when either:

A.) I did not have the DNS records setup appropriately for LetsEncrypt. (All domains you list for the SSL need to point to the server you're installing the cert on). Not doing so can create challenge issues like what you're experiencing.

B.) I had old PPA's or odd-ball extra stuff installed on my machine preventing Forge from updating my machine properly.

If those don't work, I'd try booting up a fresh load-balancer if possible (sometimes that's just easier than running down every possibility). Hope this helps and best of luck!

@dahall4,

Do you mind sharing the a general guideline/steps you took to setup load-balancer with your 4 web-servers? I have been searching online, but cant seem to find anything concrete

Hi @ralee ,

This guide assumes you're familiar with and using Forge to provision your servers and should get you a setup with:

• 1 Loadbalancer & SSL
• 2+ Web Servers
• 1 Database/Cache Server
• AWS file storage

Loadbalancer

Create a new server in Forge and check provision as loadbalancer. I've got about 5.5-6M pageview/mo. and I use a 2G D.O. droplet with no problems, if that helps you approximate the size you'll need. If you aren't using SSL, you could probably go much smaller than that as your loadbalancer handles all the SSL stuff.

Add your site to the newly provisioned loadbalancer (lets say mysite.com for this example). After pointing your DNS records to the loadbalancer, you can add an SSL to mysite.com.

I typically name this server mysite-loadbalancer-1.

Web Servers

Next create 2 or more new servers that you'd like to use as your main web-servers. I usually name them mysite-web-1 and mysite-web-2 and make them 512Mb droplets. Keeping them small allows you to add more, which allows for more connections. However, that may not work as well for you if your site does a lot of processing and requires more resources.

For each server, setup your site as you normally would, but DO NOT install an SSL, you do not need them after the loadbalancer. Also, if you plan to have any Tasks/Cron Jobs, only add them to ONE web-server. I always add them to the first one (mysite-web-1) so I know where they all are.

Last, go to the network tab of each server and delete the two firewall rules which allow traffic from anywhere. Then add a new HTTP rule, from port 80 and limit the IP address to the internal IP of your loadbalancer. Hint: it's the IP in parenthesis in Forge's blue panel header.

Database

The last server you need to create is the database/cache server, I usually call it mysite-db-1. Provision it with Forge as a normal server, but maybe with a bit of extra ram depending on how DB/Cache heavy your site is. I like to use 2GB for this as well, and while I do have a decent amount of Caching going on, I don't have a ton of DB activity per page (just a few simple queries).

Delete the default site, so you have none. Delete the firewall rules so there are none. This will stop any outside traffic from getting to your DB server.

Connecting Everything

Loadbalancer to Web-Servers

Go to your loadbalancer (mysite-loadbalancer-1) and click the Network tab. Connect it to all your web servers (leave your DB server unconnected).

Web-Servers to DB

Go back to your first web-server mysite-web-1 and click the Network tab. You should connect this and every other web server to your DB server.

Application Settings

All your hardware is setup, which just leaves a few application settings.

DB & Cache

In the .env file of your application, your database host should be set to mysite-db-1's external IP. If you are using redis (which is what I use), your redis_host should also point to this IP.

Trusted Proxy

In order to grab information from the request, such as client IP, it has to be "proxied" through the loadbalancer to the web-server. There is an excellent little package for this: fideloper/proxy. After installing this, the TRUSTED_PROXY_IP in your .env file should point to the internal IP of your loadbalancer. This allows traffic from your loadbalancer to have all the headers like: client IP, http referrer, etc.

IMPORTANT: Without a package like this all your users will look like they came from the same source... Your loadbalancer.

File Storage

To have file storage across all your web-servers you should use an external file storage service like AWS or some other CDN. I recommend one that Laravel works with out of the box.

Done

Congrats! Hopefully you found this guide helpful and you're good to go!

@dahall4 fantastic write up! I'd love extra info on server monitoring set up you are running :)

Thanks @james123! I normally would just use a service for monitoring. New Relic would usually be my pick, although sometimes I'll use something that I don't have to install on my servers, like Pingdom. There's a great article on 'Servers for Hackers' that shows how to setup New Relic.

hi, just wondering if this setup has sticky sessions?

@stocksph it does have sticky sessions as long as you are using Redis (preferred) or the database for your session data.

@dahall4 This post has been really helpful.

Could you tell us how we can connect to the DB server via HeidiSQL or similar?

Thanks

Thanks @RossCo!

You can probably find more in-depth how-to's if you google around a bit, but I'll give you a quick rundown of my setup.

I use Sequel Pro, which is probably pretty similar to HeidiSQL.

If you're using Forge to provision your servers, make an SSH Key for your computer.

In Sequel Pro, or your SQL interface, connect via SSH. Enter the IP address of your DB Server for the SSH Host, the SSH User should be "forge", and the SSH Key should link to the one you're using for Forge.

Next, your MYSQL host should be "127.0.0.1", Username should be "forge", and your password should be the username Forge emailed to you when you provisioned your server.

Hope this helps!

@dahall4

Awesome! That worked for me.

Thanks again.

@dahall4

I was wondering if you could expand your post with how you connect up Redis?

I have the same setup pretty much as yours, 1 load balancer, 2 app servers, 1 DB/cache server. My app is setup to use the Redis env settings but when I run redis-cli monitor, I don't get any connections on the DB/cache server but the 2 app servers are showing hits, albeit on alternate page loads due to the load balancer.

I setup the env file on both the app servers to point to the DB/cacheserver for Redis. Purely for testing I allowed all IPs to connect through port 6379 on the DB/cache server but still no luck. Also, the 2 app server still get hit.

env example

CACHE_DRIVER=redis
REDIS_HOST=external_db/Cache_ip
REDIS_PASSWORD=null
REDIS_PORT=6379

An example output from the monitor which might help shed light on why the 2 app servers get hit and not the DB server.

 [0 127.0.0.1:36152] "GET" "myapp:3164295944page-url-map"

Any info or pointers would be great.

Thanks!

@RossCo Hmmmm.... Your .env setup looks the same as mine. It's ok to use port 6379 if your DB/Cache server has your app servers white-listed and everything else blocked. You can do that through the Forge interface. The only other thing I can think to check, aside from making sure your app servers can access your cache server, would be that your .env settings are being read by your application. The first thing I'd try there is redeploying your app setup to flush any cached settings... especially if you modified your .env file after your last deployment.

@dahall4

Thanks for the reply - I managed to solve it earlier. I forced each server to reboot and tried it again and it worked. Must of been some weird caching but seems to work now.

@RossCo Nice!

@dahall4 Thanks for sharing your mini tutorial!

But how is that load balancer different from a round robin? eg: https://www.youtube.com/watch?v=Zp-LKI9AS5k

Also, when you check provision as loadbalancer in forge, does it also install a bunch of other stuff on that server eg mysgl/php

@devinfd Did you ever get that straightened out? I'm having the same problem and it looks like my DNS is set up properly; I can access the site on my servers but can't install a cert on the load balancer.

@dahall4 thanks for the awesome guide

I got some questions that maybe sound obvious but because I'm migrating for a live working site on a single server to a load balancer configuration I want to have everything working without having to do lot of tests.

You mention to use the Db-server for database and cache. On my case I use memcached so I changed the host to external ip of db server (not internal Ip)

For queue I use beanstalk. Shall I leave that also on the db server ? You mention that you run all your crons on site-1 so I'm not sure.

I do heavy use of the queue because I log every request(user id and request data) to API so not sure if I should leave this to localhost and add a worker on site-1 or change to external ip of db-site and start the worker in there

Thanks!

OK Guys to update my own questions. After annoying Taylor with some support requests I´ve been told it's better to use internal Ips instead of external.

My issue with the session not persisting in other hand I figured it after struggling all day with the issue. The problem was that both site server s were deplyed with forge and I needed to set BOTH APP KEY to be same in my .env Yep that silly thing took me hours ... hope it helps someone else