kingsloi's avatar

How best to handle queued file uploads to S3 behind a load balancer (2+ servers)?

I'm in the process of building an API for my company. So far so good, I bought the Servers For Hackers recently series and learnt so much! I was worried about performance, but I got this now.

My API will accept file uploads, and will receive 10,000s of requests a day. I am heavily using redis for pretty much everything, including uploading files to S3 in the following scenario:

  • User POSTs file
  • I store locally, add to queue to upload to S3
  • Queue is processed, file is uploaded to S3, local file is deleted

I've been developing locally, and staging on a single server. Now that I am preparing to move to production, I just realised that I am going to run into the same issue as sessions, and logs, etc. where the uploaded file will exist on any one server, but the queue worker is only ran on one.

I still want to queue my uploads, but how can I ensure that my files are uploaded, regardless of the server they're on?

I have thought about potentially uploading the file, store it with laravel, and then use scp to send it to a single server

Or, I didn't want to store the file_contents in redis, but would that be another solution? I am worried that I will fill my redis store up if that were the case, though.

Surely this can't be just me who has ran into this issue? Any suggestions would be greatly appreciated!

0 likes
2 replies
kingsloi's avatar
kingsloi
OP
Best Answer
Level 2

I decided to base64_encode the contents and then store that in Redis instead.

$contents = base64_encode(file_get_contents($file));
UploadFileToCloud::dispatch($filePath, $contents, $lead)->onQueue('s3-uploads');
amora2972's avatar

Hi @kingsloi , it is really weird when things come to working behind load balancers there are not too many sources provided on the internet.

I am running into the same problem now, but I am wondering how to solve it as I am not using redis, if you have any idea could you please help.

thanks.

Please or to participate in this conversation.