Amazon s3 - zip files before download

Did you ever solve this issue?

I am having the same issue with zipping large files from S3. I currently download them from S3 and then zip them locally but I have exceeded the max memory when downloading big files.

I'm looking for a service that would zip the files on S3 or some other service and then allow me to download a zip file directly from S3. Any input or ideas would be greatly appreciated!

When downloading the files try using streams rather than trying to store the content of the file in a variable, then writing that to a file. This will prevent you from having to load the entire file into memory which is likely where you are having issues.

// Download file
$source = Storage::disk('s3')->readStream('path/to/file.txt');
$tmpfname = tempnam(sys_get_temp_dir(), "tmp");
$destination = fopen($tmpfname, "w");
while (!feof($source)) {
    fwrite($destination, fread($source,8192));
}
fclose($source);
fclose($destination);

// Add file to zip
$zipfile = sys_get_temp_dir() . '/file.zip';
$zip = new ZipArchive;
$zip->open($zipfile);
$zip->addFile($tmpfname, 'file.txt');
$zip->close();
unlink($tmpfname);

// upload zip back to S3, or you could stream it to the browser if that is what you need.  Make sure to delete the local file after though

// To upload back to S3
Storage::disk('s3')->put('file.zip', $zipfile);
unlink($zipfile);

// to stream the file to the browser
use Symfony\Component\HttpFoundation\StreamedResponse;
use Symfony\Component\HttpFoundation\ResponseHeaderBag;

$response = new StreamedResponse();
$response->setCallback(function () use($zipfile){
    readfile($zipfile);
    unlink($zipfile);
});
$response->headers->set('Content-Type', 'application/x-zip-compressed');
$response->headers->set('Cache-Control', '');
$response->headers->set('Content-Length', filesize($zipfile));
$response->headers->set('Last-Modified', gmdate('D, d M Y H:i:s'));
$contentDisposition = $response->headers->makeDisposition(ResponseHeaderBag::DISPOSITION_ATTACHMENT, basename($zipfile));
$response->headers->set('Content-Disposition', $contentDisposition);

return $response;

@kotopoulos10 , @consigliere & everyone else. I have been struggling with this topic for almost 2 years now, and i am rolling out a service soon that just does that. It allows for zipping/tarring files on s3 to s3 concurrently or zipping files on s3 while downloading(piping). Best thing, it is an API and works with any language. I will be looking for early adopters/beta testers in as early as a month. It is extremely fast and 1GB files usually zip in under 5 mins so far. Contact me @ [email protected] and i will put you on the list.

Check out S3zipper API. It can be implemented in many languages.

@edwin84 Do you still maintain it, can you give brief overview what can I expect from your paid api?