How to stream download a file, that is collected from a 3 party link?

I need to collect a file, from an external link, and then send that file in response. My current code is working:

   use GuzzleHttp\Client;

    $client = new Client();
    $response = $client->get("https://...");
    if ($response->getStatusCode() != 200) return abort(404);

    $file = $response->getBody()->getContents();

    return response()->streamDownload(function () use (&$file) {
        echo $file;
    }, $fileName, [
        "Content-type" => "application/octet-stream",
        "Content-Disposition" => "attachment; filename=" . $fileName, ";",
    ]);

The problem is when the file is huge in size. I have a situation where a 1 GB file needs to be downloaded. I receive error:

Allowed memory size of 134217728 bytes exhausted (tried to allocate 300033576 bytes)

Is there a way to alter the logic to enable big file downloads? I have some logic in my endpoint, so I can not just put the link of the file directly in the view.

martinbean

3 years ago

Level 80

@ivymasterman The problem is, you may be trying to stream the response from your server, but you’re still trying to download the file in its entirety and read it into memory before sending a single byte to your end user, hence the memory issues.

Instead, you should be both stream-downloading the file from the third party so that you’re reading it in parts and outputting those parts back to the user so you’re not trying to stuff the entire file into memory at any one point.

Or just stream the file directly from the third party instead of doing this weird hop from their server, to your server, to the end user. You’re just wasting your bandwidth acting as an intermediary for no benefit.

1 like

ivymasterman

3 years ago

Level 3

@martinbean Yes, you are right. That is why I was wondering how to stream and download a file from 3 party URL. I tried altering my code to this:

return response()->streamDownload(function () use (&$firmware_file) {
        echo file_get_contents('https:....');
    }, $full_file_name, [
        "Content-type" => "application/octet-stream",
        "Content-Disposition" => "attachment; filename=" . $full_file_name, ";",
    ]);

this works until 50 MB and fails also.

martinbean

3 years ago

Level 80

@ivymasterman Because you’re still trying to read the entire file into memory if you’re using file_get_contents.

Just stream the file directly from the third party. Why are you downloading it to your server, just to send it on to the end user? Just give the end user the URL of the file on the third party so the file is directly downloaded from there instead of having to hop across to your server first.

ivymasterman

3 years ago

Level 3

@martinbean Let's say that I need to do It this way. Is there a way to do the thing that I asked, or I should find another way?

martinbean

3 years ago

Level 80

@ivymasterman I’ve already said what to do? You need to stream the file from the server, instead of trying to download it into memory.

ivymasterman

3 years ago

Level 3

@martinbean I know that, but how can I achieve that, can you help me with it, with code? How can I do that, since the upper code does not do that. I can not simply display the link.

ivymasterman

3 years ago

Level 3

@martinbean How can I simultaneously stream the download to my server, and stream the downloaded bits to the client?

miloslavkostir

1 year ago

Level 1

@ivymasterman I believe you already found the solution.

$stream = fopen('path/to/bigfile', 'r');

return response()->streamDownload(function () use ($stream) {
    while ($chunk = fread($stream, 1024)) {
        echo $chunk;
    }
    fclose($stream);
}, $full_file_name, [
    'Content-Length' => filesize('path/to/bigfile')  // for progress bar
]);

$client = new \GuzzleHttp\Client();
$response = $client->get("https://...");
$stream = $response->getBody();

return response()->streamDownload(function () use ($stream) {
    while ($chunk = $stream->read(1024)) {
        echo $chunk;
    }
}, $fileName, [
    'Content-Length' => $stream->getSize()  // for progress bar
]);

Instead of while() you can use fpassthru():

$stream = fopen('path/to/bigfile', 'r');
fpassthru($stream);
fclose($stream);

// Guzzle
$client = new \GuzzleHttp\Client();
$response = $client->get("https://...");
fpassthru($response->getBody()->detach());

Please or to participate in this conversation.