Out of memory Exception

Hello, I know this is a very common Exception. But I can not fix it... It occurs in 1% of cases, when users try to download PDF documents. Sometimes on very small documents, sometimes on larger ones (between 2MB and 20MB). 2 examples :

Allowed memory size of 536870912 bytes exhausted (tried to allocate 134217736 bytes)

Out of memory (allocated 456712192) (tried to allocate 67108872 bytes)

Here is the code :

public function download_document(Document $document, $streaming = false)
    {
        $file = $_SERVER['DOCUMENT_ROOT'] . "/public/uploads/" . $document->url;
        $name = rawurlencode($document->title);

        if ($streaming) {
            return response()->stream(function () use ($file) {
                $stream = fopen($file, 'r');
                fpassthru($stream);
                fclose($stream);
            }, 200, [
                'Content-Type' => 'application/pdf',
                'Content-Disposition' => "attachment; filename*=utf-8''" . $name . ".pdf",
                'Expires' => '0',
                'Cache-Control' => 'must-revalidate',
                'Pragma' => 'public',
                'Content-Length' => filesize($file),
            ]);
        } else {
            header('Content-Description: File Transfer');
            header('Content-Type: application/pdf');
            header("Content-Disposition: attachment; filename*=utf-8''" . $name . ".pdf");
            header('Expires: 0');
            header('Cache-Control: must-revalidate');
            header('Pragma: public');
            header('Content-Length: ' . filesize($file));
            readfile($file);
            exit;
        }
    }

I tried 2 different approaches, using streaming and without. But it doesn't seem to solve the problem. My VPS is not overloaded.

Do you have any idea? Thanks!