Problem generating big XML feed

@sr57

When you visit the URL you have to first generate fresh data and then return it.

@MerryChristmas are you generating the file in a job or a command?

@Sinnbeck

I generate file when the user visits the link. Assume, that there is hundreds of possible feeds and each has 300 MB. It's not possible to pre-generate them and keep them on disk.

So on visiting the link I will generate temp file via fwrite() and return it to user (how? i can't load it via PHP, because of memory overflow; I can't redirect to the file) and ~60s later delete the file.

@Sinnbeck

Problem is when I do traditional

foreach(Products::get() as $product){
						$output . = '........';
}

return $output;

It will just memory overflow.

That's why I went for:

foreach(Products::get() as $product){
					fwrite(...);
}

But now don't know how to return generated file without (1.) redirect (not supported by XML consumer) and (2) without loading it with PHP (yet another memory overflow).

Maybe be instead of occasional fwrite() and generating file I could somehow "stream it" and not hold the entire result in memory?

@MerryChristmas so you are creating the file and sending it in the same request? Why not generate it using a scheduled job?

@Sinnbeck

Because each user (employee) can create unlimited number of his personally structured feeds. I can't keep on disk pre-generated all these feeds. They have to be generated on demand and then thrown away. Or avoid generating file.

@MerryChristmas ok. Then I am sure about the delivery of the file/stream. You can perhaps look into streaming from the database https://laravel.com/docs/9.x/queries#streaming-results-lazily

And if you can send it as a file, you can stream that as well https://laravel.com/docs/9.x/responses#streamed-downloads

@Sinnbeck

Thanks. Looking at these link I came to final code, that seems to work:

 $fs = Storage::disk('public_folder');
$stream = $fs->readStream('/_temp/feed-wholesale.xml');

    return response()->stream(
        function () use ($stream) {
            fpassthru($stream);
        },
        200,
        [
            'Content-Type' => 'application/xml'
        ]
    );