muuucho's avatar
Level 11

Image gallery where images are stored on a remote SFTP

I have an app where only logged in users can view images. The files are stored on a remote server that is not exposed to public download, so they are downloaded using SFTP. The filenames are stored in a DB table, images. In an image CRUD, users filters on images and can get max 10 filenames at a time, for that I will let the user use various filtering, sorting and then I will use pagination to limit the result to 10 images.

I have a rough idea how to accomplish this:

  1. The user filters and sorts the images in a view and hits a button Display.
  2. A method index() in a controller asks the table for images according to the users choices and gets a collection of max 10 file names in $file_names
  3. A method download($file_names) downloads from remote server and store the files temporarily in Laravel's local storage.
  4. A method show($file_names) renders a blade view show and the sends the array of file names to it.
  5. That view loop over the array displays each image using the tag <img>.

Questions:

  1. Is it storing the files temporary in local storage the best solution for displaying them to the logged in users or are there other better (faster) solutions?
  2. How can I know when the view has rendered so that I can safely delete the locally temporary stored files?
  3. If more than one user downloads an image there will be a collision issue when trying to store the same file twice. Also, there might be a problem if a file gets deleted when a user has rendered the view, while another user still tries to reach for that file. How can I solve that?
0 likes
4 replies
LaryAI's avatar
Level 58

To address your questions and provide a solution for displaying images stored on a remote SFTP server, let's break down each part of your requirements and questions:

1. Storing Files Temporarily in Local Storage

Storing files temporarily in local storage is a common approach when dealing with remote files that need to be processed or displayed. However, there are alternatives:

  • Stream the files directly: Instead of downloading and storing them, you can stream the images directly from the SFTP server to the client. This can be done using a proxy route in Laravel that fetches the image from the SFTP server and streams it to the browser. This approach avoids local storage entirely.

2. Knowing When to Delete Temporary Files

If you choose to store files temporarily, you need a mechanism to delete them after they are no longer needed. Here are a few strategies:

  • Scheduled Task: Use Laravel's task scheduling to periodically clean up old files from the temporary storage directory.
  • Session-based Cleanup: Store a reference to the temporary files in the user's session and delete them when the session ends or after a certain period of inactivity.

3. Handling File Collisions and Concurrent Access

To avoid file collisions and ensure that files are not deleted while still in use, consider the following:

  • Unique Filenames: Generate unique filenames for each download session. You can use a combination of the user ID, timestamp, and original filename to ensure uniqueness.
  • Reference Counting: Implement a reference counting system where each file has a counter that tracks how many users are currently using it. Only delete the file when the counter reaches zero.

Example Implementation

Here's a basic example of how you might implement streaming directly from the SFTP server:

use Illuminate\Support\Facades\Storage;
use Illuminate\Http\Request;

class ImageController extends Controller
{
    public function show($filename)
    {
        // Assuming you have a configured SFTP disk in config/filesystems.php
        $disk = Storage::disk('sftp');

        if ($disk->exists($filename)) {
            // Stream the file directly to the browser
            return response()->stream(function () use ($disk, $filename) {
                echo $disk->get($filename);
            }, 200, [
                'Content-Type' => 'image/jpeg', // Adjust the content type as needed
                'Content-Disposition' => 'inline; filename="' . $filename . '"',
            ]);
        }

        abort(404);
    }
}

Summary

  • Streaming: Consider streaming images directly from the SFTP server to avoid local storage.
  • Cleanup: Use scheduled tasks or session-based cleanup for temporary files.
  • Unique Filenames: Generate unique filenames to prevent collisions.
  • Reference Counting: Implement a system to track file usage and safely delete files.

This approach should help you manage the images efficiently while ensuring that users can view them without unnecessary delays or conflicts.

Snapey's avatar
Snapey
Best Answer
Level 122

I wouldnt bother replicating the file locally.

create an image route that accepts some form of image reference. When this route is accessed, your server goes to the ftp server, downloads the image and immediately streams it to the user's browser. This script is single purpose and can be called multiple times at once. More importantly you can wrap the image details in a signed url so that only the current user can access it.

Then the user image selection stuff is just a matter of returning a page with a bunch of secure urls

1 like
muuucho's avatar
Level 11

@Snapey Alright, thanks. Does Larry's suggestion point in the right direction? If so, do I just call that method in the view, something like: {{ route('image.show', $file_name) }}

EDIT: could you please explain more about how to "wrap the image details in a signed url"?

Snapey's avatar

@muuucho described here https://laravel.com/docs/11.x/urls#signed-urls

laravel will create a signature so if you pass the user id and the file name with the url then it cannot be tampered with, so the user cannot try changing the filename, and they cannot send the links to a different user because you can compare the user id in the url with the current user id.

In addition you can put a time limit on the url, after which the url is no longer valid

1 like

Please or to participate in this conversation.