I'm writing a little file manager app for a site I'm working on. We want the file manager to map directly onto S3.
While S3 responds pretty fast, performance wise we want to make as few requests to S3 as possible. For example, to process a directory, and give a file tree, we need to make at minimum two calls, one to get the directories subdirectories, and another get the directories subfile. That takes less than 0.3 seconds.
However, as far as I can tell, the API for the Storage features do not have a way to get file metadata (e.g., size, modified date) in bulk. Thus, if a directory has like 50 files, it makes two calls, one for files, one for direcgtories. But to get that metadata it has to make 100 more calls, one each for modified and size. That amazingly only takes 10 seconds on my dev machine but it's unacceptably slow.
Any ideas on how to approach this?
Could you cache the metadata on store/update/delete, and use that instead of querying it 'live'? In redis terms, off the top of my head, I'm thinking of something like a redis hash of :
key -> root:subdir:anotherdir:file1.jpg
mtime: 100230030
ctime: 100102313
size: 1024
owner: uid:34
@ohffs I might give this a try -- good way to test out redis, which I'll need to use later in this app. How would you approach purging stale cache?
Beautiful! Laravel is so good. I add:
$cache_key = 'file_details-' . $path;
if(Cache::has($cache_key)) {
return Cache::get($cache_key);
}
And below, I get all the file details (expensive) and add this:
Cache::put($cache_key, $file_details, config('site.file_metadata_cache_store'));
And... voila. I now can capture all the file information in 0.01 seconds :)
I'll have to add cache busting for file modification functions but this works well. Thanks for tipping me off.
S3 has file information available.
Route::get('files', function () {
// Get a remote list of files.
$disk = \Storage::disk('s3');
$files = $disk->allFiles();
print "<div>";
foreach ($files as $file) {
$modified = date(DATE_RFC2822, $disk->lastModified($file));
$size = $disk->size($file);
$type = $disk->mimeType($file);
print "<li> $file : $modified ($size Bytes) : $type</li>";
}
print "</div>";
});
Route::get('dirs', function () {
// Get a remote list of files.
$disk = \Storage::disk('s3');
$dirs = $disk->allDirectories("./");
print "<div>";
foreach ($dirs as $dir) {
print "<h3> $dir </h3>";
$files = $disk->files($dir);
foreach ($files as $file) {
$modified = date(DATE_RFC2822, $disk->lastModified($file));
$size = $disk->size($file);
$type = $disk->mimeType($file);
print "<li> $file : $modified ($size Bytes) : $type</li>";
}
}
print "</div>";
});
@jimmck, yes, it has that information, but everytime you run $disk->size it makes another API call to S3, which is inordinately slow. In the codeblock you showed, if the folder had 100 files in it, you'd end up making 201 calls to AWS. 1 to get the file list, 100 to get file dates, 100 to get sizes. Each iteration makes another call. Probably trivial on local storage, but on the cloud its slow.
@erikwestlund These do run locally on my ECS instance. You have to gather stats from somewhere. Hopefully you are on the cloud server, otherwise you have to query over the network. Otherwise you need to find batch query services if they exist.
@jimmck, I profiled how long these queries take. We are talking 14 seconds over network for 65 files vs 0.01 seconds when cached.
Also, calls to S3 from EC2 are not local. They are still going over private network in AWS infrastructure. It's the difference between calling database on actual EC2 instance vs a database server in same network. Both will be plenty fast. However, once you move out of local network, trouble begins (e.g., dev server, or you deploy to EC2 instance outside where S3 bucket located, etc.)
You want one S3 bucket which can then be sent to CDN or whatever. If you deploy server in different availability area than S3 bucket it will be slow again. Which reveals why it will be important to somehow make a bulk query (or cache, which is a good solution since I don't think there's an API call baked into storage driver/S3 adapter to get file metadata in bulk/batch.
@erikwestlund You are talking about communicating over a distributed network. My instance is in Oregon. What the exact the exact topology of that is, I have no idea. Thats up to Amazon. They lay out the kinds of latency you might expect. Also the SLA's depend on how much you are paying. But your benchmarking is the correct tool. There is no magic Laravel command.
@jimmck, I think we are on exactly the same page about things :)
It would be a useful option in the files/allFiles methods on the Storage classes to retrieve metadata with the retrieved files (kind of like an ls -a command!), although I guess it might become difficult/possibly non-performant across different platforms to implement interchangeably.
Please or to participate in this conversation.