sl0wik's avatar

Thumbnail generation approach.

Hi guys,

I'm working on marketplace (like airbnb or craigslist). Growing amount of listings generating large amount of photos. Each photo need watermark, and thumbnail in various sizes.

What's you approach to thumbnail generation?

I'm considering two solutions:

Generate thumbnails on fly

Only original image is stored on server and each thumbnail is generated on fly + cached by cdn.

  • It save a lot of space
  • I can get any resolution I want
  • Relatively long time to load image for first time or after not opening it for long time
  • It might be sensitive to ddos attacks

Generate thumbnail on offer upload

All thumbnails are generated on upload.

  • There is no risk that some users will have to wait long time to load
  • Taking much more space
  • Each new resolution is more space and CPU work
0 likes
2 replies
jekinney's avatar

Depending on the size of the thumbnail your really only talking about a couple of mbs per thumbnail.

First and foremost you need to set caching headers properly and probably for a longer length. By default each page load sends a request for each file (CSS, js images etc.) in production this isn't needed generally. So even though the files are cached it still double checks every request.

Extending cache times, using versioning and setting headers will help tremendously.

When I have images that come from users, I always set a max size (dimensions) and thumbnail requirements. Can be dynamic if needed in a setting file or table.

I upload the original to the server, fire off an event. This event will have at least two listeners, usually three.

I like intervention image package, but there are other software and plugins. Each listener is responsible for resizing the original. Last listener uploads the resized to hosting (if applicable, like googles cdn or s3) and deletes the other images on the server.

I set a default image on upload to a image that says image is processing. Obviously the listener updates the path in the db after upload to final spot/resized.

Now I generally set one image to max size. Say header image on a blog. Then a mobile size for small screens, and a thumbnail (100x100 etc). The post or articles table stores the paths in the applicable column.

As a side note, if your using separate hosting, that's why I upload to the server. Manipulate the image then upload. Less issues and network lag. Also, now you see why YouTube lost millions just hosting the billions of tarabits of videos :)

sl0wik's avatar

@jekinney so lets say you have 10k listings, each one on average 5 photos = 50k photos. Each photo need at least 2 thumbnails = 100k thumbs. Assuming average size of 100k per thumb we have 10GB to generate if we need one additional image size. If we add one zero somewhere (ex. our thumbnails are HQ jumbotrons) things will get nasty.

Why not: Point cdn to /thumbnail/ folder with max expiration date available by cdn, and generate them on fly + hit link first time on image upload? In such case old listing becoming "forgotten" and after 3 months it will take moment to generate links but at least we wont deal with huge cache.

Please or to participate in this conversation.