Studiomate's avatar

Recreate the cache with a job

I'm developing an ecommerce with thousands of products. The show page of the products is cached in this way

$showProduct = Cache::remember('product_show_' . $product->id, 60 * 60 * 24, function () use ($product) {

 return [
			"id" => $product->id,
			"title" => $product->title,
			"content" => $product->content,
			"excerpt" => $product->excerpt,
			....
 ];

});

When the cache is cleaned or expired the first user to visit the single product page gets a certain delay that i would like to remove.

To achieve this my idea is to use a Job that every 12 hours restores the cache.

    public function handle()
    {
        Product::where('status', 'publish')->chunk(15, function($products){
            
            // Iterate over the products
            foreach($products as $product){

				(new productController)->getProduct($product);

            }
        });

Unfortunately, if I run this job my script uses all the memory available to PHP ( 2GB ). My understanding is that this is because the scheduled JOB is using the same process for all the cache creations. How to start different processes?

0 likes
6 replies
jivanrij's avatar

Tinker with the amount of products that are being cached in one run and the settings of the process that runs your jobs. Spread it out over a longer period, and start with the populair products.

You can also limit the use of memory the Eloquent models are using by limiting the fields that are retrieved for the models before pushing them into the Cache::remember(). https://laravel.com/docs/11.x/eloquent-relationships#relationship-counting-and-custom-select-statements

If the Products also load relations you can limit those fields like this:

Post::with('user:id,name')->get();

With getBase you can load the data but skip the whole Eloquent instance, this also reduces the memory usage. It's like retrieving stdClasses. If you don't need all the Eloquent logic to create the cache, this can help you out.

Post::select(['id', 'title'])->toBase()->get();
1 like
martinbean's avatar

@studiomate Surely the time to cache a single product isn’t that noticeable? What are you doing that would cause such a delay in retrieving a product from a database and caching it?

Regardless, when it comes to caching, you’re better off caching when a change actually occurs, rather than letting an unlucky visitor be the one to trigger the caching logic. And if a product is to remain in the cache until it’s changed, then cache it “forever” rather than just one day:

$key = sprintf('products.%d', $id);

$product = Cache::rememberForever($key, function () {
    // Look up and return product model...
});

If all products need caching, then consider doing that as part of your deployment script, to build the cache on deploy, after you’ve ran migrations etc.

LaryAI's avatar
Level 58

To address the issue of high memory usage when recreating the cache for thousands of products, you can utilize Laravel's job dispatching system to handle each product cache creation in a separate process. This can be achieved by dispatching individual jobs for each product, which will allow Laravel to manage them independently and avoid memory exhaustion.

Here's a step-by-step solution:

  1. Create a Job for Caching a Single Product:

    First, create a new job that will handle caching for a single product. You can use the php artisan make:job CacheProduct command to generate a job class.

  2. Dispatch Jobs for Each Product:

    Modify your existing job to dispatch a CacheProduct job for each product. This will ensure that each product's cache is created in a separate process.

    namespace App\Jobs;
    
    use Illuminate\Bus\Queueable;
    use Illuminate\Contracts\Queue\ShouldQueue;
    use Illuminate\Foundation\Bus\Dispatchable;
    use Illuminate\Queue\InteractsWithQueue;
    use Illuminate\Queue\SerializesModels;
    use App\Models\Product;
    use App\Jobs\CacheProduct;
    
    class CacheAllProducts implements ShouldQueue
    {
        use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;
    
        public function handle()
        {
            Product::where('status', 'publish')->chunk(15, function ($products) {
                foreach ($products as $product) {
                    CacheProduct::dispatch($product->id);
                }
            });
        }
    }
    
  3. Configure Queue Worker:

    Ensure that your queue worker is running to process these jobs. You can start a queue worker using the following command:

    php artisan queue:work
    

    You might also want to configure a supervisor to manage your queue workers in a production environment.

  4. Schedule the Job:

    Finally, schedule the CacheAllProducts job to run every 12 hours in your App\Console\Kernel.php file:

    protected function schedule(Schedule $schedule)
    {
        $schedule->job(new \App\Jobs\CacheAllProducts)->everyTwelveHours();
    }
    

By dispatching individual jobs for each product, you can significantly reduce memory usage and ensure that the cache is rebuilt efficiently without impacting the user experience.

Please or to participate in this conversation.