Daily JSON import & update

Including a bugfix, some cleanup, and some refactoring, I would do it as follow:

app/Console/Commands/GetFeeds.php

<?php
namespace App\Console\Commands;

use Illuminate\Console\Command;
// Dont forget to import your Webshop and Product models here, I do not know where they are in your project.
use App\Webshop;
use App\Product;

class GetFeeds extends Command
{
    protected $signature = 'webshops:get-feed';

    protected $description = 'Import the feeds for all webshops';

    
    public function __construct()
    {
        parent::__construct();
    }

    
    public function handle()
    {
        $webshops = Webshop::all();
    
        foreach ($webshops as $webshop) {
            $client = new \GuzzleHttp\Client();
            $url = $webshop->url;
            $response = $client->request('GET', $url);
            $feed = json_decode($response->getBody()->getContents(), true);
            $products = [];
            foreach($feed['datafeed']['programs'][0]['products'] as $product) {
                $product = Product::updateOrCreate(
                    ['sku' => $product['product_info']['sku']],
                    [
                        'title' => $product['product_info']['title'],
                        'price' => $product['product_info']['price'],
                        'url' => $product['product_info']['url'],
                        'webshop_id' => $webshop->id
                    ]
                );
                $products[] = $product->id;
            }
            // delete any product for this webshop that is not in the current feed
            Product::where('webshop_id', $webshop->id)->whereNotIn('id', $products)->delete();
        }
    }
}

This console command can be run from the scheduler, or via "php artisan webhops:get-feed"

Included some cleanup (the temp variables for Product::updateOrCreate where unnecessary) and a small bugfix: sku => category_id, I believe this should have been the SKU ?

On every updateOrCreate it will save the product's ID to an array so after the webshop import is done, we can delete every product for this webshop that is not in this list.

(change in your DB schema needed: I did not see a webshop_id on the product model, but it really should be in there if you want to be able to only delete products for a single webshop)

In your app/Console/Kernel.php:

<?php

namespace App\Console;

use App\Console\Commands\GetFeeds;
use Illuminate\Console\Scheduling\Schedule;
use Illuminate\Foundation\Console\Kernel as ConsoleKernel;

class Kernel extends ConsoleKernel
{
    protected $commands = [
        GetFeeds::class
    ];

    protected function schedule(Schedule $schedule)
    {
        $schedule->command(GetFeeds::class)->dailyAt('01:00');
    }

    protected function commands()
    {
        $this->load(__DIR__.'/Commands');

        require base_path('routes/console.php');
    }
}

This keeps your Kernel.php clean, and will still run your schedule every day at 01:00 AM

Thank you! This was exactly what I've been looking for. Was figuring out how to get the same process in a command.

But now, what if I have 100 JSON feeds (webshops) with each of them having 100+ products, wouldn't this be a really intensive task?

Are there ways to speed this up or distribute the importing of the feeds?

Yes, you could throw them onto a queue, and have several queue workers. So several could be processed simultaneously in the background.

https://laravel.com/docs/5.6/queues

doing 100 http requests and ~ 10.000 insert/updates is not something I would consider a "really intensive task" or would split up even further.

Actually, as it is now: the flow is:

1 http request ~ 100 inserts

1 http request ~ 100 inserts

etc. So the task is actually waiting more often then that it's actually doing something.

If you're concerned that the task will take too long this way, you could do it another way to speed it up (but will use more resources on the server):

GetFeeds.php:

<?php
namespace App\Console\Commands;

use Illuminate\Console\Command;
use App\Webshop;

class GetFeeds extends Command
{
    protected $signature = 'webshops:get-feed';

    protected $description = 'Import the feeds for all webshops';

    
    public function __construct()
    {
        parent::__construct();
    }

    
    public function handle()
    {
        $webshops = Webshop::all();
    
        foreach ($webshops as $webshop) {
            Artisan::queue('webshops:get-feed', [
                'webshopId' => $webshop->id, '--queue' => 'default'
            ]);
        }
    }
}

GetFeed.php:

namespace App\Console\Commands;

use Illuminate\Console\Command;

use App\Webshop;
use App\Product;

class GetFeed extends Command
{
    protected $signature = 'webshop:get-feed {webshopId}';

    protected $description = 'Import the feeds for all webshops';

    
    public function __construct()
    {
        parent::__construct();
    }

    
    public function handle()
    {
        $webshop = Webshop::find( $this->argument('webshop') );
    
        $client = new \GuzzleHttp\Client();
        $url = $webshop->url;
        $response = $client->request('GET', $url);
        $feed = json_decode($response->getBody()->getContents(), true);
        $products = [];
        foreach($feed['datafeed']['programs'][0]['products'] as $product) {
            $product = Product::updateOrCreate(
                ['sku' => $product['product_info']['sku']],
                [
                    'title' => $product['product_info']['title'],
                    'price' => $product['product_info']['price'],
                    'url' => $product['product_info']['url'],
                    'webshop_id' => $webshop->id
                ]
            );
             $products[] = $product->id;
         }
        // delete any product for this webshop that is not in the current feed
        Product::where('webshop_id', $webshop->id)->whereNotIn('id', $products)->delete();
    }
}

Now, if you have 100 webshops, and have your Queue running 8 workers, you will start the first command 'GetFeeds' which will start 100 different jobs on your queue (1 for each webshop) And because of the 8 workers, you'll be doing 8 HTTP requests at the same time, inserting their 100 products.

It would be approximately 5-6x faster, but will use a bit more resources on both the php and the MySQL part.