How Can I Handle Large Imports Async

Hello to everyone.

I have this scenario. In my app the user can create promotions for the items. There are 3 Models, Item, Promotion and PromotionLine with the following relations

class Promotion
{
	public function lines()
	{
		return $this->hasMany(PromotionLine::class);
	}
}

class PromotionLine()
{
	public function promotion()
	{
		return $this->belongsTo(Promotion::class);
	}

	public function item()
	{
		return $this->belongsTo(Item::class);
	}
}

I want to give the user the ability to upload lines in the promotion, using an excel file. I have created a livewire component for this that uses the toCollection function from the laravel excel package (https://docs.laravel-excel.com/3.1/imports/collection.html).

public function importLines()
{
	$import_file_path = $this->import_file->store('/', 'tmp');
    $import_file_fullpath = Storage::disk('tmp')->path($import_file_path);
    
    $lines = [];
    $excel_lines = Excel::toCollection(new LinesImport(), $import_file_fullpath)->first()->all();

	foreach ($excel_lines as $index => $line_data)
	{
		//import line here
	}
}

This is working fine if I have a few rows in the excel, but in case the user adds many rows the whole page crashes (obviously).

What I would like to do is, queue the import and while it is working show the user a loader and a message. When the import finishes, I will show him the errors and close the modal.

My first thought would be to put the import in an Artisan command and show him the console output. Is this possible?

Another thought is to use websockets and determine if the specific job is finished or running. Is this possible?

Can you help me get started with these?

Do you have any other ideas on how to handle this?

achatzi

3 years ago

Best Answer

Level 5

I decided to implement websockets for this, since I would be using it anyway.

So I created 2 new events, ImportChunkCompleted and ImportCompleted that I broadcast

class PromotionLinesImport
{
    public function registerEvents(): array
    {
        $events = $this->parentRegisterEvents();

        $events[AfterSheet::class] = function (AfterSheet $event) {
            $rows = cache("imports:{$this->uuid}:rows", 0);
            $rows += $this->imported_rows;
            cache()->put("imports:{$this->uuid}:rows", $rows, now()->addHours(2));

            event(new ImportChunkCompleted($this->importer, get_class($this), $this->uuid, $rows));
        };

        $events[AfterImport::class] = fn(AfterImport $event) => event(new ImportCompleted($this->importer, get_class($this), $this->uuid));

        return $events;
    }
}

Now, it shows to the user the number of rows each chunk has imported and closes the modal when it is completed.

Please or to participate in this conversation.