Be part of JetBrains PHPverse 2026 on June 9 – a free online event bringing PHP devs worldwide together.

haider885's avatar

Why laravel eloquent insert function missing values after 50k entries while using chunks?

I have an excel sheet that has more than 1 million rows in it. I want to insert and update these rows of data to a table. I have used "Maatwebsite" excel library and imported excel to a temporary table. After this, I want to insert temporary table data into the actual table.

Following is the code for fetching data from the temp table.

$temp_products = ProductTemp::limit(100000)->get()->map(function(ProductTemp $temp_product) {
        return [
            'msrp' => $temp_product->msrp,
            'price' => $temp_product->unit_price,
            'product_overview' => $temp_product->part_description,
            'manufacturer_id' => $temp_product->manufacturer_id,
            'upc' => $temp_product->upc,
            'manufacturer_part_number' => $temp_product->part_no
        ];
    });

Then I tried both the chunk() and array_chunk() functions to divide data into parts.

//$chunks = $temp_products->chunk(5000);
$arr_chunks = array_chunk($temp_products->toArray(), 5000);

Then I loop through them to insert rows in a actual table like this with the array_chunk() function.

foreach($arr_chunks as $chunk)
    {          
        Product::insert($chunk);
    }

and with chunk() function like this.

foreach($arr_chunks as $chunk)
        {
            Product::insert($chunk->toArray());           
        }

I have also tried the sleep() function with this but not helping. After 50, 000 entries it's missing values or adding more. Also, I tried chunks of 100, and 1000 but the result is the same and the page never finishes loading.

Here is the full function code, this might help with what I have tried more and what I am missing.

public function insertFromTempTable()
    {
        // dd("HERE 1111111 22222222222 4444444444");
        $temp_products = ProductTemp::all()->map(function (ProductTemp $temp_product) {
            return [
                'msrp' => $temp_product->msrp,
                'price' => $temp_product->unit_price,
                'product_overview' => $temp_product->part_description,
                'manufacturer_id' => $temp_product->manufacturer_id,
                'upc' => $temp_product->upc,
                'manufacturer_part_number' => $temp_product->part_no,
            ];
        });
        // dd($temp_products);
        $inc = 0;
        dump("Started on: " . Carbon::now()->toDateTimeString());
        // sleep(10);
        // foreach ($temp_products as $value) {
        //     Product::insert($temp_products[$inc]);
        //     $inc++;
        // }
        // dd($temp_products->toArray());
        // Product::insert($temp_products->toArray());
        // $chunks = $temp_products->chunk(100);
        $chunk_sub = 5000;
        $arr_chunks = array_chunk($temp_products->toArray(), $chunk_sub);
        // dd($arr_chunks[0]);
        // dd(gettype($chunks[0]->toArray()));
        // dd($chunks[0]->toArray());
        $chunk_temp = 0;
        foreach ($arr_chunks as $chunk) {
            // dd($chunk->toArray());
            // dump(count($chunk->toArray()));
            // Product::insert($chunk->toArray());
            $chunk_temp = $chunk_temp + $chunk_sub;
            
            Product::insert($chunk);
            
        }

        // $temp_products->chunk(2, function ($subset) {
        //     dd("HHHHHHH");
        //     $subset->each(function ($item) {
        //         dd($item);
        //     });
        // });

        dump("Ended on: " . Carbon::now()->toDateTimeString());
    }

After inserting I want to compare an excel sheet on daily basis, that might have more than 1 million rows. That will check for MSRP and price columns and update them. Please let me know which will be the best solution for all of this.

I have also tried using a raw query. There were 104, 8575 rows in the temp table but after the query run, there are 103, 9229 rows in the products table. Following is the code for the raw query.

$results = DB::table('products')->insertUsing(
        ['msrp', 'price', 'product_overview', 'manufacturer_id', 'upc', 'manufacturer_part_number',],
        function ($query) {
            $query
                ->select(['msrp', 'unit_price', 'part_description', 'manufacturer_id', 'upc', 'part_no',])
                ->from('products_temp');
        }
    );

When I dump and die the $results variable its output 104, 8575.

0 likes
2 replies
webrobert's avatar

on the forum, three backticks ` to open code, and three again to close.

1 like

Please or to participate in this conversation.