kinshara's avatar

Need to process 10k records in same time.

I have 20k records products data. and 4 customer groups, and each record of product data have 4 price based on customer groups.

and once single customer group is modified, the system need to recalculate 20k records(deleting old one, select base price, calculate the new price, and then insert new one) or may be just update.

if there are new products by import.. 9k products, it means 9k, looping each, and recalculate base on 4 groups and insert.

but the process is really slow, and sometimes my server(im using ubuntu, n nginx) going to 404 not found. is there any tips?

*i have already use bulk insert. storing them into array and insert at the end.

0 likes
9 replies
JackJones's avatar

20k records isn't very many at all, you'd need to provide the case and code

kinshara's avatar
$deleteProductPriceID   = [];
        $productPrice           = [];
        $newLog                 = [];

        foreach($products as $product) {
            $productID  = $product->productDetail->id;
            $price      = $product->productDetail->price;

            foreach($custGroups as $group) {
                array_push($deleteProductPriceID, $productID);

                $markup = $price + (($price * $group->markup_percentage) / 100);
                $markup = ceil($markup / 100) * 100;

                array_push($productPrice, [
                    'product_id'        => $productID,
                    'customer_group_id' => $group->user_group_id,
                    'price'             => $markup,
                    'updated_at'        => $dateNow,
                    'created_at'        => $dateNow,
                ]);

                array_push($newLog, [
                    'product_id'        => $productID,
                    'customer_group_id' => $group->user_group_id,
                    'price_preset_id'   => $pricePreset->id,
                    'product_price'     => $price,
                    'markup_percentage' => $group->markup_percentage,
                    'admin_id'          => $adminID,
                    'updated_at'        => $dateNow,
                    'created_at'        => $dateNow
                ]);
            }
            ProductPriceModel::whereIn('product_id', $deleteProductPriceID)->delete();
            ProductPriceModel::insert($productPrice);
            ProductPriceLogModel::insert($newLog);
        }

I use 2 loops,

  1. loop all products,
  2. loop all group inside each products
36864's avatar
foreach($products as $product) {
           $productID  = $product->productDetail->id;
           $price      = $product->productDetail->price;

Is productDetail a relationship? Did you eager load it?

Memn1993's avatar

Laravel commits after each action. So do only a commit after finnishing all. Example:

DB::beginTransaction(); //better performance
        foreach ($array as &$a) {
            //do something
            $entry->fill($a);
            $entry->save();
        }
        DB::commit(); //final commit
kinshara's avatar

Is productDetail a relationship? Did you eager load it?

nope for this code. but I have already change with to load at once by view. is it really effect much?

I would use a queue for calculations like this. They can run in the background and don't have a execution time limit like normal http requests have.

I need to give response directly after it done..

Laravel commits after each action. So do only a commit after finnishing all.

will it works will bulk insert?

and i do confuse, when I process > 10k, the respond will return 404.

but if <10k, it recalculated successfully.

Memn1993's avatar

Yes. it will works for bulk insert. I do this for 4k elements each time. You should commit only after complete a task. In case of an error you don't habe a corrupted dataset.

36864's avatar

Is productDetail a relationship? Did you eager load it?

nope for this code. but I have already change with to load at once by view. is >it really effect much?

If that relation isn't being eager loaded, it's being lazy loaded for each product. That's >10k database queries in your case. Yes, it is really effects much.

kinshara's avatar

Yes. it will works for bulk insert. I do this for 4k elements each time. You should commit only after complete a task. In case of an error you don't habe a corrupted dataset.

thank you :D. but it havent answered the main problem.. i cannot calculate all..

Please or to participate in this conversation.