ssquare
11 months ago

Help to implement logic by detecting end of file and loop out while parsing csv and dumping to database

Posted 11 months ago by ssquare

I am using CSVLeague to parse csv file and then later dumping those data to the database. The structure looks like.

$loop = true;

            while($loop){

                    $stmt = (new Statement())
                    ->offset($offset)
                    ->limit($limit)
                    ;

                    $records = $stmt->process($csv);

                    foreach ($records as $record) 
                    {
                        $rec_arr[] = array_values($record);  
                    }

                    $records_arr = $service->trimArray($rec_arr);  //removes if any empty row 
                                                    //encounters 

                    if(count($records_arr)>0){
                foreach($records_arr as $rk => $rv){
                    //here map csv column to database column and insert into database
                }
                $offset = $offset + $limit; 
            }else{
                $loop = false;  
            }

Currently, I am implementing this logic inside laravel queue. It is successfully inserting the whole set of data but it is not halting the process. It keeps stuck with message processing. However, if I removed that while loop then it will be stopped with message processed. So, I think it should be something that I am implementing some bad logic there.

Looking for an idea to tackle with this.

Update:

In case if you are curious, how I am reading csv.

     $csv = Reader::createFromPath($csv_file_path, 'r');

    $csv->setOutputBOM(Reader::BOM_UTF8);
        $csv->addStreamFilter('convert.iconv.ISO-8859-15/UTF-8');

         $csv->setHeaderOffset(0); 
         $csv_header = $csv->getHeader();

Please sign in or create an account to participate in this conversation.