Loop breaking

Level 104

Hard to say; is there an error/exception, or is it failing silently? Are any Model being created?

Are you sure all of the keys you are using to index into each $line exists? You are taking no steps to mitigate for missing/incorrect keys.

Level 2

@tykus yes 'cause this same script is running in other sistema and its working fine

Level 104

@carcleo so, it is failing silently without error/exception?

Sinnbeck

Level 102

@carcleo Then what error are you getting? If it never reaches the end, it must be throwing an error

Level 2

@tykus other things, if i create only the arra in the loop

$lines = [];
foreach ($reader as $line) {
  $lines[] = $line;
}

its arrives fine with all data of reader file

its say to me hat the problem is at the insertions

Level 2

@Sinnbeck then, its not show errors, neither on aba network.

But it go recording the data into database until break (stop)

Level 104

@carcleo okay... and???

What is the outcome of running the method; again, is there an error/exception? If there is nothing in the browsere/CLI, did you check your logs for any hints?

https://photos.google.com/album/AF1QipO9qxEceGKZ7DhYNoGirRXRI27hp2gwz5ZmI2om/photo/AF1QipOFNGCIfotfyprbcUQZJqEEzJUbaOKifojoZvcy

Level 2

@tykus POST https://acl.com/updated/students 500 (Internal Server Error)

it run local, not internet (VirtualHost Apache)

look it:

i clean the apche log and next that run it, this is the out of log

Apache Log:

[Thu Feb 06 08:50:31.132025 2025] [php:error] [pid 1168:tid 1044] [client 127.0.0.1:52086] PHP Fatal error: Maximum execution time of 30 seconds exceeded in D:\web\php\acl.com\vendor\laravel\framework\src\Illuminate\Hashing\BcryptHasher.php on line 49, referer: https://acl.com/atualiza/alunos

I think that need group the inserts and to do a upser with it. But i dont know how to do once that, for exemple, the Student depends of user_id

Level 104

@carcleo finally! There's your issue.

Maximum execution time of 30 seconds exceeded

Grouping inserts will not solve the problem. Push this work to a background job using a queue.

Level 2

@tykus How? I don know how to do it!

Level 104

@carcleo there are plenty of resources for learning and understanding queues, e.g.

Mahmoud04

Level 2

Hi @carcleo your code need some optimize try this and tell me if work

public function iterator($reader): ?array
{
    $domain = "text.com";
    $sdc = [];
    
    $existingUsers = $this->getExistingUsers($reader);
    $existingDisciplines = $this->getExistingDisciplines($reader);
    $existingClassrooms = $this->getExistingClassrooms($reader);
    
    $this->insertUsers($reader, $existingUsers, $domain);
    $existingUsers = $this->getExistingUsers($reader);

    $this->insertStudents($existingUsers);
    $this->insertDisciplines($reader, $existingDisciplines);
    $existingDisciplines = $this->getExistingDisciplines($reader);

    $this->insertClassrooms($reader, $existingClassrooms);
    $existingClassrooms = $this->getExistingClassrooms($reader);
    
    foreach ($reader as $line) {
        $ra = $line['RA'];
        $sdc[$ra]["student_id"] = $existingUsers[$ra] ?? null;
        $sdc[$ra]["discipline_id"][] = $existingDisciplines[$line['Disciplina']] ?? null;
        $sdc[$ra]["classroom_id"][] = $existingClassrooms[$line['Cód. Turma']] ?? null;
    }

    return $sdc;
}

private function getExistingUsers($reader): array
{
    return User::whereIn('registration', array_column($reader, 'RA'))->pluck('id', 'registration')->toArray();
}

private function getExistingDisciplines($reader): array
{
    return Discipline::whereIn('name', array_column($reader, 'Disciplina'))->pluck('id', 'name')->toArray();
}

private function getExistingClassrooms($reader): array
{
    return Classroom::whereIn('code', array_column($reader, 'Cód. Turma'))->pluck('id', 'code')->toArray();
}

private function insertUsers($reader, $existingUsers, $domain)
{
    $usersData = [];

    foreach ($reader as $line) {
        $ra = $line['RA'];
        $name = trim($line['Nome Aluno']);

        if (!empty($name) && !isset($existingUsers[$ra])) {
            $usersData[] = [
                'name' => $name,
                'registration' => $ra,
                'email' => 'ra' . $ra . '@' . $domain,
                'password' => Hash::make('1234'),
                'created_at' => now(),
                'updated_at' => now(),
            ];
        }
    }

    if (!empty($usersData)) {
        User::insert($usersData);
    }
}

private function insertStudents($existingUsers)
{
    $studentsData = [];

    foreach ($existingUsers as $ra => $userId) {
        $studentsData[] = [
            'user_id' => $userId,
            'created_at' => now(),
            'updated_at' => now(),
        ];
    }

    if (!empty($studentsData)) {
        Student::insert($studentsData);
    }
}

private function insertDisciplines($reader, $existingDisciplines)
{
    $disciplinesData = [];

    foreach ($reader as $line) {
        if (!isset($existingDisciplines[$line['Disciplina']])) {
            $disciplinesData[] = [
                "name" => $line['Disciplina'],
                'teacher_id' => 1,
                'created_at' => now(),
                'updated_at' => now(),
            ];
        }
    }

    if (!empty($disciplinesData)) {
        Discipline::insert($disciplinesData);
    }
}

private function insertClassrooms($reader, $existingClassrooms)
{
    $classroomsData = [];

    foreach ($reader as $line) {
        if (!isset($existingClassrooms[$line['Cód. Turma']])) {
            $classroomsData[] = [
                "code" => $line['Cód. Turma'],
                'created_at' => now(),
                'updated_at' => now(),
            ];
        }
    }

    if (!empty($classroomsData)) {
        Classroom::insert($classroomsData);
    }
}

Level 2

@malmalah0 Not works, $reader is not a array, its a xls file but $line, (that is in $reader), it yes is a array. them, getExistingUsers() not will works fine

And this way that you do not works because the final array will have repetitions of disciplines and classrooms for each student to fill the array student_discipline_classroom

Level 104

@carcleo your problem appears to stem from the amount of data you are trying to process; moving this task to a background job will give you more time to process the job (and without blocking the Response to the user).

Level 2

@tykus Not works

public function iterator ($reader) : ?array {

    $domain = "univassouras.edu.br";

    $sdc = [];
    $ra = "";

    $users = [];
    $students = [];
    $disciplines = [];
    $classrooms = [];

    foreach ($reader as $line) {

        $name = trim(rtrim((ltrim($line['Nome Aluno']))));

        $ra = $line['RA']; 

        if(!empty($name)) {   
            $users[] = [
                'registration' => $ra,
                'email' => 'ra'.$ra.'@'.$domain,
                'name' => $line['Nome Aluno'],
                'password' => Hash::make('1234')
            ];
        }
        
        $disciplines[] = [
            "name" => $line['Disciplina'],
            'teacher_id' => 1
        ];

        $classrooms[] = [
            "code" => $line['Cód. Turma']
        ];
    }
    //It doesn't get here, it runs out of time before
    //dd($users, $students);
    
    $usersChunk  = array_chunk($users, 500);
    foreach ($usersChunk as $user) {
        $user = User::firstOrCreate($user);
        $student = Student::firstOrCreate([
            "user_id" => $user->id
        ]);                    
        $sdc[$ra]["student_id"] =$student->id;

    }

    $disciplinesChuck  = array_chunk($disciplines, 500);
    foreach ($disciplinesChuck as $discipline) {
        $discipline = Discipline::firstOrCreate($discipline);
        $sdc[$ra]["discipline_id"][] = $discipline->id;
    }

    $classroomsChunk  = array_chunk($users, 500);
    foreach ($classroomsChunk as $classroom) {
        $classroom = Classroom::firstOrCreate($classroom);         
        $sdc[$ra]["classroom_id"][] = $classroom->id;   
    }
    
    return $sdc;
    
}

Level 104

@carcleo "not works" in what context? I appreciate that English might not be your first language, but you need to try to explain what you have done differently?

Level 2

@tykus i out of foreach the called at database and do serious arrays.

next, i do array_chunk in this but the program, break inside foreach

Level 104

@carcleo are you using a Queue; if yes, which queue driver is configured?

What is happening before this method; is there another long-running operation, e.g. a Request out to a third-party API?