plutarcomp
1 year ago

Multiple tasks simultaneously

Posted 1 year ago by plutarcomp

I need to process several image files from a directory, the process is to read the filename (id and type) that is stored in the directory in the moment is invoked the process (im using cron and schedule, it works great) the objetive of the process is to store the info into a database.

I have the process working, it works great but my problem is the number of files that is in the directory, because every second adds a lot more files to the directory, the time spent in the process is about 0.19 sec for file, but the amount of files is huge, about 15,000 per minute is added, so i think a multiple simultaneous process (about 10 - 40 times) of the same original process can do the job.

I need some advice or idea,

First to know how to launch multiple process at the same time of one original process.

Second how to get only the non selected filenames bcause the process takes the filenames with:

      $recibidos = Storage::disk('s3recibidos');

      if(count($recibidos) <= 0)
      {
        $lognofile = ['Archivos' => 'No hay archivos para procesar'];
        $orderLog->info('ImagesLog', $lognofile);
      }else{
        $files = $recibidos->files();
    
        if(Image::count() == 0)
        {
          $last_record = 1;
        } else{
            $last_record = Image::latest('id')->pluck('id')->first()+1;
        }
        $i=$last_record;
        $fotos_sin_info = 0;
        foreach($files as $file)
        {
          $datos = explode('_',$file);
          $tipos = str_replace('-','',$datos[2]);
          Image::create([
            'client_id' => $datos[0],
            'tipo' => $tipos,
          ]);
          $recibidos->move($file,'/procesar/'.$i.'.jpg');
          $i++;
        }

but i dont figured out how to retrieve only the non selected.

Thanks for your comments.

Please sign in or create an account to participate in this conversation.