How to use Http pool in Laravel?

just saw your comment, something like this....

$responses = Http::pool(function(Pool $pool) use($urls) {
   foreach ($urls as $url) {
     $pool->head($url);
   }
});

@webrobert I just simplified your method a bit.

public function getEventAndInvitees(string $calendlyId)
{
    $eventUrl = "https://api.calendly.com/scheduled_events/{$calendlyId}";
    $inviteesUrl = "https://api.calendly.com/scheduled_events/{$calendlyId}/invitees?status=active";
    $token = config('services.calendly.token');

    $responses = Http::withToken($token)->pool([
        $eventUrl,
        $inviteesUrl
    ]);

    if ( ! $responses[0]->ok() || ! $responses[1]->ok() ) {
        abort('503', 'Houston we have a problem');
    }

    return $responses;
}

@tisuchi, oh nice! I didn't realize you could use pool that way.

@thebigk have you tried the other code example for get requests...

DB::table('internal_links')
  ->distinct('href')
  ->orderBy('id')
  ->chunk(2, function($urls) {

      $responses = Http::pool($urls);

      foreach($responses as $response) {
          dd($response->status());
      }

  });
return Command::SUCCESS;

I also wonder, ConnectException exception is thrown in the event of a networking error. do you get it straight away or is it after some number of requests? Maybe too many too fast?

@webrobert The exception is thrown if the URL is invalid or the domain doesn't exist. That's the reason I need to find out a way to handle the exceptions. It's not about the speed of requests.

The exception is thrown if the URL is invalid or the domain doesn't exist. That's the reason I need to find out a way to handle the exceptions.

@thebigk, how would we know?

You originally wrote

I've decided to make use of the concurrent requests feature offered by Laravel's Http client. I'm however, not able to put the code together. Would appreciate your help.

Not how do I handle the exceptions. 🤷🏽‍♂️

@webrobert - I tried running that code for the suspicious URLs. :-/

@webrobert Yep, already read that. However, this doesn't seem to work with concurrent requests.

Where do I handle it in the following code?

DB::table('internal_links')
  ->distinct('href')
  ->orderBy('id')
  ->chunk(2, function($urls) {

      $responses = Http::pool($urls);

      foreach($responses as $response) {
          dd($response->status()); // This throws error. 
      }

  });
return Command::SUCCESS;

@thebigk, I see. Okay, how about this...

$responses = Http::pool(function (\Illuminate\Http\Client\Pool $pool) use($urls) {
    foreach ($urls as $url) { $pool->head($url); }
});

$goodUrls = collect($responses)
    ->map( fn($response) => $response instanceof \Illuminate\Http\Client\Response
        ? (string) $response->effectiveUri()
        : null 
    )
    ->filter();


dd($goodUrls);

then you can just compare agains the collection and deal with bad urls..

EDIT:

Here is a bit of a refactor. I think this is my final pass for now...

$urls = collect([
    'http://google.com',
    'http://googsdfsdle.com',
    'http://yelp.com',
    'http://testwwerer.com',
]);

$responses = Http::pool(function (\Illuminate\Http\Client\Pool $pool) use($urls) {
    $urls->each( fn($url) => $pool->head($url) );
});

$goodKeys= collect($responses)
    ->map( fn($response) => 
        $response instanceof \Illuminate\Http\Client\Response 
        && $response->ok() 
    )
    ->filter();

$badUrls = $urls->diffKeys($goodKeys)->all();

@thebigk how did it go?

@thebigk ahh we’ll run my code. It’s tested and works. Plus it’s way shorter than that.

@webrobert - In this code, I think we aren't checking for valid URLs and URLs that don't have domains associated with them (dead urls).

From what I understand, it only checks for ok() responses. I'm also wondering - is there a possibility of adding rate limiting to this?

Also, where exactly would I make entry into the database for each URL?

$urls = collect([
    'http://google.com',
    'http://googsdfsdle.com',
    'http://yelp.com',
    'http://testwwerer.com',
]);

$responses = Http::pool(function (\Illuminate\Http\Client\Pool $pool) use($urls) {
    $urls->each( fn($url) => $pool->head($url) );
});

$goodKeys= collect($responses)
    ->map( fn($response) => 
        $response instanceof \Illuminate\Http\Client\Response 
        && $response->ok() 
    )
    ->filter();

$badUrls = $urls->diffKeys($goodKeys)->all();

@thebigk the issue was that when a bad url is entered head doesn’t throw the error. But it’s an exception is there instead of a response. So you can’t check if it’s ok on an exception. So all we do it check for a response instance first and if it has one then it was a valid url. Otherwise it was BS.

Then you can compare the keys to get the good or bad urls. And can Act on them.

What are you wanting to do next? Save urls or remove them from the database?

@webrobert I used your code as follows -

public function handle()
    {
        // Delete this after it does what it does.
        DB::table('internal_links')->distinct('href')->chunkById(10, function($urls) {
            $urls_collection = collect();
            foreach($urls as $url) {
                $urls_collection->push($url->href);
            }
            $urls = $urls_collection;
            $responses = Http::pool(function (\Illuminate\Http\Client\Pool $pool) use($urls) {
                $urls->each( fn($url) => $pool->head($url) );
            });

        $goodKeys= collect($responses)
            ->map( fn($response) =>
                $response instanceof \Illuminate\Http\Client\Response
                && $response->ok()
            )
            ->filter();

        dump($goodKeys);

        $badUrls = $urls->diffKeys($goodKeys)->all();
        dd($badUrls);
        });
        return Command::SUCCESS;
    }

This gives me goodKeys as [true, true...] and an array of the bad URLs. I think this should do the job. I just checked and I'm going to use it on ~50K external URLs (internals will be dealt with separately).

1. How can I speed this up without affecting performance?
2. My plan is to discard all the bad ULRs (anything that is not 200). But how do I get the URLs of the good responses so that I can update the database accordingly?

@thebigk

DB::table('internal_links')->distinct('href')->chunkById(10, function($urls) {

    $responses = Http::pool(function (\Illuminate\Http\Client\Pool $pool) use($urls) {
        $urls->each( fn($url) => $pool->head($url->href) );
    });

    $goodKeys= collect($responses)
        ->map( fn($response) =>
            $response instanceof \Illuminate\Http\Client\Response
            && $response->ok()
        )
        ->filter();

    // good internal_links
    $urls->intersectByKeys($goodKeys)->each(function ($url) {
        // do stuff
    });

    // bad internal_links
    $urls->diffKeys($goodKeys)->each(function ($url) {
        // do stuff
    });
});

return Command::SUCCESS;

0R

DB::table('internal_links')->distinct('href')->chunkById(10, function($urls) {

    $responses = Http::pool(function (\Illuminate\Http\Client\Pool $pool) use($urls) {
        $urls->each( fn($url) => $pool->head($url->href) );
    });

    $goodKeys = collect($responses)
        ->map( fn($response) =>
            $response instanceof \Illuminate\Http\Client\Response
            && $response->ok()
        );

    $urls->each(function ($url, $key) use($goodKeys) {
        $goodKeys[$key]
            ? dump("save {$url->href} its good")
            : dump("kill {$url->href} its bad");
    });
});