Luke, your videos are awesome! I appreciate all the obvious hard work you put into them!

I have a specific question about this ->retry() method and how to best return the failed attempts.

For my particular use case, I am importing data from an external API, taking that data and inserting it into my own database, and doing this regularly with a scheduler in the console.php. That being said, I have to log the errors and re-attempt the import if I experience errors. (Versus just returning a friendly well formatted javascript error from my request.)

In addition to that, the nature of the data I have to import requires sometimes to call dozens of external api end-points in the pool, so I have "chunked" those as well.

So I have a function that handles it all currently shown here, with my own retry technique cumbersomely coded into it. I feel like this little tidbit I've learned with ->retry(3) will solve this very problem I have, but I am wondering if all it does is remove my WHILE call in this longer cumbersome function, or if there is an inherently better way to handle the ACTUAL failed requests than what I am doing.

Currently, I am gathering all the failed responses into an array and returning ONLY those endpoints which failed after 3 attempts. Then in the calling function I am checking the count of the returned array and if > 0 I run my fetchWithPool again on ONLY those failed URLS this time.

** I added this ->retry(3) and commented it out, but curious if this is where it would go, and if so, would it make the retry counter and the while loop uneccessary at that point?

What happens after the 3rd try if the response is a 500 error with DOCTYPE etc... will show up as a $response->failed() in my function?

I am open to being told any better way to do this, as I am a hobbyist novice coder.

Below is my function that currently does all this. Any advice would be greatly appreciated.

PS - The number 19 has special meaning in my personal life, so I was curious if it did for you as well?

public static function fetchWithPool(array $urls, $headers = null)
    {
        $chunkSize = 8; // Reduce the chunk size to avoid overwhelming the server
        $projUrlChunks = array_chunk($urls, $chunkSize);
        $returnData = [];
        $delayBetweenRequests = 500; // Delay in milliseconds
        $maxRetries = 3; // Maximum number of retries for failed requests

        $options = $headers ?? [];
        $options['verify'] = false;

        foreach ($projUrlChunks as $chunk) {
            $responses = [];
            $retryCount = 0;

            while ($retryCount < $maxRetries) {

                $responses = Http::pool(function ($pool) use ($chunk, $options) {
                    
                    foreach ($chunk as $chunkKey=>$item) {
                    
                        $url = $item['url'];
                        $pool->withOptions($options)->get($url);

                        // $pool->as("pool_{$chunkKey}")
                        //         ->withOptions($options)
                        //         ->retry(3)
                        //         ->get($url);
                    
                    }
                });

                // Check for failed requests
                $failedRequests = [];

                foreach ($responses as $index => $response) {

                    if ($response instanceof Response && method_exists($response, 'failed')) {
                        if ($response->failed()) {
                            // Handle the failed response
                            Log::error('Request failed', [
                                'url' => $response->effectiveUri(),
                                'status' => $response->status(),
                                'body' => $response->body(),
                            ]);
                            $failedRequests[] = $chunk[$index];

                        } else {
                            // Process successful responses culled immediately
                            $data = $response->json();
                            $responseData = ['data' => $data];

                            // Include all additional properties from the original item in the $urls array
                            // [ 'url' => 'https://www.example.com', 'key' => 'value', 'key2' => 'value2' ]
                            // done this way to accomodate the different ways the $urls array can be structured
                            foreach ($chunk[$index] as $key => $value) {
                                if ($key !== 'url') {
                                    $responseData[$key] = $value;
                                }
                            }

                            $returnData[] = $responseData;
                        }

                    } else {
                        // Handle the case where $response is null or not an object
                        // You can log an error or take other appropriate actions
                        $msg = "The responses object was not formed correctly, and this process
                                just needs to be re-run all together to try again. (refresh it.)";

                        Log::error('Request failed', [
                            'url' => null,
                            'status' => 500,
                            'body' => $msg,
                        ]);
                        
                        return $msg;
                    }
                }

                if (empty($failedRequests)) {
                    $chunk = $failedRequests;
                    break; // Exit the retry loop if there are no failed requests
                }

                // Prepare for the next retry
                $chunk = $failedRequests;
                $retryCount++;
                usleep($delayBetweenRequests * 1000); // Convert milliseconds to microseconds
            }

            // all that should be left in $chunk is $failed requests
            // Handle failed responses that exhausted all retries
            foreach ($chunk as $item) {
                $responseData = ['data' => null, 'error' => "Request failed after $maxRetries retries"];
                // Include all additional properties from the original item
                foreach ($item as $key => $value) {
                    if ($key !== 'url') {
                        $responseData[$key] = $value;
                    }
                }
                $returnData[] = $responseData;
            }
        }

        return $returnData;
    }

Retrying Asynchronous Requests