How to use service like smartproxy with my outgoing pooled requests?

Here is my current pooled request. $chunks is simply an associate array of urls with a url property (and other irrelevant data.)

The only $options I currently send with the request is verify=false;

$options['verify'] = false;

$responses = Http::pool(function ($pool) use ($chunk, $options) {
		foreach ($chunk as $chunkKey => $item) {
				$url = $item['url'];
				$pool->as("pool_{$chunkKey}")
						->withOptions($options)
						->retry(3, 1000) // Retry each individual request up to 3 times
						->get($url);
		}
});

I do not want to continue sending these request from my own personal IP address, and would like to learn how to be sending these requests from a proxied service like smartproxy.com or similar.

There documentation is not very thorough and simple gives this as the "PHP way of doing it."

<?php

    $url = 'ip.smartproxy.com/json';
    $proxy = 'gate.smartproxy.com';
    $port = 7000;
    $user = 'username';
    $psw = 'password';

    $ch = curl_init($url);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);

    curl_setopt($ch, CURLOPT_PROXY, "$proxy:$port");
    curl_setopt($ch, CURLOPT_PROXYUSERPWD, "$user:$psw");
    $result = curl_exec($ch);
    curl_close($ch);

    if ($result) {
    echo $result . PHP_EOL;
    }
?>

Any ideas on starting points? I feel like once I figure it out it will be simple, but for now, it's just simple enough to be impossible! 😄

LaryAI

1 year ago

Level 58

To use a proxy service like Smartproxy with your outgoing HTTP requests in Laravel, you can modify your existing code to include proxy settings in the request options. Here's how you can do it using Laravel's HTTP client:

Set Up Proxy Options: You need to include the proxy settings in the $options array. This includes the proxy URL, port, and authentication credentials.
Modify the Pool Request: Update your pool request to include these proxy settings.

Here's how you can modify your code:

$options['verify'] = false;

// Add proxy settings
$proxy = 'gate.smartproxy.com:7000';
$user = 'username';
$password = 'password';

$options['proxy'] = [
    'http'  => "http://$user:$password@$proxy",
    'https' => "http://$user:$password@$proxy",
];

$responses = Http::pool(function ($pool) use ($chunk, $options) {
    foreach ($chunk as $chunkKey => $item) {
        $url = $item['url'];
        $pool->as("pool_{$chunkKey}")
            ->withOptions($options)
            ->retry(3, 1000) // Retry each individual request up to 3 times
            ->get($url);
    }
});

Explanation:

Proxy Settings: The proxy option in the $options array is set to include both http and https protocols. The format is http://username:password@proxy:port.
Authentication: The proxy credentials (username and password) are included in the proxy URL.
Pool Request: The withOptions($options) method is used to apply these settings to each request in the pool.

This setup should route your HTTP requests through the Smartproxy service, using the specified proxy server and credentials. Make sure to replace 'username' and 'password' with your actual Smartproxy credentials.

JimNayzium

1 year ago

Level 2

So I am using OxyLabs instead of smartproxy, and this works:

   $proxy = 'pr.oxylabs.io:7777';
            $query = curl_init("{$chunk[0]['url']}");
            curl_setopt($query, CURLOPT_RETURNTRANSFER, 1);
            curl_setopt($query, CURLOPT_PROXY, "https://$proxy");
            curl_setopt($query, CURLOPT_PROXYUSERPWD, "customer-$username:$password");
            $output = curl_exec($query);
            curl_close($query);
        
            if ($output) {
                return $output;
            } else {
                return 'Failed to get response';
            }

It's fair to assume my username and password are correct as the above CURL works for certain.

But I cannot make the pool() work:

            $proxy    = 'pr.oxylabs.io:7777';

            $options['proxy'] = [
                'http' => "http://$username:$password@$proxy",
                'https' => "http://$username:$password@$proxy",
            ];

            $responses = Http::pool(function ($pool) use ($chunk, $options) {
                foreach ($chunk as $chunkKey => $item) {
                    $url = $item['url'];
                    $pool->as("pool_{$chunkKey}")
                        ->withOptions($options)
                        ->retry(3, 1000) // Retry each individual request up to 3 times
                        ->get($url);
                }
            });

What am I missing? I keep getting a status code of 407 indicating that the user cannot be authenticated at OxyLabs but using the curl It is authenticated.

Please or to participate in this conversation.