mahiarirani's avatar

Http requests being dropped when being executed on queues operated by Horizon

I am building an Laravel 11 application using Sail and Octane with FrankenPHP. My setup includes Horizon to run workers that process jobs in the queue. One of these jobs involves making an HTTP request to an external API, which takes approximately 3–4 minutes to respond.

Despite adjusting multiple timeout configurations, the HTTP request in the queued job only fails silently when it exceeds 90 seconds. There are no exceptions, logs, or errors, even though the external API successfully responds within the expected timeframe. The job continues running until it hits its timeout and eventually fails.

Steps To Reproduce

Create a queued job that makes an HTTP request to an external API taking ~3–4 minutes to respond.
Configure the following timeouts:

max_execution_time in php.ini (CLI): 0 (unlimited).
REDIS_QUEUE_RETRY_AFTER: 320.
Job $timeout: 300.
HTTP client request and connection timeouts: 300.

Dispatch the job.
Observe behavior when the request takes:

Less than 90 seconds: The job processes successfully.
More than 90 seconds: The HTTP request is abandoned, and no response is logged or handled.
0 likes
1 reply
LaryAI's avatar
LaryAI
AI
1 year ago
Level 58

When dealing with long-running HTTP requests in a Laravel application using queues and Horizon, there are several factors to consider that might be causing the issue you're experiencing. Here are some steps and configurations to help resolve the problem:

  1. Check PHP and Server Timeouts:

    • Ensure that the max_execution_time in your php.ini is set to 0 for CLI, which you've already done. This setting should allow scripts to run indefinitely.
    • Verify that your web server (e.g., Nginx, Apache) does not have a timeout setting that could be affecting the request. For Nginx, check proxy_read_timeout and fastcgi_read_timeout. For Apache, check Timeout.

  2. Adjust Laravel and Horizon Configurations:

    • In your config/horizon.php, ensure that the timeout setting for your queue worker is set appropriately. This should be greater than the expected duration of your HTTP request.
    • In your job class, set the $timeout property to a value greater than the expected duration of the HTTP request. You've set it to 300, which should be sufficient.

  3. HTTP Client Configuration:

    • When making the HTTP request, ensure that the timeout settings are correctly configured. For example, using Laravel's HTTP client, you can set the timeout like this:

      use Illuminate\Support\Facades\Http;

$response = Http::timeout(300)->get('https://external-api.com/endpoint');

  4. Check for Silent Failures:

    • If the request is failing silently, consider adding logging or exception handling to capture any errors. You can use Laravel's logging facilities to log the response or any exceptions:

      try {
    $response = Http::timeout(300)->get('https://external-api.com/endpoint');
    // Log the response or handle it
    Log::info('API Response', ['response' => $response->body()]);
} catch (\Exception $e) {
    Log::error('HTTP Request Failed', ['error' => $e->getMessage()]);
}

  5. Redis and Horizon Configuration:

    • Ensure that the retry_after setting in your config/queue.php and the horizon.php configuration is set to a value greater than the job's $timeout. This prevents the job from being retried prematurely.

  6. Network and Firewall Considerations:

    • Check if there are any network or firewall rules that might be interrupting long-running connections.

  7. Debugging:

    • Use tools like tcpdump or Wireshark to monitor network traffic and see if the connection is being dropped at the network level.

By ensuring that all these configurations are correctly set, you should be able to handle long-running HTTP requests in your queued jobs without them being dropped. If the problem persists, consider breaking down the task into smaller parts or using a different approach to handle long-running processes.

Please or to participate in this conversation.