Queue Stops Without Exception

Hello Everyone,

I am implementing orders export with the help of Laravel queue system. When the user clicks export button, I am dispatching a ExportJob, which retrieves the data from the external API and store it in the database.

While I run in my local, the export just runs fine(running with php artisan serve and php artisan queue:work). But while I configure the same thing in production, the queue doesn't throw any exception, but I can see the logs stops printing. I have put logs on each order in for loop. So it does stops interminently.

Here is my config/queue.php

<?php

return [

    /*
    |--------------------------------------------------------------------------
    | Default Queue Connection Name
    |--------------------------------------------------------------------------
    |
    | Laravel's queue supports a variety of backends via a single, unified
    | API, giving you convenient access to each backend using identical
    | syntax for each. The default queue connection is defined below.
    |
    */

    'default' => env('QUEUE_CONNECTION', 'database'),

    /*
    |--------------------------------------------------------------------------
    | Queue Connections
    |--------------------------------------------------------------------------
    |
    | Here you may configure the connection options for every queue backend
    | used by your application. An example configuration is provided for
    | each backend supported by Laravel. You're also free to add more.
    |
    | Drivers: "sync", "database", "beanstalkd", "sqs", "redis", "null"
    |
    */

    'connections' => [

        'sync' => [
            'driver' => 'sync',
        ],

        'database' => [
            'driver' => 'database',
            'connection' => env('DB_QUEUE_CONNECTION'),
            'table' => env('DB_QUEUE_TABLE', 'jobs'),
            'queue' => env('DB_QUEUE', 'default'),
            'retry_after' => (int) env('DB_QUEUE_RETRY_AFTER', 90),
            'after_commit' => false,
            'timeout' => 3600
        ],

        'beanstalkd' => [
            'driver' => 'beanstalkd',
            'host' => env('BEANSTALKD_QUEUE_HOST', 'localhost'),
            'queue' => env('BEANSTALKD_QUEUE', 'default'),
            'retry_after' => (int) env('BEANSTALKD_QUEUE_RETRY_AFTER', 90),
            'block_for' => 0,
            'after_commit' => false,
        ],

        'sqs' => [
            'driver' => 'sqs',
            'key' => env('AWS_ACCESS_KEY_ID'),
            'secret' => env('AWS_SECRET_ACCESS_KEY'),
            'prefix' => env('SQS_PREFIX', 'https://sqs.us-east-1.amazonaws.com/your-account-id'),
            'queue' => env('SQS_QUEUE', 'default'),
            'suffix' => env('SQS_SUFFIX'),
            'region' => env('AWS_DEFAULT_REGION', 'us-east-1'),
            'after_commit' => false,
        ],

        'redis' => [
            'driver' => 'redis',
            'connection' => env('REDIS_QUEUE_CONNECTION', 'default'),
            'queue' => env('REDIS_QUEUE', 'default'),
            'retry_after' => (int) env('REDIS_QUEUE_RETRY_AFTER', 90),
            'block_for' => null,
            'after_commit' => false,
        ],

    ],

    /*
    |--------------------------------------------------------------------------
    | Job Batching
    |--------------------------------------------------------------------------
    |
    | The following options configure the database and table that store job
    | batching information. These options can be updated to any database
    | connection and table which has been defined by your application.
    |
    */

    'batching' => [
        'database' => env('DB_CONNECTION', 'sqlite'),
        'table' => 'job_batches',
    ],

    /*
    |--------------------------------------------------------------------------
    | Failed Queue Jobs
    |--------------------------------------------------------------------------
    |
    | These options configure the behavior of failed queue job logging so you
    | can control how and where failed jobs are stored. Laravel ships with
    | support for storing failed jobs in a simple file or in a database.
    |
    | Supported drivers: "database-uuids", "dynamodb", "file", "null"
    |
    */

    'failed' => [
        'driver' => env('QUEUE_FAILED_DRIVER', 'database-uuids'),
        'database' => env('DB_CONNECTION', 'sqlite'),
        'table' => 'failed_jobs',
    ],

];

And supervisor configuration file.

[program:orders-queue]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/exporter/artisan queue:work --queue=default --sleep=3 --tries=0 --max-time=0
autostart=true
autorestart=true
stopasgroup=true
killasgroup=true
user=www-data
numprocs=1
redirect_stderr=true
stdout_logfile=/var/www/exporter/laravel-queue.log

Can anyone suggest why this happens and if anyone has any ways to resolve this? Thanks in advance.

Regards, Bala

LaryAI

1 year ago

Level 58

When dealing with Laravel queues that stop without throwing exceptions, there are several potential issues to consider. Here are some steps and suggestions to help diagnose and resolve the problem:

Check Supervisor Logs: Ensure that the Supervisor logs are being written correctly. Sometimes, the issue might be with Supervisor itself. Check /var/log/supervisor/supervisord.log for any errors or warnings.
Increase Timeout and Retry Settings: You might be hitting a timeout or retry limit. In your queue.php configuration, ensure that the retry_after and timeout settings are appropriate for the job's expected execution time. Consider increasing these values if necessary.
Database Connection: Ensure that the database connection used by the queue is stable and not timing out. You can test this by running a simple query in a loop to see if the connection drops.
Memory Limit: Check if the job is consuming too much memory. You can increase the PHP memory limit in your php.ini or by setting memory_limit in your Supervisor command:
```
command=php -d memory_limit=512M /var/www/exporter/artisan queue:work --queue=default --sleep=3 --tries=0 --max-time=0
```
Logging: Add more detailed logging within your job to pinpoint where it stops. This can help identify if it's a specific part of the job that's causing the issue.
Check for Long-Running Jobs: If your job is long-running, consider breaking it into smaller jobs or using Laravel's job batching feature to handle parts of the task separately.
Environment Differences: Ensure that the production environment matches your local environment in terms of PHP version, Laravel version, and any other dependencies.
Supervisor Configuration: Double-check your Supervisor configuration. Ensure that the command path is correct and that the user has the necessary permissions.
Queue Worker Command: Consider using queue:listen instead of queue:work for debugging purposes, as it will restart the worker after each job, which can help identify issues with long-running processes.
Check for External API Limits: If your job interacts with an external API, ensure that you are not hitting any rate limits or experiencing network issues.

By following these steps, you should be able to identify the root cause of the issue and implement a solution. If the problem persists, consider reaching out to the community with specific logs and details for further assistance.

Snapey

1 year ago

Level 122

running out of memory is often the cause of things stopping with no error message.

Make sure there are limits on what you fetch from the API in one job

1 like

Please or to participate in this conversation.