production.ERROR: Call to a member function beginTransaction() on null

I have the same mistake.

I think it is an error related to the database queue with supervisor. It is printing the error out every three seconds even though the website has connected correctly to the database and is working as it should.

This is my supervisor config file contents

[program:my-app]
process_name=%(program_name)s_%(process_num)02d
command=/usr/bin/php /srv/users/serverpilot/apps/my-app/artisan queue:work database --sleep=3 --tries=3
autostart=true
autorestart=true
user=serverpilot
numprocs=8
redirect_stderr=true
stdout_logfile=/srv/users/serverpilot/apps/my-app/supervisor.log

I am certain it is related to supervisor as I just ran

sudo supervisorctl stop my-app:*

and the errors immediately stopped being posted to the laravel log file.

So there seems to be an error where supervisor is not connecting to the database correctly?

Should the user be root and do I need to include a password in the config file? Also is /usr/bin/php correct for the command line?

It is strange because even though it was printing errors it still seemed to process the jobs fine.

After stopping and then starting supervisor again, it works normally for a day or so then the error starts again.

Beginning with:

production.ERROR: PDOException: SQLSTATE[HY000] [2002] Connection refused in /srv/users/serverpilot/apps/my-app/vendor/laravel/framework/src/Illuminate/Database/Connectors/Connector.php:68

I am not sure but I think it may be a permission error, as after I push an update to the server with git and git hooks then run ls -l in the app root directory. The artisan file and some others have root as the owner and group.

Could this be what is causing the error, because I have user=serverpilot in the supervisor config?

I have just run sudo chown -R supervisor:supervisor /srv/users/supervisor/apps So will see if the error still appears after a day or so.

The strange thing is that the connection refused error message seems to be appearing at exactly 06:25:17 in the log each morning.

After the connection refused error appears it is then followed by the "Call to a member function beginTransaction() on null" error being printed every 3 seconds until I restart the supervisor worker.

Does anybody have any ideas at all as to what might be causing the connection to be refused at this time each day?

@ttwb no idea what is causing this, but we are having exactly the same problem currently. We will post more as/when/if we find anything...

@ttwb FYI ours started to error at 06:25:15 this morning!

Facing the same issue. After a day or so, have to stop and start supervisor for exceptions to stop.

I've upgraded from 5.2 to 5.4

Could be related: https://github.com/laravel/framework/issues/15179

Same errors were logged at [2017-03-02 06:25:15] again this morning.

I am also using laravel 5.4.12 @komirad @ifpingram are you both using the database as the queue driver? Do you have similar supervisor configurations? Are your servers with digitalocean?

It's very odd that it seems to be happening at the exact same time each day.

@ttwb we can't find a solution to this, so what we have done is we restart the relevant supervisor jobs via a cron every hour. It's not ideal, but it seems to work, as we've not had these PDO exceptions thrown this morning...

@ifpingram I was thinking of doing something similar, thanks. Would you be able to post the cron job config you've set up to do this?

@ifpingram I set an hourly cron job in the root users crontab -e @hourly /root/restart-supervisor.sh

Where restart-supervisor.sh contains:

#!/bin/sh

sudo supervisorctl stop my-app:*
sudo supervisorctl start my-app:*

The errors were again logged at 06:25:19 this morning and began throwing the beginTransaction() on null exception every 3 seconds.

It did however then stop at 07:00:01 when the cron job was run on the hour. Ideally need to stop it from being logged at all as the log file had already grown 10mb in size.

I have a feeling it may be something to do with the server?

@ttwb I just put the command in Forge as:

157276 Hourly 0 * * * * root /usr/bin/supervisorctl restart laravel-worker-scriptname:*

I haven't had a chance yet to analyse the logs, but should do so next week to see if it is causing errors still for a while like yours is.

No idea if it is the server, or something in the code base / supervisor. This is on a box provisioned by Laravel Forge (Ubuntu) using php7.1; is that a similar server environment to yours?

@ifpingram I changed the crontab to

26 06 * * * /root/restart-supervisor.sh

So that it runs at 06:25am each day and I checked the log this morning and no errors had been printed. Hopefully this temporary fix will do for the time being.

Our server is Ubuntu 16.04 with digital ocean. Using serverpilot, php 7.1 only been running for just over a week.

@ttwb good stuff - fingers crossed it works! I have a feeling that this temporary fix will become permanent, as it looks like it might take quite a lot of work to get to the bottom of, and I don't have the time to investigate any further. :( I'll keep an eye on the thread in case you have the chance to investigate further yourself, but I would say that this hack should suffice...

@ifpingram I have found the source of the problem, turns out it was related to the server.

I am going to assume your server is with Digital Ocean as mine was.

If you go to /var/log/syslog on your server and scroll to 06:25:00 where the errors start occurring you can see that it happens because the mysqld process gets killed to free up memory.

If you look at the droplet usage graphs in your DO dashboard you can see a spike at this time.

I considered upgrading the droplet as I was on the $5 plan, however I then saw this thread:

https://www.digitalocean.com/community/questions/getting-regular-out-of-memory-kill-process-how-to-resolve-this-isse

Where people were saying they were still getting the issue even after upgrading. There was no response from DO support and I couldn't find a solution to it anywhere.

I decided in the end to migrate my sites to Vultr. I am on the $5 but getting twice the memory and more SSD space for the same price.

Have had it running a few days now and not had a single error in the log. Very happy with the decision to migrate so far.

@ttwb good find! Yes, this is on a Digital Ocean VM, although we're not reporting any errors since adding the supervisor restarting job, but if / once it rears it's ugly head again we'll follow your findings above. I just don't have the time at present to re-provision a new server... Many thanks for all your digging here; much appreciated!

I'm getting the same issue, on Digital Ocean, at 6:26 AM GMT. My error logs start producing this.

Did anyone actually find a permanent solution to this that isn't just restarting supervisor job every hour?

@bretto36 are you on the $5 plan? To be honest the only permanent solution I know of is migrating your server to vultr. This is what I've done and not had a single error since, the issue is with the digital ocean servers.

@tttwb Yes i'm on the $5 plan. Thanks for the info! I'm just using DO as a staging server. Will be using aws for the production, but might switch my staging over to vultr. Thanks for the headsup

Same issue here! (DigitalOcean) and the same time, same shit!!

I'm going to move to AWS or Vultr!

@tttwb and @ifpingram I guess it is fixed now in latest Laravel 5.4.24

Fixed database connection issue in queue worker #19263