frutality's avatar

Octane exited with code zero

Hello,

I am running Laravel Octane in production and staging environments for several months now. Recently, I encountered weird issue in staging env.

According to systemctl status laravel-octane.service --no-pager -l, Octane exited with code 0, which means there were no errors/exceptions as I understand:

× laravel-octane.service - Laravel Octane
     Loaded: loaded (/etc/systemd/system/laravel-octane.service; enabled; preset: enabled)
     Active: failed (Result: start-limit-hit) since Fri 2025-09-12 06:46:28 UTC; 4 days ago
   Duration: 28.488s
    Process: 20398 ExecStart=php /home/web/app/artisan octane:start --max-requests=2500 --workers=${WORKERS} (code=exited, status=0/SUCCESS)
   Main PID: 20398 (code=exited, status=0/SUCCESS)
        CPU: 2.604s

I am running it using following systemd unit file:

[Unit]
Description=Laravel Octane
After=network.target
StartLimitIntervalSec=500
StartLimitBurst=5

[Service]
Type=simple
TimeoutStopSec=180
Restart=always
RestartSec=5s
User=www-data
Group=www-data
Environment="WORKERS=8"
ExecStart=php /home/web/app/artisan octane:start --max-requests=2500 --workers=${WORKERS}

[Install]
WantedBy=multi-user.target

Weird thing is, it exited on two different machines running in different cloud zones, at same time.

Worst thing is, it tried to start several times, but for some reason failed. And reason remains unknown to me. There are zero errors or exceptions in logfile. And according to nginx logs, there was no heavy load. Actually, since it is staging env, I only can see some health-check requests in logs preceding exit time and nothing else at all.

Any advice on how to proceed with debugging would be welcome. If something like that happen in production, it will be unpleasant. So what logs can I theoretically add?

There were zero deployments around that time, zero infrustructure changes.

P.S. Manual start of this unit was successful. Unit works. Question is, what actually happened?

0 likes
4 replies
JussiMannisto's avatar

Do you have something like unattended upgrades enabled? If so, you should check the relevant logs to see if some other service (e.g. your DBMS) was updated and restarted, and Octane failed because of that. Although I'd expect that to throw some sort of error.

1 like
frutality's avatar

I would expect some logs too. Also, this machine is only running Octane and Queue worker. DB, Cache and everything else are on different machines in cloud.

Indeed there are unattended upgrades enabled, and I think you made very right guess with that. There was an update at that time:

sudo cat /var/log/unattended-upgrades/unattended-upgrades.log


2025-09-12 06:34:29,276 INFO Starting unattended upgrades script
2025-09-12 06:34:29,277 INFO Allowed origins are: o=Ubuntu,a=noble, o=Ubuntu,a=noble-security, o=UbuntuESMApps,a=noble-apps-security, o=UbuntuESM,a=noble-infra-security
2025-09-12 06:34:29,277 INFO Initial blacklist: 
2025-09-12 06:34:29,277 INFO Initial whitelist (not strict): 
2025-09-12 06:34:44,024 INFO Removing unused kernel packages: linux-headers-6.8.0-64
2025-09-12 06:34:54,390 INFO Packages that were successfully auto-removed: linux-headers-6.8.0-64
2025-09-12 06:34:54,391 INFO Packages that are kept back: 
2025-09-12 06:34:54,711 INFO Packages that will be upgraded: apport apport-core-dump-handler cloud-init iputils-ping iputils-tracepath jq libgnutls30t64 libjq1 libperl5.38t64 libpq5 libpython3.12-minimal libpython3.12-stdlib libpython3.12t64 libsqlite3-0 libssh-4 libtiff6 libudisks2-0 libxml2 linux-headers-generic linux-headers-virtual linux-image-virtual linux-libc-dev linux-tools-common linux-virtual nginx nginx-common perl perl-base perl-modules-5.38 postgresql-client-16 python3-apport python3-problem-report python3-urllib3 python3.12 python3.12-minimal sudo udisks2
2025-09-12 06:34:54,711 INFO Writing dpkg log to /var/log/unattended-upgrades/unattended-upgrades-dpkg.log
2025-09-12 06:39:36,651 INFO All upgrades installed
2025-09-12 06:40:05,228 INFO Packages that were successfully auto-removed: linux-headers-6.8.0-60 linux-headers-6.8.0-60-generic linux-image-6.8.0-60-generic linux-modules-6.8.0-60-generic linux-tools-6.8.0-60 linux-tools-6.8.0-60-generic
2025-09-12 06:40:05,228 INFO Packages that are kept back:

However, looking at list of upgraded packages, I do not understand which upgrade exactly became the reason for restarting. There is also this logfile:

It is more interesting. Lines with services restart are repeating several times in short span of time. Like, system tried to restart services after successful upgrade of many packages. There is list of packages: postgresql-client-16 libpq5 libxml2 libgnutls30t64 libsqlite3-0 libtiff6 libssh-4. After each upgrade there is that log line:

Restarting services...
 systemctl restart fwupd.service laravel-octane.service laravel-queue-db.service packagekit.service unscd.service

I think I am going to disable unattended upgrades for now, but here is the question: why some unrelated package update results in services restart? It's not like my service is directly dependent on postgres-client, libsql, libxml etc.

Thank you for the hint!

JussiMannisto's avatar

why some unrelated package update results in services restart? It's not like my service is directly dependent on postgres-client, libsql, libxml etc.

Even if your DB is on a separate server, you still need the Postgres client to connect to it. PHP also depends on libxml2, at least. Unattended upgrades ran needrestart, which detected that Octane was running one of the old binaries, and restarted it.

Unattended upgrades install only security updates by default. You can turn it off, but then you have to be on the ball with manual security updates on all of the servers.

You may prefer manual security updates, but I'd at least consider fixing the unit file instead. That would make sure the service gets started if something goes wrong in the future. I'm not that familiar with Octane, though, and I'm not sure why you got those 0 error code exits during start attempts.

frutality's avatar

Yeah, I agree, that having security updates by default is good. I would not recommend anyone disable it without serious reason.

I mean, for now, as a workaround, I am disabling it. Until we find out an actual solution, it is not very cool that our service could go completely down randomly.

Also in our case, there is no public access to these machines from internet. They are behind load-balancer. And we updating base image on a regular basis.

Anyway, right now I am not sure where to start fixing unit file. As I understand, it already has adequate settings in terms of restart timeout. Will be researching now.

Please or to participate in this conversation.