Laravel 12 – php artisan schedule:run causes periodic CPU spikes even with no scheduled tasks

It is not exact time. There is some randomness there. The pattern is not always the same. I can not upload a direct link since this is my first day at the forum. But i will try to upload this: postimg.cc slash ZBVxPndK. If you can recreate the link you will see the chart of the CPU.

Does it really affect anything or bring problems? Load average 0.6 seems pretty low "peaks", I wouldn't even notice them. If there are no real problems just leave it as is.

This values are from my dev server. In the actual production one the impact is way worse.

Way worse how exactly? What LA do you have? Your app doesn't respond? What real impact do you experience?

This is a chart from production. https://postimg.cc/ZWyhym6n i have a 6 core machine which is 30/6 which leaves 5 processes competing a the same time.

LA ~15 seems pretty high for any system, it seems unhealthy, even without those peaks. What is your app average load? How many users or requests per second? Is it an ordinary web app or something more complex/resource consuming?

Which process(es) use cpu up to LA 15 for example at 12:00 AM on your graph?

Why load is down at 2:00 AM so drastically?

There are ~1000 request per minute. Usually the load is during the working hours. Outside of working hours we have not many requests. The most load would be like 1000 per minute

If you have 1000 requests per minute then lottery [2, 100] should raise sessions cleanup 20 times per minute, not once per hour. I don't believe this lottery is related with your problem.

Do you have anything else dev-related on your production environment? Telescope, xdebug, debugbar?.. Any cron tasks or system services?

Did you see which process consume CPU when a peak occurs? You can use htop and wait, or use atop to store CPU history and see what was going on afterwards.

Your peaks happen every 50 minutes on graph; does this value remind you something?

Can it be infrastructure problems? For example, your app is hosted on a virtual machine which is hosted on a hypervisor. If a hypervisor is backed up once per hour (typical configuration) and has poor disk or CPU performance it can trigger high CPU consumption and cause symptoms similar to what you see.

I just upped the ratio to 20, 100. Lets see how this impacts.

Thank you for your suggestion. I increased the chances of the lottery and now i have more frequent spikes (smaller spikes).

Is there any way how i can solve this issue long term? My production server is heavily impacted from this action. All monitoring tools raise alarms on heavy load.

I have a question if you don't mind. These Laravel cleanup functions are working only when schedule:run is setup in cron. Because when i remove the schedule:run from the cron the problem goes away for me.

what is your session driver? do you have some specific setup where file or database deletes are expensive?

if you can't do anything else, you can at least use a driver that does auto clearing of old data for you.

it should only spike for a few ms, so I would be very surprised if it causes an issue. Perhaps you have other under-capacity issues, and this is a red-herring.

Have you added anything to app service provider to preload data?

@snapey i have a chart of the issue in production. The app is running on a 6 code machine. so 30/6 is 5 threads waiting. Which makes the impact pretty severe in production. Increasing the lottery chances brought the values slightly lower but nothing of great improvement.

One question to you. There cleanup functions run only when schedule:run is configured? Because when i remove the schedule:run from cron the issue disappears.

The cleanup runs as part of the framework booting. The framework is booted on every http request and every artisan command. Are other commands taking a long time to run?

The production system gets ~1000 requests per minute for a small operation. For example: get user from DB who is making the request compare the DB data with input and perform an api request in another service is data does not match and store the update in database.

All production commands are performing well nothing out of ordinary.

But i have removed everything from dev server so there is nothing happening really. I just call the schedule:run and there is no commands to execute. It still spikes. Im thinking to remove the schedule:run from cron and do it as endpoint for runing my command. But this will be overkill and i still need to understand why these spikes ocur when there is nothing there.

It sounds like this has something to do with your app as a whole. As @snapey pointed out, the framework is booted when the scheduler runs, even when there are no scheduled tasks.

You have Telescope installed as a require dependency instead of require-dev. Are you actually running it in production? If so, its pruning operations might cause load spikes.

The telescope is from my development server. I tried to remove telescope completely at some point since i suspected it may cause the issue. Even without telescope the problem persists.