10 min read

What /up Cannot See: The Daily Health Check Behind My Laravel App, and the 3 Bugs It Had First

A root-owned log file made my queued jobs fail for a day while the homepage answered 200. The daily check I built for it, and its three bugs.

What /up Cannot See: The Daily Health Check Behind My Laravel App, and the 3 Bugs It Had First

On 9 May, the queued jobs on hafiz.dev started failing and kept failing for the rest of the day. The newsletter send failed. The Dev.to cross-post failed. The homepage answered 200 the whole time, so anything watching it from the outside would have shown green. The cause was a duplicate schedule:run line I had added to root's crontab while checking the scheduler.

That's the gap between "the site is up" and "the app works". Laravel's /up route sits on the first side of it. So in October I wrote a daily check for the second side, called ops:health, that runs inside the app as the web user and sends a Telegram message only when something fails.

It had three bugs in its first week, all mine.

What /up actually checks

Every new Laravel 11+ app registers /up in bootstrap/app.php. Here's the controller behind it, Illuminate\Foundation\Http\HealthCheck, as of 13.42:

public function __invoke(Request $request)
{
    $exception = null;

    try {
        Event::dispatch(new DiagnosingHealth);
    } catch (\Throwable $e) {
        if (app()->hasDebugModeEnabled()) {
            throw $e;
        }

        report($e);

        $exception = $e->getMessage();
    }

    $status = $exception ? 500 : 200;

    // ... renders health-up.blade.php, or JSON, with that status
}

That's all of it. The framework booted, an event fired, and nothing threw. There's no database query in there unless you add one with a DiagnosingHealth listener.

That's fine for a load balancer, which only needs to know whether to send traffic. It's also what an external uptime monitor sees when it pings your homepage. Neither one runs a queued job or looks at who owns your log file.

The failure behind a green 200

Laravel's daily log channel creates a new file on the first write of the day, and whoever writes first owns it. In May that was root. The duplicate cron line ran schedule:run as root, a scheduled command wrote a log line, and from then on www-data could read the day's log but not append to it.

To show it step by step, I reproduced it in a php:8.4-cli container with a fresh Laravel 13.42 app, its storage owned by www-data. As root, one artisan command wrote one log line. Then, as www-data, a queue worker picked up a job that logs one line:

  2026-11-13 11:13:50 App\Jobs\SendNewsletter ........................ RUNNING
  2026-11-13 11:13:50 App\Jobs\SendNewsletter .................... 9.70ms FAIL

   UnexpectedValueException

  The stream or file "/app/storage/logs/laravel-2026-11-13.log" could not be opened in
  append mode: Failed to open stream: Permission denied
  The exception occurred while attempting to log: Newsletter sent

The job failed on its only log line. The worker's attempt to report that failure hit the same file, and the queue:work --once process exited with code 1. The job did land in failed_jobs. And a request for /, handled as www-data in the same container, returned 200, because rendering that page never writes to the log.

That's the May incident in one container. In production it lasted until I noticed and ran chown.

What a log-based alert sees

The obvious alert is "tell me when errors show up in the log". So what did the log hold after that failure?

Before you read on

Same container, a moment later. The job has failed and the worker has exited. Your alert scans storage/logs for new ERROR lines.

What's in storage/logs now?

$ ls -l storage/logs/
-rw-r--r-- 1 root root 63 Nov 13 11:13 laravel-2026-11-13.log

$ cat storage/logs/laravel-2026-11-13.log
[2026-11-13 11:13:43] local.INFO: checking something as root

Nothing new. Root's one line, 63 bytes, and no other file. Every attempt to record the failure needed the file that was broken, so the errors had nowhere to go. Laravel does have an emergency logger, but LogManager only uses it when a channel can't be created. Here the channel was created fine and the write failed.

ops:health also reads the failed_jobs table, where the job did land. And it checks the paths www-data needs to write, including today's log file (trimmed):

$paths = [
    storage_path('logs'),
    storage_path('logs/laravel-'.now()->format('Y-m-d').'.log'),
    storage_path('framework/views'),
    storage_path('app/public'),
    base_path('bootstrap/cache'),
];

foreach ($paths as $path) {
    if (file_exists($path) && ! is_writable($path)) {
        $owner = posix_getpwuid(fileowner($path))['name'] ?? '?';
        $this->failures[] = "storage: {$path} is not writable (owner {$owner})";
    }
}

is_writable() answers for the user running the check, and running things as root is what caused the incident in the first place. In the container, on the root-owned log:

root:     bool(true)
www-data: bool(false)

Run the check as root and it passes on the exact file that's broken. So the command warns when it starts as root, and I only ever run it with sudo -u www-data. The scheduled run already does, because schedule:run lives in the www-data crontab. The laravel-ok package has a Permission Check with the same trap. Its README says it checks permissions "for the current user", so it's only as honest as the user you run it as.

What the daily check looks at

ops:health is one artisan command, about 370 lines, with no package behind it. In order:

  • Log errors in the last 24 hours, grouped by message, top three reported.
  • Failed jobs, with the newest job's class and time.
  • Stale queue: any job waiting more than 15 minutes means the worker isn't running.
  • Scheduled posts still unpublished 15 minutes after their time.
  • Every published post renders without an error, without a broken component, and with every image it shows present on disk. A broken post still answers 200, and posts are what this site is for.
  • Scheduler heartbeat younger than 15 minutes.
  • Storage writable by the web user, as above.
  • Disk above 10% free.
  • Key pages answering 200, which is the uptime-monitor part, done from inside.

It runs from the scheduler every morning:

Schedule::call(fn () => file_put_contents(OpsHealth::heartbeatPath(), now()->toIso8601String()))
    ->everyFiveMinutes()
    ->name('ops-heartbeat');

Schedule::command('ops:health --notify')
    ->dailyAt('07:00')
    ->onFailure(fn () => logger('ops:health found failures'));

--notify sends one Telegram message listing the failures, and nothing when every check passes. A message that arrives every morning gets ignored by the third morning, so silence has to mean healthy.

That rule is also how I found the bugs.

Three bugs it had in its first week

1. Scanner noise looked like errors

The log from 27 October alone held at least 28 Livewire hydration errors. "Cannot update locked property", "Livewire encountered corrupt data", that family. They came from scanners probing /livewire/update for CVE-2025-54068, and the app runs Livewire 3.6.4, the patched release. So the attempts failed, which is exactly what produced the errors.

A check that fails on these alerts every day for nothing. Ignoring all Livewire errors would hide a real one. So the command keeps a list of exact message prefixes, counts matches as "scanner errors ignored", and prints the count without failing. A prefix only goes on the list after I've traced it to a scanner.

2. The heartbeat lived in the cache, and deploys clear the cache

The first version stored the heartbeat in the cache. deploy.sh runs php artisan optimize:clear, which clears the cache, so every deploy deleted the heartbeat (a deploy already does enough to a running app, from clearing caches to the SIGTERM every queue worker gets). The next check reported a dead scheduler on a server where the scheduler was fine.

The fix was a plain file in storage/app, excluded from deploys. If you use spatie/laravel-health, its ScheduleCheck also keeps its heartbeat in the cache. Its docs highly recommend giving it a cache store of its own. I'd follow that advice, and make sure your deploy never clears that store.

3. It alerted on its own alert

When a scheduled command exits with a non-zero code, Laravel's scheduler logs an error. ops:health returns 1 when a check fails, so a failing run leaves this behind:

[2026-10-30 07:00:08] production.ERROR: Scheduled command ['/usr/bin/php8.3' 'artisan' ops:health --notify] failed with exit code [1].

That line sits inside the next run's 24-hour window. So the next morning's run found one error, failed on it, and logged a fresh copy for the morning after. From 30 October to 3 November, the only error each run found was the line the previous run had left.

The fix skips that one message:

if (Str::startsWith($message, 'Scheduled command [') && str_contains($message, "'artisan' ops:health")) {
    continue;
}

The run that failed already sent its alert. Counting its leftover line again only adds noise. Five mornings, five identical messages.

The layer the app can't see

Some failures happen before PHP runs at all. On this server the PHP-FPM pool is shared with other sites. On 3 November a scanner burst hit hafiz.dev, the pool ran out of workers, and the FPM log said so:

[03-Nov-2026 14:31:31] WARNING: [pool www] server reached pm.max_children setting (12), consider raising it

While that lasts, requests for every site in the pool wait for a free worker. A request that's still waiting hasn't reached Laravel, so storage/logs has nothing to say about it, and ops:health has no way to know. That's why the daily check is only half of it. The other half reads the logs around the app (the FPM log and the nginx error log, filtered to this site) when I check production by hand.

Three layers of a Laravel health check An uptime monitor or the /up route only sees whether the framework boots and a page answers 200, so it stayed green while a root-owned log file made the queued jobs fail. A daily check inside the app, running as www-data, sees failed jobs, a stale queue, the scheduler heartbeat, storage permissions and whether every post renders, and it catches the root-owned log. The PHP-FPM and nginx logs around the app show requests that wait or fail before PHP runs, such as the shared pool reaching pm.max_children, which the app itself never sees. Layer What it can see From this app Outside the app uptime monitor, /up Does the framework boot, and does a page answer 200? 9 May: green all day, while the queued jobs failed Inside the app ops:health as www-data daily at 07:00 Failed jobs, a stale queue, the scheduler heartbeat, storage writable, every post rendering Root-owned log: a failed_jobs row, and the storage check fails with owner root Around the app PHP-FPM and nginx logs Requests that wait or fail before PHP runs The shared pool reaching pm.max_children

Should you use a package instead?

Maybe. spatie/laravel-health ships 21 checks, including the database connection, queue, schedule, disk space, debug mode and security advisories. It also gives you a results page, a JSON endpoint and notifications. If you want those, start there.

None of its built-in checks looks at who owns your files, though, and none of them knows what your app is for. Both packages let you write custom checks, and those are the ones that matter. For this site that's storage writable by www-data, the failed jobs table, and every published post rendering. For a shop it might be the checkout rendering and yesterday's orders being exported.

I went with one command because it has no dependencies and I can read all of it in a few minutes. And I knew exactly which failures I was guarding against, because I'd already had them. If you're choosing between tools for the debugging side rather than the alerting side, Telescope, Pulse and Nightwatch answer different questions from this one.

FAQ

What does Laravel's /up route check?

It boots the framework and dispatches a DiagnosingHealth event. It returns 200 if nothing throws and 500 if something does. It doesn't query the database unless you register a DiagnosingHealth listener that does.

Why should health checks run as www-data instead of root?

Because is_writable() and similar checks answer for the current user, and root can write anywhere. A root-owned file passes the check as root and fails the real app. Running artisan as root can also create the root-owned file you're checking for.

How do I run artisan as the web user?

Use sudo -u www-data php artisan .... For tinker, add env HOME=/tmp. Without it, PsySH tries to write its config under the web user's home directory and stops with Writing to directory /var/www/.config/psysh is not allowed.

How do I keep scanner noise out of a log-based alert?

Match exact message prefixes you've traced, count them, and report the count without failing. Don't ignore a whole package's errors. And keep Livewire and Filament patched, which is what makes the noise harmless in the first place. The Filament 5.10 security fixes are a recent example of why.

Should the alert fire every day with a summary?

No. A daily "all good" message trains you to ignore the channel. Send nothing when every check passes, so a message always means something failed.

Two rules for whatever you build

The slow job alerts in Laravel 13.40 apply the same thinking one level down, to the job that didn't fail but should still worry you. At either level, run the check as the user the app runs as, and make silence mean healthy.

Share: X/Twitter | LinkedIn | | RSS
Hafiz Riaz

About Hafiz

Senior Full Stack Developer. I build production software with Laravel, Filament, Vue, and AI integrations, and write about the real decisions behind shipping it.

Get in touch →

Get web development tips via email

Join 50+ developers • No spam • Unsubscribe anytime