Laravel Queue Jobs: Processing 10,000 Tasks Without Breaking

Your application just hit production. Users are uploading files, sending emails, generating reports, and suddenly everything grinds to a halt. The culprit? You're processing everything synchronously, and your server can't keep up.

I ran into this exact problem on a client's document processing platform. They needed to handle bulk PDF uploads, extract text, generate thumbnails, and run OCR -- sometimes 500 files at once. The first version locked up their entire application for minutes while users stared at loading spinners. After implementing Laravel queues with proper Redis configuration and monitoring, we processed 10,000+ documents daily without a single timeout.

Here's what you'll learn: setting up Redis for production queue workloads, implementing job batching for complex workflows, handling failures gracefully, and monitoring everything with Horizon. By the end, you'll have a bulletproof queue system that scales.

Why Laravel Queues Matter for Production Apps

Laravel queues move time-consuming tasks to background workers so your application stays responsive. Instead of making users wait while you send emails or process images, you dispatch jobs to a queue and return a response immediately.

The real power shows up when you need to process hundreds or thousands of tasks. I've used queues for everything from bulk email campaigns to video transcoding to generating thousands of PDF reports. Without queues, these operations would be impossible in any reasonable timeframe.

Before we get into Redis configuration and job classes, it's worth seeing exactly what happens when you dispatch a job. Click through this to get the full picture:

That's the full picture. Now let's build it properly. Queues add complexity though. You need a reliable driver (Redis is the right call), workers that don't crash, proper failure handling, and monitoring. Let's set all of this up right from the start.

Setting Up Redis as Your Queue Driver

Laravel supports multiple queue drivers, but Redis is the sweet spot for most applications. It's fast, reliable, and handles job priorities better than database queues. If you're coming from a database driver and noticing slowdowns, this is your fix.

First, install Redis and the PHP extension:

# Install Redis server
sudo apt-get install redis-server

# Install PHP Redis extension
sudo pecl install redis
sudo echo "extension=redis.so" > /etc/php/8.3/mods-available/redis.ini
sudo phpenmod redis

Configure Redis in your .env:

QUEUE_CONNECTION=redis
REDIS_HOST=127.0.0.1
REDIS_PASSWORD=null
REDIS_PORT=6379
REDIS_DB=0

For production, I always set up a dedicated Redis database for queues. This separates queue data from cache data and makes monitoring cleaner. Update config/database.php:

'redis' => [
    'client' => env('REDIS_CLIENT', 'phpredis'),

    'default' => [
        'url' => env('REDIS_URL'),
        'host' => env('REDIS_HOST', '127.0.0.1'),
        'password' => env('REDIS_PASSWORD'),
        'port' => env('REDIS_PORT', '6379'),
        'database' => env('REDIS_DB', '0'),
    ],

    'cache' => [
        'url' => env('REDIS_URL'),
        'host' => env('REDIS_HOST', '127.0.0.1'),
        'password' => env('REDIS_PASSWORD'),
        'port' => env('REDIS_PORT', '6379'),
        'database' => env('REDIS_CACHE_DB', '1'),
    ],

    // Dedicated database for queues
    'queues' => [
        'url' => env('REDIS_URL'),
        'host' => env('REDIS_HOST', '127.0.0.1'),
        'password' => env('REDIS_PASSWORD'),
        'port' => env('REDIS_PORT', '6379'),
        'database' => env('REDIS_QUEUE_DB', '2'),
    ],
],

Then update your queue configuration in config/queue.php:

'connections' => [
    'redis' => [
        'driver' => 'redis',
        'connection' => 'queues', // Use dedicated Redis database
        'queue' => env('REDIS_QUEUE', 'default'),
        'retry_after' => 90,
        'block_for' => null,
    ],
],

Why retry_after at 90 seconds? If a job takes longer than this value, Laravel assumes the worker died and makes the job available again. Set this based on your longest-running job plus a safety margin. Get it wrong and you'll end up with duplicate processing.

Creating Your First Production-Ready Job

Let's build a real job that processes uploaded documents. This example extracts text and generates thumbnails.

php artisan make:job ProcessDocument

If you're on Laravel 12+, the generated job uses the unified Queueable trait from Illuminate\Foundation\Queue\Queueable. This replaces the old four-trait pattern (Dispatchable, InteractsWithQueue, Queueable, SerializesModels) that older tutorials still show. One trait does it all now.

Here's a production-ready implementation:

<?php

namespace App\Jobs;

use App\Models\Document;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Queue\Queueable;
use Illuminate\Support\Facades\Log;
use Illuminate\Support\Facades\Storage;

class ProcessDocument implements ShouldQueue
{
    use Queueable;

    public int $tries = 3;
    public int $timeout = 120;
    public bool $failOnTimeout = true;
    public bool $deleteWhenMissingModels = true;

    public function __construct(
        public Document $document
    ) {}

    public function handle(): void
    {
        try {
            $path = Storage::path($this->document->file_path);

            // Extract text from PDF
            $parser = new \Smalot\PdfParser\Parser();
            $text = $parser->parseFile($path)->getText();

            $this->document->update([
                'content' => $text,
                'word_count' => str_word_count($text),
            ]);

            // Generate thumbnail
            $thumbnailPath = 'thumbnails/' . $this->document->id . '.jpg';
            \Intervention\Image\Facades\Image::make($path)
                ->resize(300, null, fn ($c) => $c->aspectRatio())
                ->save(Storage::path($thumbnailPath));

            $this->document->update(['thumbnail_path' => $thumbnailPath]);

            Log::info('Document processed', [
                'document_id' => $this->document->id,
                'word_count' => $this->document->word_count,
            ]);

        } catch (\Exception $e) {
            Log::error('Document processing failed', [
                'document_id' => $this->document->id,
                'error' => $e->getMessage(),
                'attempt' => $this->attempts(),
            ]);
            throw $e;
        }
    }

    public function failed(\Throwable $exception): void
    {
        $this->document->update(['status' => 'failed']);

        // Notify the user their document failed
        $this->document->user->notify(
            new DocumentProcessingFailed($this->document, $exception->getMessage())
        );
    }
}

The failed() method is critical. Most tutorials skip it. But that's the code that runs when all retries are exhausted -- it's your last chance to clean up state and notify users. Don't skip it.

Handling Job Priorities with Multiple Queues

Not all jobs are equal. Sending a password reset email should happen faster than generating a monthly report. Priority queues solve this.

Define queue names in your job:

// High priority: dispatch to 'critical' queue
ProcessPasswordReset::dispatch($user)->onQueue('critical');

// Normal priority
SendWelcomeEmail::dispatch($user)->onQueue('emails');

// Low priority: batch reports can wait
GenerateMonthlyReport::dispatch($user)->onQueue('reports');

Configure your workers to process these in priority order:

php artisan queue:work --queue=critical,emails,default,reports

Workers process the critical queue first. Only when it's empty do they move to emails, then default, then reports. This keeps your most important jobs flowing even under heavy load.

Processing in Bulk: Job Batching

When you need to process hundreds of files and track overall progress, job batching is the right tool. Laravel's batch system handles this elegantly.

use Illuminate\Bus\Batch;
use Illuminate\Support\Facades\Bus;

public function processBulkUpload(Request $request): JsonResponse
{
    $documents = $request->user()
        ->documents()
        ->whereNull('content')
        ->get();

    $jobs = $documents->map(
        fn ($doc) => new ProcessDocument($doc)
    )->toArray();

    $batch = Bus::batch($jobs)
        ->then(function (Batch $batch) {
            // All jobs completed successfully
            Log::info("Batch {$batch->id} completed", [
                'total' => $batch->totalJobs,
                'processed' => $batch->processedJobs(),
            ]);
        })
        ->catch(function (Batch $batch, \Throwable $e) {
            // At least one job failed
            Log::error("Batch {$batch->id} had failures", [
                'failed' => $batch->failedJobs,
                'error' => $e->getMessage(),
            ]);
        })
        ->finally(function (Batch $batch) {
            // Runs regardless of success or failure
            event(new BulkProcessingComplete($batch->id));
        })
        ->allowFailures() // Don't cancel entire batch on one failure
        ->dispatch();

    return response()->json([
        'batch_id' => $batch->id,
        'total' => $batch->totalJobs,
    ]);
}

The allowFailures() call is important for bulk operations. Without it, one bad file cancels the entire batch. With it, the batch continues and you can review individual failures afterward.

Track batch progress in real time:

public function batchStatus(string $batchId): JsonResponse
{
    $batch = Bus::findBatch($batchId);

    return response()->json([
        'total' => $batch->totalJobs,
        'processed' => $batch->processedJobs(),
        'failed' => $batch->failedJobs,
        'progress' => $batch->progress(), // 0-100
        'finished' => $batch->finished(),
    ]);
}

Monitoring with Laravel Horizon

For any app processing more than a few dozen jobs per hour, Horizon is not optional. It gives you real-time visibility into what your queues are actually doing.

composer require laravel/horizon
php artisan horizon:install
php artisan migrate

The piece most people miss is auto-balancing configuration. Set it up in config/horizon.php:

'environments' => [
    'production' => [
        'supervisor-1' => [
            'maxProcesses' => 20,
            'balanceMaxShift' => 1,
            'balanceCooldown' => 3,
        ],

        'document-processor' => [
            'connection' => 'redis',
            'queue' => ['critical', 'documents'],
            'balance' => 'auto',
            'minProcesses' => 2,
            'maxProcesses' => 10,
        ],

        'email-worker' => [
            'connection' => 'redis',
            'queue' => ['emails', 'notifications'],
            'balance' => 'simple',
            'processes' => 5,
        ],
    ],
],

Horizon watches queue depth and automatically scales workers up or down. During a bulk upload storm, it spins up more document processors. When things calm down, it scales back. You set the bounds and it manages the rest.

Access the dashboard at /horizon. Restrict it in production:

// app/Providers/HorizonServiceProvider.php
protected function gate(): void
{
    Gate::define('viewHorizon', function ($user) {
        return in_array($user->email, [
            '[email protected]',
        ]);
    });
}

Run Horizon via Supervisor instead of artisan directly:

[program:horizon]
process_name=%(program_name)s
command=php /var/www/yourapp/artisan horizon
autostart=true
autorestart=true
user=www-data
redirect_stderr=true
stdout_logfile=/var/www/yourapp/storage/logs/horizon.log
stopwaitsecs=3600

The stopwaitsecs=3600 is critical. It gives Horizon up to an hour to finish in-flight jobs before forcing a stop. Without it, a Supervisor restart during deployment kills jobs mid-execution.

Handling Failures Like a Pro

The visualizer above shows it clearly: failures are normal. The question is what your app does about them.

Exponential Backoff

Don't retry immediately. Give the external service time to recover:

public function backoff(): array
{
    return [10, 30, 60]; // seconds between attempts
}

First retry after 10 seconds. Second after 30. Third after 60. Way better than hammering a struggling API three times in three seconds.

Rate Limiting Middleware

If you're calling external APIs, use the built-in rate limiter:

use Illuminate\Queue\Middleware\RateLimited;

public function middleware(): array
{
    return [new RateLimited('external-api')];
}

Define the limiter in a service provider:

RateLimiter::for('external-api', function (object $job) {
    return Limit::perMinute(60);
});

This prevents a burst of jobs from getting your IP blocked on a third-party API.

Preventing Duplicate Processing

For jobs that should never run concurrently for the same model, use the overlap middleware:

use Illuminate\Queue\Middleware\WithoutOverlapping;

public function middleware(): array
{
    return [new WithoutOverlapping($this->document->id)];
}

Two workers can't process the same document simultaneously. The second attempt waits for the first to finish.

Five Mistakes That Break Queue Systems

I've made all of these personally.

Not restarting workers after deployment. Workers cache your application code in memory. A deploy doesn't update running workers. Add php artisan queue:restart (or horizon:terminate) to your deploy script. Every time.

Setting retry_after too low. If your longest job takes 60 seconds and retry_after is 90, you're fine. But if a slow job takes 100 seconds, Laravel marks it as failed and requeues it, potentially creating duplicates. Add a buffer.

No failed() method on jobs. When retries run out and a job fails permanently, you need to clean up. Update the database record, notify the user, fire an event. Leaving this empty means your app quietly gets into an inconsistent state.

Using database queues above 100 jobs/minute. Works fine for low volume. Falls over fast at scale. Redis is the right call for production.

Not monitoring failed jobs. A growing failed_jobs table is a ticking time bomb. Either set up Horizon alerts or build a simple check. I usually fire a Slack notification when failed jobs exceed a threshold in any given hour.

If you're scheduling jobs to run at specific intervals, I always build the cron expressions first to make sure the timing is right before wiring them into the scheduler.

Production Checklist Before Going Live

Before switching on production queues, verify these are done:

Redis is running and configured on a dedicated database. Supervisor is managing your workers (or Horizon, if you went that route). Workers restart automatically after deployment. The failed jobs table exists -- it's automatic in new Laravel 12+ apps but older apps need the migration. Job timeouts match your longest actual operations. Priority queues are configured for different workloads. You've tested failure scenarios including the failed() method path. You have monitoring -- either Horizon or custom alerting on failed job count.

The difference between a queue system that "works in development" and one that holds up in production is mostly this list.

Advanced Patterns: Job Chaining and Deferred Jobs

Sometimes jobs need to run in sequence. Job chaining guarantees order:

ProcessDocument::withChain([
    new ExtractText($document),
    new GenerateThumbnail($document),
    new UpdateSearchIndex($document),
    new NotifyUser($document->user),
])->dispatch($document);

If any job in the chain fails, the rest don't execute. This is the right pattern for onboarding flows where each step depends on the previous one.

Deferred Jobs

Laravel 12 introduced the deferred connection for tasks you want to run synchronously but after the response is sent:

RecordAnalyticsEvent::dispatch($event)->onConnection('deferred');

Handy for analytics logging or audit trails -- lightweight enough that you don't want queue overhead, but you don't want to slow down the user response either.

Testing Queue Jobs

Don't skip this. Two tests cover most of what you need:

use Illuminate\Support\Facades\Queue;

public function test_document_upload_dispatches_job(): void
{
    Queue::fake();

    $document = Document::factory()->create();
    ProcessDocument::dispatch($document);

    Queue::assertPushed(ProcessDocument::class, function ($job) use ($document) {
        return $job->document->id === $document->id;
    });
}

public function test_job_processes_document_correctly(): void
{
    $document = Document::factory()->create([
        'file_path' => 'test-document.pdf',
        'content' => null,
    ]);

    $job = new ProcessDocument($document);
    $job->handle();

    $this->assertNotNull($document->fresh()->content);
    $this->assertNotNull($document->fresh()->thumbnail_path);
}

Test your failed() methods too. They're critical code paths that almost never get tested until something breaks in production.

When NOT to Use Queues

Queues aren't always the answer. Skip them for operations under 200ms, critical flows where users need immediate feedback, simple CRUD operations, and data that must be immediately consistent (like updating a balance before displaying it).

Queues add operational complexity. Use them when the responsiveness and scalability benefits clearly outweigh that cost. If you're curious about other ways to improve performance, I wrote about query optimization and general performance wins that pair well with queue improvements.

Frequently Asked Questions

Should I use Redis or the database driver for Laravel queues?

Redis, almost always. Database queues work for low-volume apps (under ~100 jobs per minute), but they add load to your database and don't scale. Redis is purpose-built for this kind of fast, in-memory data handling. The only exception is if you genuinely can't run Redis in your environment.

How many queue workers should I run?

Start with 4-8 for most apps and scale from monitoring data. Each worker handles one job at a time. If you're processing 100 jobs per minute with an average execution time of 5 seconds, you need at least 8-9 workers. Horizon's auto-balancing helps a lot here.

What happens if a worker crashes mid-job?

The job stays in Redis. After retry_after seconds pass, Laravel makes it available again. This is why getting that value right matters. Too low and you get duplicates. Too high and failed jobs sit idle too long.

How do I handle jobs that shouldn't run concurrently?

Use the WithoutOverlapping middleware as shown above. It uses Redis locks to prevent multiple instances of the same job from running simultaneously.

Can I use queues with Laravel Octane?

Yes, but run your queue workers separately from the Octane server. Octane keeps your application in memory between requests, which can cause state bleed issues if workers share the same process space.

Wrapping Up

Laravel queues transform how you build applications. What used to require complex infrastructure is now a few commands and configuration files.

Start simple. Get basic queue processing working with Redis, then add Horizon for monitoring, priority queues for workload separation, and batching for bulk operations. I've built apps processing millions of jobs monthly with exactly these patterns.

Your queue system should be invisible to users. When it's working right, nobody notices. When it's failing, everyone knows.

Need help implementing Laravel queues in your application? I've built queue systems for document processing, email campaigns, video transcoding, and more. Let's talk about your project and build something reliable.