Background Workers: Processing Jobs Without Blocking Users

What Is a Background Worker?

A background worker is a separate process that consumes jobs from a queue and executes them outside of the request-response cycle.

The user hits your API. Your API returns immediately. Somewhere else, a worker picks up the job and does the actual work — resizing an image, sending an email, generating a report, syncing data.

The user never waits for it. They don't even know it's happening.

The Problem It Solves

Some tasks take too long to run in a web request:

• Sending 1,000 emails after a bulk action
• Processing a video upload
• Running an analytics report over millions of rows
• Calling a slow third-party API

If you run these inline, the HTTP request stays open until they finish. That means:

• Users wait 30 seconds staring at a spinner
• Web server threads are occupied for the full duration
• A single slow job can exhaust your thread pool and bring down the API

Background workers move this work out of the request-response path entirely.

How It Works

The pattern is always the same:

Your API creates a job record and puts it in a queue (Redis, SQS, RabbitMQ). The API returns. Separately, worker processes poll the queue, claim jobs, execute them, and acknowledge completion. If a job fails, it stays in the queue and gets retried up to N times, then moves to a dead-letter queue.

Workers are typically separate long-running processes, not serverless functions — though both are valid depending on job characteristics.

When to Add Them

Add background workers when:

• Processing is async by nature — the user doesn't need the result right now
• Jobs are long-running — more than a few hundred milliseconds is too long for inline processing
• You have retry requirements — if the job fails, it needs to be retried without user intervention
• You need to rate-limit external calls — workers can be throttled independently of your API
• Jobs need to run on a schedule — cron-style recurring tasks

When NOT to Add Them

• When the user needs the result immediately — auth checks, payment confirmations, form validation
• Simple synchronous flows with no downstream processing
• When you haven't actually hit the wall yet — don't add worker infrastructure speculatively

Separate Workers From Your API

This is where most teams make the mistake. They add a job queue but run the workers as threads inside the same API process.

The problem: a burst of heavy jobs consumes all the threads in your API process. API response times tank. You've coupled the very thing you were trying to decouple.

Workers should run as separate processes or separate container instances from your API. They have their own resource allocation, they can be scaled independently, and a runaway job can't affect your API's ability to serve requests.

Real World

Shopify processes billions of background jobs per day using Sidekiq (Ruby) with Redis as the queue. Every order confirmation email, inventory webhook, and third-party app callback is a background job.

GitHub uses background workers for everything that happens after you push code — CI job triggering, notification emails, feed updates, webhook deliveries. The git push itself is synchronous. Everything downstream is async.

Takeaways

• Background workers process jobs outside the HTTP request-response cycle
• Use them for async, long-running, or retry-required work
• Job queue (Redis, SQS) → Worker picks up → Processes → Acknowledges
• Always run workers as separate processes from your API — not threads inside it
• Failed jobs should retry with backoff and eventually land in a dead-letter queue
• Scale workers independently from your API based on queue depth