Background tasks in Django 6: Do you still need Celery?

Why background tasks are useful in the first place

A normal Django request is supposed to receive an HTTP request, do some work and return a response. That works well until the work starts taking several seconds.

Sending an email is a good example. Imagine a user creates an account and the view sends a welcome email before returning the response:

from django.core.mail import send_mail
from django.shortcuts import redirect


def signup(request):
    user = create_user(request.POST)

    send_mail(
        subject="Welcome",
        message="Thanks for signing up.",
        from_email=None,
        recipient_list=[user.email],
    )

    return redirect("dashboard")

If the mail server takes three seconds to respond, the user waits those three seconds too. The account has already been created, but the browser is still sitting there because Django hasn't finished the request.

There are plenty of other jobs with the same problem: generating PDFs, resizing images, importing CSV files, calling a slow API, creating reports or sending a large batch of emails. None of those jobs need to block the HTTP response in most applications.

This is the problem tools like Celery have solved for Django projects for years. The web process puts a message on a queue, a worker picks it up somewhere else, and the user gets a response without waiting for the job to finish.

What changed in Django 6?

Django 6 now has its own Tasks framework. We can define a task using @task and enqueue it using .enqueue().

A basic task looks like this:

from django.core.mail import send_mail
from django.tasks import task


@task
def send_welcome_email(email):
    send_mail(
        subject="Welcome",
        message="Thanks for signing up.",
        from_email=None,
        recipient_list=[email],
    )

Then the view becomes:

def signup(request):
    user = create_user(request.POST)

    send_welcome_email.enqueue(
        email=user.email,
    )

    return redirect("dashboard")

That API is much nicer than having every Django project invent its own abstraction around background work. It also means reusable Django apps can describe background tasks without depending directly on Celery, Redis, RabbitMQ or another queue implementation.

But there is an important detail hidden behind that small example: Django doesn't actually include a production worker.

Django gives us the API, not the worker

This is the part I misunderstood the first time I read about the new Tasks framework.

I assumed Django 6 had added something similar to Celery where I could start a Django worker and have it process jobs in the background. It hasn't.

Django defines how a task is created, queued and represented. A task backend decides where the task goes and how it is eventually executed. For a real production application, you still need external infrastructure capable of storing the queue and running workers.

Django itself includes two backends:

ImmediateBackend
DummyBackend

Both are intended for development and testing.

The default is the immediate backend:

TASKS = {
    "default": {
        "BACKEND": "django.tasks.backends.immediate.ImmediateBackend",
    }
}

The name tells us exactly what it does. When we enqueue a task, it runs immediately in the same process.

So this:

send_welcome_email.enqueue(
    email=user.email,
)

Looks like a background task, but with the default backend the user still waits for the email to finish.

That's intentional. It means I can start changing application code to use Django's Tasks API before I've decided how production jobs will be processed.

Create a small Django 6 example

Let's build something simple so we can see how the API works.

Create a project if you don't already have one:

python3 -m venv venv
source venv/bin/activate

pip install django

django-admin startproject config .
python manage.py startapp accounts

Add the app:

INSTALLED_APPS = [
    # ...

    "accounts",
]

For this example, I'm going to use Django's console email backend so emails appear in the terminal instead of being sent to a real mail server:

EMAIL_BACKEND = (
    "django.core.mail.backends.console.EmailBackend"
)

Now create accounts/tasks.py:

from django.core.mail import send_mail
from django.tasks import task


@task
def send_welcome_email(email):
    return send_mail(
        subject="Welcome",
        message="Thanks for signing up.",
        from_email=None,
        recipient_list=[email],
    )

Django doesn't require tasks to live in a file called tasks.py, but that's the convention I would use. It also makes background work easy to find when I come back to the project later.

Enqueue the task

We can test the task from the Django shell:

python manage.py shell

Then:

from accounts.tasks import send_welcome_email

result = send_welcome_email.enqueue(
    email="stein@example.com",
)

With the default backend, you'll immediately see the email printed in the terminal.

We also get a result object back:

result.status

Because the immediate backend runs the function right away, the task has already finished by the time the call returns.

This isn't useful as a real background queue, but it makes local development very easy. I can write application code using the same task interface I'll use later without starting Redis and three other processes just to work on a signup form.

Tasks can accept arguments

Most background jobs need some information from the code that enqueues them.

For example:

@task
def send_invoice_email(
    email,
    invoice_number,
):
    send_mail(
        subject=f"Invoice {invoice_number}",
        message="Your invoice is ready.",
        from_email=None,
        recipient_list=[email],
    )

And:

send_invoice_email.enqueue(
    email="stein@example.com",
    invoice_number="INV-1004",
)

There is one restriction that is easy to miss. Django serializes task arguments using JSON, so arguments need to survive a JSON round trip.

I would therefore avoid doing this:

send_invoice_email.enqueue(
    invoice=invoice,
)

A Django model instance isn't JSON serializable anyway, and passing entire model objects between processes is usually a bad idea.

Pass the identifier instead:

send_invoice_email.enqueue(
    invoice_id=invoice.pk,
)

Then retrieve the object inside the task:

@task
def send_invoice_email(invoice_id):
    invoice = Invoice.objects.get(
        pk=invoice_id,
    )

    send_mail(
        subject=f"Invoice {invoice.number}",
        message="Your invoice is ready.",
        from_email=None,
        recipient_list=[invoice.customer.email],
    )

I prefer this even in systems where richer serialization is possible. The queue message stays small, and the worker reads the current version of the database row when it actually executes.

Be careful with database transactions

Moving work into another process introduces a problem that isn't obvious when you're using the immediate backend.

Imagine we create an invoice and immediately enqueue a task:

with transaction.atomic():
    invoice = Invoice.objects.create(
        customer=customer,
        amount=100,
    )

    send_invoice_email.enqueue(
        invoice_id=invoice.pk,
    )

A real worker could receive that task before the database transaction has committed. The worker has a separate database connection, so it might try to load the invoice before the invoice is visible to it.

Django recommends enqueueing the task after the transaction commits:

from functools import partial

from django.db import transaction


with transaction.atomic():
    invoice = Invoice.objects.create(
        customer=customer,
        amount=100,
    )

    transaction.on_commit(
        partial(
            send_invoice_email.enqueue,
            invoice_id=invoice.pk,
        )
    )

This is the type of bug that can be difficult to notice during development because the immediate backend executes everything in the same process. It usually appears once a real worker is running fast enough to beat the database transaction.

We can also use queues and priorities

Django's task definition supports options such as queue names and priorities:

from django.tasks import task


@task(
    queue_name="emails",
    priority=10,
)
def send_welcome_email(email):
    ...

Or I can change some settings when enqueueing a particular task:

send_welcome_email.using(
    priority=50,
).enqueue(
    email=user.email,
)

This is useful because the application can describe how a job should be treated without knowing much about the infrastructure that executes it.

There is a catch again: the backend has to support the feature. A priority value doesn't help if the production backend can't actually prioritize messages.

What about delayed tasks?

Django tasks can also have a run_after value. For example, we could ask a compatible backend not to run a task until at least ten minutes from now:

from datetime import timedelta


send_welcome_email.using(
    run_after=timedelta(minutes=10),
).enqueue(
    email=user.email,
)

Again, this depends on the backend. Django exposes the concept, but the backend has to implement deferred execution.

This also isn't the same thing as a recurring scheduler. Saying "run this after ten minutes" is different from saying "run this every Monday at 08:00".

That difference becomes important when comparing Django's Tasks framework with Celery.

What does production look like?

At some point, we need to replace the immediate backend.

Django supports third-party task backends through the TASKS setting:

TASKS = {
    "default": {
        "BACKEND": "some_package.backends.TaskBackend",
    }
}

The actual configuration depends on the backend you choose. A production backend needs somewhere durable to store queued work and a worker that consumes those jobs outside the Django request process.

The important point is that installing Django 6 doesn't make that infrastructure disappear.

If I have:

Browser
    |
    v
Django
    |
    v
Task queue
    |
    v
Worker

Django 6 now gives me a standard interface between Django and the task system. The queue and worker still have to exist somewhere.

So did Django replace Celery?

No, at least not in the sense that I can uninstall Celery, remove Redis, delete my worker service and expect Django to take over everything.

Celery is a complete distributed task queue system. It has workers, broker integrations, retries, routing, rate limits, scheduled jobs through Celery Beat, result backends, monitoring and tools for combining multiple tasks into workflows.

Django 6 has created a standard task API inside Django. That solves a different problem.

I think of Django's Tasks framework as the interface my application code talks to. Celery has traditionally been both an API my code talks to and a large part of the infrastructure that actually processes the work.

The same task in Celery

If I were using Celery today, the equivalent email task might look like this:

from celery import shared_task
from django.core.mail import send_mail


@shared_task
def send_welcome_email(email):
    send_mail(
        subject="Welcome",
        message="Thanks for signing up.",
        from_email=None,
        recipient_list=[email],
    )

And I'd enqueue it with:

send_welcome_email.delay(
    user.email,
)

The Django 6 version isn't dramatically different:

from django.tasks import task


@task
def send_welcome_email(email):
    ...

And:

send_welcome_email.enqueue(
    email=user.email,
)

The interesting change isn't really that Django found a shorter way to decorate a function. It's that Django now has its own task abstraction, so application code doesn't necessarily have to know which task system is underneath it.

Celery still has a lot that Django Tasks doesn't

If I'm building an application with complicated background processing, Celery still gives me a mature set of tools.

Retries are a good example. A task that calls an external API might fail because the service is temporarily unavailable. Celery has built-in retry behavior and configurable retry delays.

Celery also has Celery Beat for recurring jobs. That's useful for things such as sending a report every morning, cleaning old records every night or synchronizing data every fifteen minutes.

There is also Canvas, which lets Celery build workflows out of tasks. You can create chains where one task feeds another, groups that run jobs in parallel and more complicated combinations when the application needs them.

Routing, monitoring, worker concurrency and broker support have also been developed over many years. If I already depend on those features, Django 6 hasn't given me a reason to suddenly remove Celery.

Would I use Django Tasks in a new project?

Yes, I probably would for application code where the abstraction fits.

I like the idea that my accounts app can define:

@task
def send_welcome_email(email):
    ...

Without the app having to care whether the final project uses a database-backed queue, a cloud service or some other task backend.

That separation also makes reusable apps more interesting. A reusable Django app no longer has to tell everyone "install Celery if you want this feature" just because it needs to run something outside the request cycle.

But I'd make the infrastructure decision before putting the application into production. The immediate backend is convenient locally, but it doesn't turn slow work into background work.

Would I remove Celery from an existing project?

Probably not simply because Django 6 exists.

If I have a stable Celery setup with Redis or RabbitMQ, workers managed in production, retries configured and periodic jobs already running, replacing it creates work without automatically making the application better.

I would first ask what problem I'm trying to solve.

If the answer is "Celery works fine, but Django now has Tasks", I wouldn't touch it.

If Celery has become more infrastructure than the application needs, then Django's standard Tasks API and a simpler production backend might be worth testing. That decision depends much more on what the application actually does than on which API looks cleaner.

When I would still choose Celery

I'd still reach for Celery when background jobs are an important part of the application rather than an occasional convenience.

If I need recurring schedules, complicated retry policies, several worker queues, routing rules, rate limits, task workflows or established monitoring, Celery already solves those problems.

It's also a technology that a lot of Django developers already know. There are years of documentation, production experience and answers to strange edge cases that only appear after a task queue has been running for a while.

I don't think a new Django feature makes all of that irrelevant.

When Django Tasks makes more sense to me

For simpler applications, I like what Django 6 is doing.

Maybe I only need to send emails, process an uploaded file and generate an occasional report. I don't necessarily want application code tied directly to a large task queue library for those three jobs.

Using Django's Tasks API lets me write that code against Django itself and choose a production backend separately.

It also gives Django a common vocabulary for background work. Instead of every package exposing its own Celery integration, Django apps can increasingly depend on the same @task and enqueue() concepts.

One thing I would not do

I wouldn't deploy this:

TASKS = {
    "default": {
        "BACKEND": (
            "django.tasks.backends."
            "immediate.ImmediateBackend"
        ),
    }
}

And assume I've moved my expensive work into the background.

The immediate backend executes the function during the request. If the task takes eight seconds, the request can still take eight seconds.

It's a development backend, not a tiny built-in Celery replacement.

My take on Django 6 Tasks

I think the new Tasks framework is a good addition to Django, but the name can make it sound like Django now includes more infrastructure than it actually does.

The part Django added is the common interface. We can define tasks, enqueue them, assign queues and priorities, delay them when the backend supports it and work with task results through Django's API.

The part Django intentionally didn't add is the production execution system. We still need workers and a durable queue somewhere outside the web request.

So if you're already using Celery successfully, I wouldn't start removing it just because you've upgraded to Django 6. Celery still covers a much larger set of background-processing problems.

For a new project with fairly simple background jobs, I would seriously consider starting with Django's Tasks API. It keeps the application code tied to Django rather than a particular task system, and I can choose the production backend based on what the project actually needs.

For me, that's the interesting part of the change. Django 6 hasn't made background workers disappear. It has made them less of a concern for the code that wants to use them.