How to Decouple Modules with Django Signal

Recently I was responsible for developing a backend-heavy application. The data flow in this application is complex, and the processing logic has piled up in a redundant mess. The project’s tech stack is Django + Vuejs. The frontend is bundled with Webpack, managed in modules, and mainly displays data. The backend involves many modules, many processing rules, and many data tables, so every time I had to modify an earlier feature I spent a lot of time reviewing the code. This made me realize that decoupling modules is extremely important in a complex application. Below are some reflections and practices.

1. The Observer Pattern

In practice, I mainly use Django Signal to decouple modules. Django Signal is Django’s implementation and application of the observer pattern. So it is worth first understanding the observer pattern.

The observer pattern is a type of software design pattern. People usually express their understanding of the observer pattern with the equation: publish + subscribe = observer pattern. In fact, this equation is not entirely correct.

Differences between the publish-subscribe pattern and the observer pattern:

The publish-subscribe pattern relies on a message queue (RabbitMQ, RocketMQ, ActiveMQ, Kafka, ZeroMQ, MetaMq, etc.) for communication and is asynchronous; the observer pattern is usually synchronous
The publish-subscribe pattern is loosely coupled — the publisher and subscriber may even belong to different applications; the observer pattern belongs to a single application

In terms of implementation, the observer pattern requires maintaining a subscription list. When the state changes, all objects in the list are notified automatically.

2. Django Signal

Signal is a signal dispatcher provided by the Django framework. A sender sends a signal, notifying a series of receivers, which in turn triggers the receivers to perform some operation.

Note that Django signals are synchronous. If abused, they will affect Django’s processing efficiency.

Below I will take Django 1.8.3 as an example, starting from a usage case and moving on to the source code, to introduce how Signal is implemented in Django.

2.1 A Simple Usage Case

Here is a small requirement: after a save on the MyModel table, trigger some execution logic.

Load the Signal

myApp/__init__.py

1
2
# -*- coding: utf-8 -*-
default_app_config = 'myApp.apps.MyAppConfig'

myApp/apps.py

1
2
3
4
5
6
7
8
9
# -*- coding: utf-8 -*-
from django.apps import AppConfig


class MyAppConfig(AppConfig):
    name = 'myApp'

    def ready(self):
        import myApp.signals.handlers

Bind the signal handler function

myApp/signals/handlers.py

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
# -*- coding: utf-8 -*-
from django.dispatch import receiver
from django.db.models.signals import post_save

from myApp.models import MyModel


@receiver(post_save, sender=MyModel, dispatch_uid="mymodel_post_save")
def my_model_handler(sender, **kwargs):
    # Write here the logic that runs after MyModel executes save
    pass

2.2 Understanding Django Signal’s Processing Logic from the Source Code

The example above uses very little code yet enjoys all the convenience of the signal handling mechanism Django provides. But if you stop at usage, you may not gain a deeper understanding of Django Signal. Below, let us look at Django Signal’s processing logic from the source code.

Declaring a signal

Django ships with a large number of Model-related signals that can be used directly. The signal used in the example above, post_save, is an instance of the ModelSignal class, and ModelSignal in turn inherits from the Signal class.

django/db/models/signal.py

1
2
3
4
5
6
7
8
9
from django.dispatch import Signal


class ModelSignal(Signal):
    def connect(self, receiver, sender=None, weak=True, dispatch_uid=None):
        super(ModelSignal, self).connect(
            receiver, sender=sender, weak=weak, dispatch_uid=dispatch_uid
        )
post_save = ModelSignal(providing_args=["instance", "raw", "created", "using", "update_fields"], use_caching=True)

Registering a signal handler function

The receiver function Django provides is a decorator; the decorated function is registered as a parameter into the list of receiver objects.

django/dispatch/__init__.py

1
from django.dispatch.dispatcher import Signal, receiver

django/dispatch/dispatcher.py

1
2
3
4
5
6
7
8
9
def receiver(signal, **kwargs):
    def _decorator(func):
        if isinstance(signal, (list, tuple)):
            for s in signal:
                s.connect(func, **kwargs)
        else:
            signal.connect(func, **kwargs)
        return func
    return _decorator

django/dispatch/dispatcher.py

1
2
3
4
5
class Signal(object):
    def __init__(self, providing_args=None, use_caching=False):
        self.receivers = []
    def connect(self, receiver, sender=None, weak=True, dispatch_uid=None):
        self.receivers.append((lookup_key, receiver))

Sending the signal

After the save completes, Django proactively emits the post_save signal; for a custom signal, you need to trigger it yourself.

django/db/models/base.py

1
2
3
4
5
6
7
class Model(six.with_metaclass(ModelBase)):
    # Trigger Model-related signals
    def save_base(self, raw=False, force_insert=False,
                  force_update=False, using=None, update_fields=None):
        # Signal that the save is complete
        signals.post_save.send(sender=origin, instance=self, created=(not updated),
                               update_fields=update_fields, raw=raw, using=using)

Handling the signal

Handling the signal is, in fact, simply calling the functions in the receiver list one by one.

django/dispatch/dispatcher.py

1
2
3
4
5
6
7
8
class Signal(object):

    def send(self, sender, **named):
        responses = []
        for receiver in self._live_receivers(sender):
            response = receiver(signal=self, sender=sender, **named)
            responses.append((receiver, response))
        return responses

3. Decoupling with Signals, Async Tasks

After studying the observer pattern and understanding Django Signal, you have basically mastered the fundamentals of decoupling Django modules. Next, you need to further clarify the coupling mechanism between modules and lay down project conventions, and then you can put it into practice cleanly.

Let us map out the request processing chain:

After a request passes through the access layer and middleware, the URL dispatcher matches it to the appropriate processing module, and ultimately some module is responsible for returning the response. Each module connects to the database, message queue, and object storage to persist state.

Each module consists of four parts:

AppLogic, the application logic of the module
Signal, the signals built into the module
SignalHandle, the signal handling handles the module cares about
CeleryTasks, the module’s async tasks

Modules are coupled to each other entirely through signals:

Because Django Signal is a synchronous processing mechanism, you can combine it with Celery and RabbitMQ to support asynchronous processing.

Below is an example of a signal handling asynchronous logic:

myApp/tasks.py

1
2
3
4
5
6
7
# -*- coding: utf-8 -*-
from celery import task


@task(ignore_result=True)
def my_task(instance):
    pass

myApp/signals/handlers.py

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
# -*- coding: utf-8 -*-
from django.dispatch import receiver
from django.db.models.signals import post_save

from myApp.models import MyModel
from myApp.tasks import my_task


@receiver(post_save, sender=MyModel, dispatch_uid="mymodel_post_save")
def my_model_handler(sender, **kwargs):
    instance = kwargs['instance']
    # async
    my_task.apply_async(args=[instance])
    # sync
    pass