Django Full-Stack Optimization Guide

As the data volume exploded, the system responded very slowly. We carried out a series of optimizations on the application, and the system’s response time improved by an order of magnitude. Overall, optimizations in file compression and faster network access gave a noticeable boost to frontend performance, while optimizations in stored procedures, caching, and logic code gave a noticeable boost to backend performance. This article collects the optimization ideas and methods.

1. Mapping the Chain

Before optimizing, mapping the entire chain is especially important.

Optimization is a systems-engineering effort; you cannot get great results from a simple add or remove. Nor did we deliberately design the system to be slow — rather, the preconditions under which the system was designed have changed, and the whole system needs to be re-examined. Optimization is about finding those changes and letting the system adapt to them.

The figure below lays out the whole system chain: an access layer, a logic layer, and a storage layer.

2. Project Background and Optimization Approach

This is a Web application combining a store and a community forum, with a mini-program client as well. It has to support not only product publishing, updates, transactions, and day-to-day operations, but also social features such as community Q&A, follows, and likes. For certain historical reasons, the system has as many as 142 database tables.

In one phrase: big and messy. At the same time, there was only one frontend developer and one backend developer.

The project was developed with a frontend/backend separation model, and the optimization followed the same idea. Frontend and backend are handled separately: first use some analysis tools to find the time-consuming nodes along the chain, then optimize.

Below are some optimization measures already taken or about to be taken.

3. Frontend

3.1 Optimizing Based on PageSpeed Insights Analysis

PageSpeed Insights is a website analysis and optimization tool developed by Google. There is a web version and a Google Chrome extension version.

After installing the extension, open the developer tools with F12 and click [ANALYZE]. You will see PageSpeed’s score and suggested changes for the page.

Clicking each suggestion shows more detail. PageSpeed Insight’s analysis covers the following areas:

Optimize caching — keep your app’s data and logic entirely off the network
Reduce response time — reduce the number of request-response round trips
Reduce request size — reduce upload size
Reduce payload size — reduce response, download, and cached page sizes
Optimize browser rendering — improve the browser’s page layout

3.2 Hosting Static Files on a CDN

The main advantages of CDN technology:

Faster access
Users connect to the nearest CDN server to get content, which gives a noticeable speedup.
Availability
Under high-stress conditions such as excessive user traffic, intermittent peaks, and potential server failures, the CDN still ensures users can get the content.
Security
CDN services can effectively mitigate various attack behaviors.

Hosting frontend static files on a CDN is a very important part of frontend optimization. Of course, this kind of change requires support from the deployment system.

3.3 Splitting Frontend Static Files by Page

Splitting the frontend’s bundled static files lets users load on demand when they visit the app, rather than requesting every file on the first page load. The frontend uses webpack 3.6.0; just add the following to the plugins property of webpackConfig:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
(new webpack.optimize.ModuleConcatenationPlugin(),
  new webpack.optimize.CommonsChunkPlugin({
    name: "vendors",
    minChunks(module) {
      return (
        module.resource &&
        /\.js$/.test(module.resource) &&
        module.resource.indexOf(path.join(__dirname, "../node_modules")) === 0
      );
    },
  }),
  new webpack.optimize.CommonsChunkPlugin({
    name: "manifest",
    minChunks: Infinity,
  }),
  new webpack.optimize.CommonsChunkPlugin({
    name: "app",
    async: "vendor-async",
    children: true,
    minChunks: 3,
  }));

3.4 Reducing Unnecessary API Calls

Because frontend developers changed frequently and handovers were common, the implementation logic was inconsistent. At the same time, frontend code quality varied, and the implementation was hard to extend and maintain. The following situations mainly exist:

An interface already called in a common module is called again in a routed sub-page
An interface that belongs to only one routed sub-page is called by every page
The same interface is called multiple times on a page

This part mainly requires frontend developers to review the code and optimize at the code-logic level.

4. Backend

This section is divided into two parts. The first introduces several performance analysis tools; building on that analyzed data, the second half focuses on how to optimize.

django-debug-toolbar is a good Django performance inspection tool. It mainly provides performance checks in the following areas:

How many SQL statements were executed
How much time was spent on the database
What special query operations ran, and how long each query took
Which code generated those queries
Which templates were used to render the page
How cold/hot caching affects performance

Install

1
pip install django-debug-toolbar

Configure

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
INSTALLED_APPS = [
    # ...
    'django.contrib.staticfiles',
    # ...
    'debug_toolbar',
]
MIDDLEWARE = [
    # ...
    'debug_toolbar.middleware.DebugToolbarMiddleware',
    # ...
]

url.py

1
2
3
4
5
6
7
8
from django.conf import settings
from django.conf.urls import include, url

if settings.DEBUG:
    import debug_toolbar
    urlpatterns = [
        url(r'^__debug__/', include(debug_toolbar.urls)),
    ] + urlpatterns

Once configured successfully, you can see the related performance inspection data:

4.2 Analyzing Ajax with django-debug-panel

django-debug-panel builds on django-debug-toolbar and offers better support for single-page applications and Ajax requests.

Install

1
pip install django-debug-panel

Configure

settings.py

1
2
3
4
INSTALLED_APPS = (
    # ...
    'debug_panel',
)

Use panel’s middleware in place of toolbar’s middleware.

middlewares.py

1
2
3
4
5
6
MIDDLEWARE_CLASSES = (
    ...
    # 'debug_toolbar.middleware.DebugToolbarMiddleware',
    'debug_panel.middleware.DebugPanelMiddleware',
    ...
)

Install the Chrome extension: Django Debug Panel

Once configured successfully, you can see the related performance inspection data:

Often we use the requests library to call third-party APIs. The performance of those third-party APIs also needs attention. django-debug-toolbar-requests builds on django-debug-toolbar and adds support for requests calls.

Install

1
pip install django-debug-toolbar-requests

Configure

1
2
3
4
INSTALLED_APPS =（
    # ...
    'requests_toolbar'，
）

Add RequestsDebugPanel to DEBUG_TOOLBAR_PANELS

1
2
3
4
DEBUG_TOOLBAR_PANELS = (
    # ...
    'requests_toolbar.panels.RequestsDebugPanel',
)

Once configured successfully, you can see the related performance inspection data:

4.4 Django Database Optimization

Add indexes to frequently searched fields

When defining a Model in Django, you can add an index to a field with db_index=True:

1
2
3
class Country(models.Model):
    name = models.CharField(unique=True, db_index=True, max_length=50)
    short_name = models.CharField(unique=True, max_length=5)

Adding indexes to key fields gives a big performance boost, especially when the data volume is large and there are many query operations. See this document for reference

But indexes are not “the more the better.” An index can greatly speed up data queries, but it slows down inserts, deletes, and updates to the table. Striking the right balance matters.

Use QuerySets caching sensibly

Django’s QuerySets have a cache; once fetched, they stay in memory for a while. Look at this example:

1
2
3
entry = Entry.objects.get(id=1)
entry.blog   # Blog object is retrieved at this point
entry.blog   # cached version, no DB access

1
2
3
entry = Entry.objects.get(id=1)
entry.authors.all()   # query performed
entry.authors.all()   # query performed again

Calls to functions such as all, count, and exists require connecting to the database, but attribute access does not. For custom attributes, use cached_property to add a caching strategy — see this document for reference.

Use iterator() to fetch objects and avoid QuerySets memory consumption

1
2
3
allbooks = Book.objects.filter(author = 'chenshaowen')
for book in allbooks.iterator():
    do_something(book)

A QuerySet caches its results so that repeated evaluation does not cause extra queries. iterator(), by contrast, reads results directly and performs no caching at the QuerySet level. For a QuerySet that returns a large number of objects that are accessed only once, this gives better performance and significantly reduces memory usage.

Prefer batch operations

Batch operations can effectively reduce the number of database connections.

Batch insert data

1
2
3
4
product_list_to_insert = list()
for x in range(10):
    product_list_to_insert.append(Product(name='product name ' + str(x), price=x))
Product.objects.bulk_create(product_list_to_insert)

Batch update data

1
Product.objects.filter(name__contains='name').update(name='new name')

Batch delete data

1
Product.objects.filter(name__contains='name query').delete()

Many-to-many relations

1
my_band.members.add(me, my_friend)

4.5 Using Cache Sensibly

Caching is a sharp tool for optimizing performance. The idea behind caching is to trade space for time and avoid repeated computation.

Common caching approaches in Django:

Development/debug cache
In-memory cache
File cache
Database cache
Memcache (using the python-memcached module)
Memcache (using the pylibmc module)

By granularity, caching can be divided more finely:

Site-wide cache

Just add the middleware:

1
2
3
4
5
    ... += [
    'django.middleware.cache.UpdateCacheMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.cache.FetchFromCacheMiddleware',
    ]

View-level cache

1
2
3
4
5
from django.views.decorators.cache import cache_page

@cache_page(60 * 15)
def my_view(request):
    pass

Template fragment cache

1
2
3
4
{% load cache %}
{% cache 500 sidebar %}
    .. sidebar ..
{% endcache %}

Custom cache

The django.core.cache.caches provided by Django lets users set up and maintain caches themselves.

1
2
3
4
5
>>> from django.core.cache import caches
>>> cache1 = caches['myalias']
>>> cache2 = caches['myalias']
>>> cache1 is cache2
True

Cache in threads

Besides Django’s built-in caching mechanism, Python’s dynamic nature allows even finer-grained caching at the thread level:

1
2
3
4
5
6
7
8
9
def get_value(self,cache_key,default=None):
    value=getattr(local,cache_key,None)
    if value:
        return value
    value=cache.get(cache_key,default=None)
    if value is None:
        return default
    setattr(local,cache_key,value)
    return value

Caching data in the current thread’s local works well for scenarios that access the same cache multiple times within a single request.

In effect, this is using an in-memory cache.

4.4 Code Logic Optimization

Remember to compile regular expressions

1
re.compile(…)

Prefer .sort() for sorting
- O(nlogn)
- Using key is more efficient than cmp
Use list iterable expressions

1
2
3
4
5
6
def function1(l):
    result = []
    for i in l:
        if i % 2 == 0:
            result.append(i)
return result

The list comprehension is 36% faster than the above

1
2
def function2(l):
    return [i for i in l if i % 2 == 0]

Reduce function calls, prefer accessing local variables

1
2
3
4
5
6
7
8
def add_two(i):
    return i + 2

def function1(l):
    result = []
    for item in l:
      result.append(chr(add_two(item)))
    return result

After reducing function calls, it is 40% faster than the above

1
2
3
4
5
6
def function2(l):
    result = []
    lchr = chr
    for item in l:
        result.append(lchr(item + 1))
    return result

Use a set to test element existence

1
2
3
4
l = range(10000)

def function1():
    return 9000 in l

After using a set, it is 70000% faster than the above

1
2
3
4
s = set(range(10000))

def function2(item):
    return item in s