Compiled from the “Development Tips” series, collecting common problems and solutions in Python and Django development.

1. Sorting Python Iterables

Iterable objects fall into 3 main categories:

• All sequence types, such as list, str, and tuple
• Some non-sequence types, such as dict and file
• Objects that contain an __iter__() or __getitem__() method

sort is a method of the list object, while sorted can sort any iterable object. The difference is that sort rearranges the list in place, whereas sorted() produces a new list. Basic concepts related to sorting:

• iterable
  The iterable object
• cmp
  Comparison function, requires two arguments
• key
  The element used for comparison, takes only one argument
• reverse
  Sort rule: reverse = True for descending, reverse = False for ascending

1.1 The sort Function

Prototype:

1

list.sort(cmp=None, key=None, reverse=False)

Usage example:

1
2
3
4

my_list = ['3', '4', 'u', 9]
my_list.sort(reverse=True)
print my_list
['u', '4', '3', 9]

1
2
3
4
5
6

def takeSecond(elem):
    return elem[1]
mylist = [(2, 2), (3, 4), (42, 1), (41, 1), (1, 3)]
mylist.sort(key=takeSecond)
print mylist
[(42, 1), (41, 1), (2, 2), (1, 3), (3, 4)]

1.2 The sorted Function

Prototype:

1

sorted(iterable, cmp=None, key=None, reverse=False)

Usage example:

1
2
3

mylist = [5, 2, 3, 1, 4]
print sorted(mylist, reverse=True)
[5, 4, 3, 2, 1]

1
2
3

mylist = [('b', 2), ('a', 1), ('c', 3), ('d', 4)]
print sorted(mylist, cmp=lambda x, y: cmp(x[1], y[1]))
[('a', 1), ('b', 2), ('c', 3), ('d', 4)]

You can specify multiple sort keys: first by x[1], then by x[0].

1
2
3

mylist = [('d', 2), ('a', 4), ('b', 3), ('c', 2)]
print sorted(mylist, key=lambda x: (x[1], x[0]))
[('c', 2), ('d', 2), ('b', 3), ('a', 4)]

2. Serialization and Deserialization in Python

Serialization is the process of converting an in-memory object into a storable or transmittable sequence. Deserialization is the process of converting that serialized sequence back into an in-memory object. Json and Pickle are the two serialization modules commonly used in Python.

Json VS Pickle:

• Json converts between in-memory objects and Json strings; Pickle converts between in-memory objects and byte objects
• The Json format is widely used outside Python as well; Pickle is unique to Python
• Json can only serialize Python’s built-in basic data-type objects, while Pickle can serialize any object, including functions
• cPickle is the C-language implementation of Pickle, commonly used to replace Pickle for better performance

Usage example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

# -*- coding: utf-8 -*-
import json
import pickle


obj = {'a': 'b', 'c': 'd'}
ps = pickle.dumps(obj)
print ps
# "(dp0\nS'a'\np1\nS'b'\np2\nsS'c'\np3\nS'd'\np4\ns."
js = json.dumps(obj)
print js
# {"a": "b", "c": "d"}
print pickle.loads(ps)
# {'a': 'b', 'c': 'd'}
print json.loads(js)
# {u'a': u'b', u'c': u'd'}

In addition, strings can be compressed to save storage:

1
2
3
4
5
6
7
8

import cPickle as pickle
import zlib

# 序列化，并压缩
compressed = zlib.compress(pickle.dumps(obj))

# 解压缩，反序列化
obj = pickle.loads(zlib.decompress(compressed))

3. Python’s Logging Module

Python’s logging module consists mainly of four parts:

1. Loggers: the interface programs can call directly
2. Handlers: output log records to the appropriate destination
3. Filters: provide finer-grained decisions on whether a log is emitted
4. Formatters: customize the layout of the final printed record

Look at the following example, the file log1.py

1
2
3
4
5
6
7
8

import logging

logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(name)s - %(levelname)s - %(message)s')
logger = logging.getLogger(__name__)

logger.info('info log')
logger.debug('debug log')
logger.warning('Warning log')

Running it directly outputs:

1
2

2018-10-26 21:08:02,905 - __main__ - INFO - info log
2018-10-26 21:08:02,907 - __main__ - WARNING - Warning log

The file log2

1

from log1 import logger

Running it directly outputs:

1
2

2018-10-26 21:11:36,849 - log1 - INFO - info log
2018-10-26 21:11:36,849 - log1 - WARNING - Warning log

Some format parameters:

• %(levelno)s: prints the numeric log level
• %(levelname)s: prints the log level name
• %(pathname)s: prints the path of the currently executing program, which is actually sys.argv[0]
• %(filename)s: prints the name of the currently executing program
• %(funcName)s: prints the current function of the log
• %(lineno)d: prints the current line number of the log
• %(asctime)s: prints the time of the log
• %(thread)d: prints the thread ID
• %(threadName)s: prints the thread name
• %(process)d: prints the process ID
• %(processName)s: prints the thread name
• %(module)s: prints the module name
• %(message)s: prints the log message

4. Python Debugging Tools

• pdb

The Python Debugger is the official debugger, built into the Python standard modules.

Usage: $python -m pdb scriptfile or pdb.set_trace() in the code

• ipdb

An ipython-based pdb, an enhanced version of pdb.

Usage: $ipdb scriptfile or $python –pdb scriptfile

• PuDB

A full-screen, console-based visual debugger.

Usage: $python -m pudb.run scriptfile or $pudb scriptfile

5. Django CSRF

django.middleware.csrf.CsrfViewMiddleware processing logic:

Before entering the views function for processing, if there is a csrf token in the cookies, it is set in request.Meta; otherwise a token is generated. If the method is not GET, HEAD, etc., CSRF is validated.

Validation rule: the csrf token taken from the Cookie is compared with the csrfmiddlewaretoken or HTTP_X_CSRFTOKEN in the POST request. If the two are equal, validation passes; otherwise a 403 is returned.

Before returning the response, if CSRF_COOKIE_USED has been set, the csrf token is set into the Cookie. There are usually two ways to set CSRF_COOKIE_USED:

• Directly use the django.middleware.csrf.get_token function to obtain the csrf token
• Configure the django.template.context_processors.csrf context processor to render the csrf token in the template, which in fact still calls the get_token function

Below is an excerpt from part of the Django source:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

class CsrfViewMiddleware(object):
    def process_view(self, request, callback, callback_args, callback_kwargs):
        try:
            csrf_token = _sanitize_token(
                request.COOKIES[settings.CSRF_COOKIE_NAME])
            request.META['CSRF_COOKIE'] = csrf_token
        except KeyError:
            request.META["CSRF_COOKIE"] = _get_new_csrf_key()
        if request.method == "POST":
                request_csrf_token = request.POST.get('csrfmiddlewaretoken', '')
            if request_csrf_token == "":
                request_csrf_token = request.META.get('HTTP_X_CSRFTOKEN', '')
            if not constant_time_compare(request_csrf_token, csrf_token):
                return self._reject(request, REASON_BAD_TOKEN)

    def process_response(self, request, response):
        if request.META.get("CSRF_COOKIE") is None:
            return response

        if not request.META.get("CSRF_COOKIE_USED", False):
            return response
        response.set_cookie(settings.CSRF_COOKIE_NAME,
                            request.META["CSRF_COOKIE"],
                            max_age=settings.CSRF_COOKIE_AGE,
                            domain=settings.CSRF_COOKIE_DOMAIN,
                            path=settings.CSRF_COOKIE_PATH,
                            secure=settings.CSRF_COOKIE_SECURE,
                            httponly=settings.CSRF_COOKIE_HTTPONLY
                            )
        return response

Parameter explanations:

• max_age: the lifetime of the cookie, defaults to None; the cookie expires immediately when the browser closes
• expires: the point in time when the cookie expires, defaults to None; the cookie expires immediately when the browser closes
• path: the path where the Cookie takes effect; / means the root path, and a cookie on the root path can be accessed by pages at any url
• domain: defaults to None, the domain of the page that sets the Cookie is None
• secure: used to make the Cookie sent only over requests guaranteed to be secure. When the request is HTTPS or another secure protocol, a Cookie with the secure option can be saved to the browser or sent to the server.
• httponly: can only be transmitted over the http protocol and cannot be read by JavaScript (not absolute; it can be captured at a lower level by packet sniffing and can also be overridden). Defaults to False.

Django’s default configuration for CSRF:

1
2
3
4
5
6
7

# Settings for CSRF cookie.
CSRF_COOKIE_NAME = 'csrftoken'
CSRF_COOKIE_AGE = 60 * 60 * 24 * 7 * 52
CSRF_COOKIE_DOMAIN = None
CSRF_COOKIE_PATH = '/'
CSRF_COOKIE_SECURE = False
CSRF_COOKIE_HTTPONLY = False

6. The djcelery_crontabschedule already exists Error

Versions used:

• Django==1.8.3
• celery==3.1.18
• django-celery==3.1.16

When upgrading to django-celery==3.2.2, running python manage.py migrate reports an error:

1
2
3

File "/app/.heroku/python/lib/python2.7/site-packages/pymysql/err.py", line 115, in _check_mysql_exception
    raise InternalError(errno, errorvalue)
django.db.utils.InternalError: (1050, u"Table 'djcelery_crontabschedule' already exists")

The django-celery GitHub changelog mentions:

Starting with the 3.1.17 release, Django migrations were added.

If tables such as djcelery_* already exist, running $python manage.py migrate will then report an error. When upgrading dependency library versions, keep version compatibility in mind.

7. Handling Update Signals for Django Models

The created field lets you distinguish between create and update operations on a Django Model.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

from django.db.models.signals import post_save


@receiver(post_save, sender=User)
def handle_when_user_updated(sender, instance, created, **kwargs):
    if not created:
        # User object updated
        pass
    else:
        # User object created
        pass

8. Serving Static Files with WhiteNoise

Django’s built-in static file server is very inefficient, and WhiteNoise is a good replacement. It has the following characteristics:

• Usually used for PaaS services
• Supports wsgi applications, with special adaptation for Django
• Works better together with a CDN
• Combined with Gunicorn, it uses the sendfile system call, giving very high processing efficiency
• Compared with Nginx, WhiteNoise serves static files in a simpler way, but its efficiency is only 15% of Nginx’s

9. Recording Django Model Modification History

• django-simple-history

  How it works: each model that needs tracking requires its own separate table, and when an instance is modified a new record is created directly in that table.

• django-reversion

  How it works: when model data is modified, the modification is serialized into a Version table. Only one table is needed to record all modification history.

10. Python Memory Analysis Methods

Mainly involves four tools:

• memory_profile: analyzes the memory usage of each line of code
• objgraph: traces the relationships between objects in memory
• guppy: tracks heap usage at runtime
• pyrasite: injects code into a process

It is done in two steps:

1. Simulate the production environment, using pyrasite and guppy to obtain heap information
2. Based on the information from the previous step, locate a specific block in the code, then use memory_profile or objgraph to do further analysis

11. Python Multiple Inheritance

When calling methods in Python multiple inheritance, a depth-first search algorithm is used. In order, it searches one parent class deeply all the way down, then searches the second parent class, until the method is found.

• Old-style classes

Old-style class lookup order: C(A, B) => C -> A -> B

• New-style classes

One problem with new-style classes is that any subclass of multiple inheritance has a common ancestor class, Object, so it is always diamond inheritance. New-style classes look up methods in the order of the classes stored in Class.**mro**.

New-style class lookup order: C(A, B) => C -> A -> B -> Object

12. Replacing MySQL-python with pymysql

MySQL-python has stopped being updated, only supports Python2 and not Python3, and has complex dependencies that make it troublesome to install.

pymysql was created to replace MySQL-python. It is written purely in Python, its interface is compatible with MySQL-python, it is easy to install, and it supports Python3.

The replacement method is as follows:

1
2
3

import pymysql

pymysql.install_as_MySQLdb()

13. Connecting to Databases from Python3

Python3 mainly has two database connection clients: mysqlclient and PyMySQL.

• mysqlclient is implemented in C
• PyMySQL is implemented in Python

In terms of performance, mysqlclient is an order of magnitude faster than PyMySQL. However, under PyPy, PyMySQL and mysqlclient perform about the same.

If you need to use gevent or eventlet monkeypatched sockets, then choose PyMySQL.

14. MySQL Error Table ‘performance_schema.session_variables’ doesn’t exist

Running the following commands resolves it:

1
2
3

mysql -u root -p
mysql> set @@global.show_compatibility_56=ON;
Query OK, 0 rows affected (0.00 sec)

Reference: Documented SHOW command behavior for show_compatibilty OFF and PFS builds

15. Exception Handling in Python2 and Python3

Two forms supported by both Python2 and Python3:

• With an argument

1
2

except （ExceptionType） as Argument:
    # 访问 Argument

• Without an argument

1

except ExceptionType：

A form only supported by Python2:

1
2

except ExceptionType, Argument:
    # 访问 Argument

16. get_object_or_404 and get_queryset in Django

get_object_or_404 retrieves an object using get, otherwise returns a 404

1
2
3
4
5
6
7
8
9

from django.shortcuts import get_object_or_404
from django.forms.models import model_to_dict
from django.http import JsonResponse
from .models import Fruit


def filter404(request):
    obj = get_object_or_404(Fruit, title='aa')
    return JsonResponse(model_to_dict(obj))

Using get_queryset lets you customize query behavior globally, including in admin

1
2
3

class FruitManager(models.Manager):
    def get_queryset(self):
        return super(FruitManager, self).get_queryset().filter(is_delete=False)

17. Tree Structure Storage

• Materialized path

Each node stores the encoding of its complete path.

1
2
3
4
5

Name        Path
William     1
Jones       1/1
Blake       1/2
Adams       1/2/1

The advantage is that both reads and writes are very fast.

• Adjacency list model

The adjacency list representation stores the tree by keeping links to some adjacent nodes.

1
2
3

Name        Parent     Next
William     null       Joseph
Jones       William    Blake

The advantage is that the structure is simple and easy to understand, but when the data volume is large, recursion-based queries are very inefficient.

• Nested set

Each node stores some indices (usually left and right values).

1
2
3
4

Name        left   right
William     1      10
Jones       2      3
Blake       4      7

The advantage is that querying data is very fast, but on update a large number of nodes need to be modified.

• Interval nested set

The intervals are mapped into two-dimensional space, and each node corresponds to a score according to a rule. Hierarchical queries involving node positions do not need to access the database. The advantage is that performance is very good, but there is a bit of a barrier to implementing and understanding it.

django-treebeard implements materialized paths, nested sets, and adjacency lists. django-mptt mixes nested sets and adjacency lists, can efficiently query child nodes, and can rebuild the tree when it is corrupted.

18. Pytest Cannot Find Module Error

Error message:

1
2
3
4
5
6

=================================== ERRORS ====================================
_____________ ERROR collecting home_application/test/test_mptt.py _____________
ImportError while importing test module 'C:\pytest\home_application\test\test_mptt.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
ImportError: No module named home_application.test.test_mptt

Cause:

Under the test directory, an __init__.py file exists, which causes Pytest to treat the whole test as a single module. What we actually need is for Pytest to enter the directory, find files starting with test_, and run the tests. Simply deleting the __init__.py file is enough.

19. Several Ways to Continue Lines in Python

1, Using a backslash

1
2
3
4
5
6
7
8
9

a = 'sdfaf' \
    'test'

a = '1' + '2' + '3' + \
    '4' + '5'

if False and \
    True:
    pass

2, Using parentheses

1
2
3
4
5
6
7
8
9

a = ('sdfaf'
    'test')

a = ('1' + '2' + '3' +
    '4' + '5')

if(False and
    True):
    pass

20. Python Lists and Tuples Iterate at About the Same Speed

Run the following test code in IPython:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

import timeit

a = range(9999)

def test_list():
    for i in a:
        i = i * i

timeit.timeit('test_list()', 'from __main__ import test_list', number=1000)
# 0.29664087295532227

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

import timeit

a = tuple(range(9999))

def test_tuple():
    for i in a:
        i = i * i

timeit.timeit('test_tuple()', 'from __main__ import test_tuple', number=1000)
# 0.3050811290740967

From the test results, list and tuple iterate at about the same speed. But you often hear the claim that tuple is faster than list, which actually refers to creation speed. In addition, their lookup speeds are about the same too.

In CPython, creating a tuple allocates a fixed, contiguous block of memory all at once; creating a list allocates two blocks of memory, one recording Python Object information and one used to store data.

21. Python’s Class Constructor type()

1

type(name, bases, dict)

Parameter descriptions:

• name, a string, specifies the name of the new class, assigned to the new class’s __name__
• bases, a tuple, specifies the base classes of the new class, assigned to the new class’s __bases__
• dict, a dictionary type, specifies the attributes of the new class, assigned to the new class’s __dict__

As a dynamic language, Python can build classes dynamically to achieve many wonderful features and save a great deal of code.

22. The EAFP Principle in Python

Easier to ask for forgiveness than permission. This common Python coding style assumes the existence of valid keys or attributes and catches exceptions if the assumption proves false. This clean and fast style is characterized by the presence of many try and except statements. The technique contrasts with the 07001 common to many other languages such as C.

For example:

EAFP style

1
2
3
4

try:
    x = my_dict["key"]
except KeyError:
    # handle missing key

LBYL style

1
2
3
4

if "key" in my_dict:
    x = my_dict["key"]
else:
    # handle missing key

LBYL requires searching the dictionary twice, and its readability is also worse than EAFP.