This page looks best with JavaScript enabled

Getting Started with Character Encoding in Python

 ·  ☕ 3 min read

Background: I had just finished a Django data-query web project. The data came from internal API queries, and every query had to call several APIs to fetch the data and render it on the front-end page. Since the relevant data does not change often, I designed a cache to improve front-end response speed and to keep queries working even when the API was unavailable. The data returned by the API is in JSON format, while the cached data uses MySQL’s Unicode encoding. That gave the data two sources — the API and MySQL — which led to encoding errors.

1. Encoding Formats

1.1 ASCII

A computer can only process two digits, 0 and 1, so to let it handle text information (that is, character strings) this information has to be encoded. The earliest computer encoding was ASCII, which uses 8 bits to represent one letter and defines 128 characters, 33 of which cannot be displayed. Eight bits can represent at most 255 characters, which is only enough for English-speaking countries — clearly not enough for the whole world.

1.2 GB2312

To meet the need for computers to process Chinese characters, China published the GB2312 encoding standard, which uses two bytes — 16 bits — to represent one graphic character. It includes 6,763 Chinese characters, of which 3,755 are Level 1 and 3,008 are Level 2; it also includes 682 characters covering Latin letters, Greek letters, Japanese hiragana and katakana, and Russian Cyrillic letters.

1.3 GBK

Because GB2312 does not cover all Chinese characters — some ancient Chinese glyphs and special characters are missing — GBK uses two bytes, 16 bits, to represent one graphic character, and is fully compatible with GB2312. GBK includes a total of 21,886 Chinese characters and graphic symbols.

1.4 Unicode

To let computers process their own national languages, each country defined its own encoding standard, and conflicts of various kinds were inevitable. To unify encoding, Unicode emerged, using multi-byte encoding. The Unicode standard keeps evolving; the common form uses two bytes — 16 bits — to represent a character, and uncommon characters can be extended to multiple bytes.

1.5 UTF-8

To represent more characters, Unicode uses long-byte encoding, which wastes storage and bandwidth. UTF-8 emerged in response, using a variable-length encoding scheme; in fact UTF-8 is one implementation of the rules of the Unicode character set. If the first bit of a byte is 0, it represents a single character; if the first bit of a byte is 1, the encoding extends to the next byte for the decision. This encoding scheme made UTF-8 spread rapidly.

1.6 Usage Scenarios for Various Encodings

ASCII: suitable for English-language environments
GB2312, GBK: suitable for Chinese character encoding
Unicode: general-purpose, often used as an intermediate encoding when converting other encodings
UTF-8: saves storage space and bandwidth, and is very widely used

2. Python Encoding

Python’s default script encoding is ASCII. If non-ASCII characters are used, the encoding is usually declared on the first or second line, # -*- coding=utf-8 -*- or #coding=utf-8. Other character sets such as gbk can be used, but keeping utf8 is strongly recommended.

2.1 Dictionary Type

A dictionary is a mutable container model that can store objects of any type. Keys must be unique, but values need not be. The format is as follows:

1

    d = {key1 : value1, key2 : value2 }

Run under ipython; the returned type:

1
2
3
4
5

    In [1] :dic_a = ｛'name':'tom'｝
    In [2] :type(dic_a)
    Out [2] :dict
    In [3] :dic_a['name']
    Out [3] :'tom'

2.2 Lists and Tuples

A list’s data items do not need to be of the same type; a list is defined with comma-separated units inside square brackets, and duplicate elements are allowed. The format is as follows:

1

 list= ['value1', 'value2', num1, num1]

Run under ipython:

1
2
3
4
5

    In [1] :list_a = ['a','b',1,2]
    In [2] :type(list_a)
    Out [2] :list
    In [3] :list_a[0]
    Out [3] :'a'

A tuple is defined with comma-separated units inside parentheses, but a tuple’s contents cannot be changed. The format is as follows:

1

 tuple= ('value1', 'value2', num1, num1)

Run under ipython:

1
2
3
4
5

    In [1] :tuple_a = ('a','b',1,2)
    In [2] :type(tuple_a)
    Out [2] :tuple
    In [3] :tuple_a[0]
    Out [3] :'a'

2.3 Python’s String Data Types

Python has two string types: the str type (a sequence of 8 bits) and the unicode type (each unit is a unicode object). When Python reads text content, the object it keeps is of type str. A string encoded as Unicode is written with u’'.

1
2
3
4
5
6

    In [1] :u'我'
    Out [1] :u'\u6211'
    In [2] :u'我'.encode('gbk')
    Out [2] :'\xce\xd2'
    In [3] :u'我'.encode('utf8')
    Out [3] :'\xe6\x88\x91'

The encodings above are shown in hexadecimal; using print prints the represented character directly. As you can see, the characters are the same, but different encodings cost different amounts of storage.

2.4 JSON Data

A json string is in fact a string, only wrapped in single quotes, but it must follow certain character rules. There are four:

  • Parallel data items are separated by a comma (", “)
  • A mapping is expressed with a colon (” : “)
  • A collection of parallel data items (an array) is expressed with square brackets (”[]")
  • A collection of mappings (an object) is expressed with curly braces ("{}")

The json module provides two functions, json.dumps() and json.loads(), to encode and decode json data. They are very simple to use; note that after dumps a tuple becomes a list, and after loads it is not fully restored. The json module also provides the dump and load functions, which apply when json needs to be stored to a file or a socket, whereas when handling it as a string, use the dumps and loads functions. One more thing to note: loads encodes the string as Unicode, so if you need to keep the original encoding, use ast.literal_eval.
Run under python; the returned types:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

    In [1] :import json
    In [2] :obj_a={'name':'tom','age':20,'position':(1,2,3)}
    In [3] :json_a = json.dumps(obj_a)
    In [4] :json_a
    Out [4] :'{"position":[1,2,3],"age":20,"name":"tom"}'
    In [5] :type(json_a)
    Out [5] :str
    In [6] :obj_b=json.loads(json_a)
    In [7] :obj_b
    Out [7] :{u'age': 20, u'name': u'tom', u'position': [1, 2, 3]}
    In [8] :type(obj_b)
    Out [8] :dict
    In [9] :ast.literal_eval(json_a)
    Out [9] : {'age':20,'name':'tom','position':[1,2,3]}
    In [10] :type(ast.literal_eval(json_a))
    Out [10] : dict

Python also provides the ord() and chr() functions to convert between letters and numbers.

3. MySQL Character Encoding

3.1 Basic Concepts

MySQL character sets involve two concepts: the character set and the collation. The character set defines how MySQL stores strings, and the collation defines how strings are compared. Character sets and collations have a one-to-many relationship; MySQL supports more than 30 character sets and more than 70 collations.

3.2 Choosing a Collation

Using utf8_general_ci is fast, while utf8_unicode_ci is more accurate. utf8_unicode_ci’s accuracy mainly shows in German and French; for typical domestic application scenarios, the general rule is enough. In addition, since unicode and general are both case-insensitive, utf8_bin_ci also has its own use cases.

3.3 Notes

Usually choosing the utf8 character set directly is enough. But errors occur when there is a need to store Emoji. MySQL’s utf8 encoding is at most 3 bytes, whereas an Emoji is 4 bytes. To be able to store Emoji, use the utf8mb4 character set.

4. References

微信公众号
WRITTEN BY
微信公众号

See Also