Haystack Full-Text Search

A quick word on the project requirements: the team needed to publish documentation externally. The documents are written in Markdown and need to be published as HTML. At first we used an Nginx + Jekyll solution. As the documentation grew, the document system developed a strong need for search. I discussed this in another article, but among those approaches some produced unsatisfactory search results and others depended on additional services, which felt rather heavy. Hence the implementation described in this article.

1. Tool Introduction

Whoosh is a full-text search component implemented in pure Python. Whoosh is not only feature-complete but also very fast.
Haystack is a third-party Django app that provides full-text search. It can index and search the content of a Model. At the same time, Django-haystack supports four full-text search engine backends: Whoosh, Solr, Xapian, and Elasticsearch. In essence it is a full-text search framework, and you are free to choose and combine on use.
Jieba is a Python Chinese word segmentation component with many features; this article uses its ChineseAnalyzer Chinese segmentation capability.

2. Design

Approach

1. Use Jekyll as the tool for converting Markdown to HTML, ultimately obtaining local WYSIWYG HTML documents
1. Use the Python scraping tool BeautifulSoup to parse the static HTML and import it into the DB
1. Use Jieba for word segmentation and Whoosh to build the query index
1. Have Django match the .html URLs directly and fetch data from the database to serve the documents externally, which ensures the links from the Nginx + Jekyll approach remain valid.

3. Implementation

3.1 Creating the document app

In the project directory, create a Django app named: document. The document system has a two-level directory structure, where the first level is the category and the second level is the document.

For example:

doc/type1/aaa.html
doc/type2/bbb.html

document/models.py

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
from django.db import models
class Document(models.Model):
    '''
    @summary: jekyll生成的文档
    '''
    file_name = models.CharField(u'文件名', max_length=255)
    uri = models.CharField(u'URI', max_length=255)
    tag = models.CharField(u'标签', max_length=255)
    title = models.CharField(u'标题', max_length=255)
    doc_html_text = models.TextField(u'文档（txt格式）')
    doc_html = models.TextField(u'文档（HTML格式）')
    doc_html_all = models.TextField(u'整个文档（HTML格式）')
    created_time = models.DateTimeField(u'创建时间', auto_now_add=True)

3.2 Reading the HTML

Here BeautifulSoup is used to do some simple filtering of the content in the HTML files. This is to strip out the text of the navigation section and improve search matching accuracy. The document content is wrapped in the markdown-body class, and the title is wrapped in the bk-title-style detail-title-right classes.

document/utils.py

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
# -*- coding: utf-8 -*-
import os
import copy
from bs4 import BeautifulSoup
from .models import Document
PROJECT_ROOT = os.path.dirname(os.path.abspath(__file__))
PROJECT_DIR, PROJECT_MODULE_NAME = os.path.split(PROJECT_ROOT)

def import_html_to_db(path=[], tag=''):
    root_path = os.path.join(PROJECT_DIR, *path)
    for root, dirs, files in os.walk(root_path, True):
        for file in files:
            if file.find('.') and file.split('.')[-1] == 'html':
                _root = copy.deepcopy(root)
                _uri = _root.replace(root_path, '')
                if _uri.startswith(os.path.sep):
                    _uri = _uri[1:]
                with open(os.path.join(PROJECT_ROOT, root, file)) as _f:
                    _doc_html = _f.read()
                    doc_html_obj = BeautifulSoup(_doc_html)
                if doc_html_obj.find_all('div', class_='markdown-body'):
                    Document.objects.create(
                        file_name=file,
                        uri=_uri,
                        tag=tag,
                        title=doc_html_obj.find_all('h3', class_='bk-title-style detail-title-right')[0].text,
                        doc_html=doc_html_obj.find_all('div', class_='markdown-body')[0],
                        doc_html_text=doc_html_obj.find_all('div', class_='markdown-body')[0].text.replace('\n', ' '),
                        doc_html_all=_doc_html
                    )

3.3 Installing and Configuring haystack

Install the dependency packages

1
2
3
pip install django-haystack
pip install whoosh
pip install jieba

Configure the index

document/search_indexes.py, the file name must be search_indexes.py

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
# -*- coding: utf-8 -*-
from haystack import indexes
from .models import Document

class DocumentIndex(indexes.SearchIndex, indexes.Indexable):
    text = indexes.CharField(document=True, use_template=True)
    doc_html = indexes.CharField(model_attr='doc_html')

    def get_model(self):
        return Document

    def index_queryset(self, using=None):
        return self.get_model().objects.all()

Configure the search engine

Copy haystack/backends/whoosh_backend.py and rename it to document/whoosh_cn_backend. Change the tokenizer to jieba; the default tokenizer has poor support for Chinese.

You only need to replace the original import of StemmingAnalyzer with jieba’s ChineseAnalyzer.

1
2
# from whoosh.analysis import StemmingAnalyzer
from jieba.analyse import ChineseAnalyzer as StemmingAnalyzer

settings.py configuration

1
2
3
4
5
6
7
8
9
INSTALLED_APPS_CUSTOM = (
    'haystack'
)
HAYSTACK_CONNECTIONS = {
    'default': {
        'ENGINE': 'document.whoosh_cn_backend.WhooshEngine',
        'PATH': os.path.join(os.path.dirname(__file__), 'whoosh_index'),
    },
}

Generate the index

1
python manage.py rebuild_index

After running the command, a folder named whoosh_index is generated in the same directory as settings.py, containing the index information.

Automatically update the index on change

Configure this in settings.py

1
HAYSTACK_SIGNAL_PROCESSOR = 'haystack.signals.RealtimeSignalProcessor'

4. Using It in Django

4.1 Using the haystack default routes

Configure urls.py

1
url(r'^search/', include('haystack.urls')),

In the template directory, add the query-related field configuration and template

template/search/search.html, the template file

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
<form method="get" action="">
  <table>
    {{ form.as_table }}
    <tr>
      <td></td>
      <td>
        <input type="submit" value="Search" />
      </td>
    </tr>
  </table>
  <h3>结果</h3>

  {% for result in page.object_list %}
  <a href="/document/{{result.uri}}/{{result.file_name}"
    >{{ result.object.title }}</a
  ><br />
  {% empty %}
  <p>没有搜索到结果.</p>
  {% endfor %}
</form>

template/search/indexes/document/document_text.txt, the query field configuration

Note the subdirectory indexes here; the folder name is a convention and must follow exactly this format. The first document is the Django app name, the second document is the Model table name, and the suffix is _text. In the text, you configure the fields to be indexed.

1
{{ object.doc_html_text }} {{ object.title }}

4.2 Custom View API

haystack also provides query functions for retrieving the matching Model objects.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
from django.shortcuts import render
from haystack.forms import ModelSearchForm
from haystack.query import SearchQuerySet

def search(request):
    page_size = int(request.GET.get('page_size', '10'))
    page_num = int(request.GET.get('page', '1'))
    form = ModelSearchForm(request.GET, searchqueryset=None, load_all=True)
    searchqueryset = form.search()
    results = [r.pk for r in searchqueryset]
    docs = Document.objects.filter(tag=request.TAG, pk__in=results)[(page_num - 1) * page_size: page_num * page_size]
    return render(request, 'search/search.html', {'docs': docs,'total': len(docs)})