1. Problem Description

Background: a SaaS application developed with Django, offering document services externally. Its search feature is implemented through Django Haystack.

Problem: the search feature is sometimes available and sometimes not. After repeated testing, the available and unavailable states were found to alternate, each occurring with a probability of roughly 50%.

A few more details on how the search feature is implemented:

Before Django Haystack can provide search, the following command must be run:

1

python manage.py update_index

Generating index files:

1
2
3

_MAIN_1.toc
MAIN_l4dclez7n5lsj047.seg
MAIN_WRITELOCK

Only when valid index files exist can search be served. So the index-update command was executed directly in Python via subprocess, with the following code:

1
2
3

cmd = "echo 'y'| python manage.py update_index"
subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
out, err = p.communicate()

2. Locating the Problem

When you find the service’s available and unavailable states alternating, the first thing that comes to mind is that Nginx’s default load-balancing policy is round robin, alternately assigning requests to different backends.

The above is a diagram of the SaaS deployment. The developer deploys the SaaS — the app server in the figure — via Docker. To achieve app server high availability, the PaaS platform automatically instantiates the SaaS twice at deployment time, ensuring that two Docker instances serve simultaneously. In front of the app server, Nginx load-balances and distributes user requests, using exactly the round-robin strategy.

This should be a very accurate diagnosis; what comes next is how to ensure both instances have rebuilt their indexes. But this one problem consumed a great deal of time.

2.1 Django Once Code

On StackOverFlow, two ways were found to make Django execute something only once at startup:

• Using the top-level urls.py

The urls.py module is imported and executed only once.

1
2
3
4

from django.confs.urls.defaults import *
from my_app import one_time_startup
urlpatterns = ...
one_time_startup()

• Using a Django App’s apps.py

The apps.py file can configure some custom initialization for a Django App.

1
2
3
4
5
6
7

# myapp/apps.py
from django.apps import AppConfig
class MyAppConfig(AppConfig):
    name = 'myapp'
    verbose_name = 'My Application'
    def ready(self):
        pass # one_time_startup code here

1
2

#  myapp/__init__.py
default_app_config = 'myapp.apps.MyAppConfig'

Inside the one_time_startup() function, implement the index rebuild.

By the deployment logic, when the PaaS platform instantiates the app server, each instance rebuilds the index, so search availability should be 100%, and the search function was verified to work locally.

But after actually going live, the search service was not available. And in the deployment logs, no index-rebuild log appeared either. It seems the PaaS platform forbids executing certain built-in commands when starting an instance.

2.2 Rebuilding the Index via a Django URL Visit

If the index files cannot be rebuilt when instantiating the app server, then what about calling an endpoint directly?

1
2
3
4
5

# urls.py
from django.conf.urls import include, patterns
urlpatterns = patterns('document.views',
                       (r'^build/$', 'build'),
                       )

1
2
3
4
5
6

# views.py
import subprocess
def build(request):
    cmd = "echo 'y'| python manage.py update_index"
    subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
    out, err = p.communicate()

So a URL endpoint was written that executes the index-rebuild command when visited. After the SaaS was released, it was visited twice in a row, updating the index on each of the two instances respectively.

But this approach is hard to operate and cannot guarantee that no one else visits between the two requests.

Since containers are used for deployment, the local data from the previous deployment is destroyed on every deployment. So each deployment starts fresh, and it suffices to put the index-rebuild logic in the view for the home page, adding a check before rebuilding the index.

1
2
3
4

def article(request):
    if not os.path.exists('索引文件'):
        pass # 重建索引
    # 返回文档内容

By this reasoning, at this point, if an instance has no index file, it rebuilds; if an instance has an index file, it skips the rebuild. Both instances should have index files, and search availability should be 100%. Yet it was not!

To save CPU and memory resources, the two servers host hundreds of app server instances. Each index rebuild takes nearly 20 seconds.

During those 20 seconds, if another request comes in, because the index is being created and no index file can be detected, it triggers yet another index rebuild. After testing many times, search availability was still not 100%, and a rather strange phenomenon appeared: sometimes available, sometimes not, and sometimes a 500. The initial suspicion was that repeatedly triggering index rebuilds consumed a large amount of service resources and produced 500s when the service was unavailable, while the index having been cleared before the rebuild made search unavailable on the instance that was rebuilding.

2.3 Using a Third-Party Service

For a service that is sometimes available and sometimes not, and whose release is especially tedious and error-prone, this is clearly unacceptable. So an NFS service was used, which can be seen as a third-party mounted-directory service. After instantiating the app server, mount the locally exclusive RES directory onto the RES directory inside the app server instance container. Best of all, all instances share the RES directory.

The configuration is very simple: configure Haystack’s index directory inside the RES directory:

1
2
3
4
5
6

HAYSTACK_CONNECTIONS = {
    'default': {
        'ENGINE': 'document.whoosh_cn_backend.WhooshEngine',
        'PATH': os.path.join(PROJECT_DIR, 'RES','whoosh_index'),
    },
}

Then, visit via URL to execute the index-rebuild command:

1
2
3
4
5

# urls.py
from django.conf.urls import include, patterns
urlpatterns = patterns('document.views',
                       (r'^build/$', 'build'),
                       )

1
2
3
4
5
6

# views.py
import subprocess
def rebuild(request):
    cmd = "echo 'y'| python manage.py update_index"
    subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
    out, err = p.communicate()

2.4 Summary

For anomalies where things work locally but the service cannot be provided online, the problem is usually in the deployment.

Understanding the deployment process and logic is very important for SaaS development, especially for people who rely on a PaaS for application development, testing, and deployment.

The example above is in fact a conflict between high availability and high consistency. High availability means multiple service instances are needed, while high consistency requires all service instances to have consistent data. The way to resolve this kind of conflict is to separate out the data service; the example above does so by mounting an NFS service.

Use third-party services as much as possible, and do not hold state in SaaS.

3. Stateless

Being stateless is one of the principles of high-concurrency design. If a service instance does not store persistent data locally, and multiple instances respond with exactly the same result to the same request, then the service is called stateless.

A stateless service is very easy to scale horizontally. By adding more instances, you can significantly improve the service’s concurrency performance.

As shown above, for a stateful service, each service maintains a state internally.

A stateless service simply separates the state out of the service and shares it. As shown above, node A and node B provide the same application service while sharing state. This decouples the lifecycle of the application service from the lifecycle of the state. If the state service — that is, the data service — is highly available, then all the application services are highly available as well.

4. References

• https://www.geekhub.cn/a/894.html