This page looks best with JavaScript enabled

Large File Transfer in Django

 ·  ☕ 3 min read

1. Returning the File Directly

If the static file is at media/test.zip in the project root, you first have to read the file into memory and then transfer it. The code looks like this:

settings.py configuration

1
2
PROJECT_ROOT = os.path.dirname(os.path.abspath(__file__))
MEDIA_ROOT = os.path.join(PROJECT_ROOT, 'media/')

yourapp/views.py

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
from django.conf import settings
from django.http import HttpResponse
from django.core.files.storage import FileSystemStorage

def download_file_direct_from_file(request):
  file_system = FileSystemStorage(settings.MEDIA_ROOT)
  file_name = 'test.zip'
  with file_system.open(file_name) as f:
      response = HttpResponse(f)
      response['Content-Type'] = 'application/%s' % file_name.split('.')[-1]
      response['Content-Disposition'] = 'attachment; filename="%s"'% file_name
  return response

If the file comes from an API call fetched by a third party, you only need to assign the variable f in with file_system.open(file_name) as f to the response body you retrieved, for example: f = requests.get(url).content. The code examples below will not be explained one by one; they use downloading a local file as the example.

2. Returning a Streaming File

Django’s HttpResponse object supports an iterator as its initialization argument. Replacing the file object with an iterator or generator can optimize how Django handles large files. At the same time, Django provides the StreamingHttpResponse object to replace the HttpResponse object, in order to support sending a file to the browser as a stream.

You can implement an iterator yourself

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
from django.conf import settings
from django.http import StreamingHttpResponse

def download_file_direct(request):
    file_name = 'test.zip'
    full_path_file_name = ''.join([settings.MEDIA_ROOT, file_name])

    def file_read(full_path_name, chunk_size=512):
        with open(full_path_name, 'rb') as f:
            while True:
                chunks = f.read(chunk_size)
                if chunks:
                    yield chunks
                else:
                    break

    response = StreamingHttpResponse(file_read(full_path_file_name))
    response['Content-Type'] = 'application/octet-stream'
    response['Content-Disposition'] = 'attachment;filename=%s' % (file_name)
    return response

You can also use the FileWrapper class that Django provides to turn the file object into an iterator.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
from django.conf import settings
from django.core.files.storage import FileSystemStorage
from django.core.servers.basehttp import FileWrapper
from django.http import StreamingHttpResponse

def download_file_chunk_from_file(request):
    file_system = FileSystemStorage(settings.MEDIA_ROOT)
    file_name = 'test.zip'
    response = StreamingHttpResponse(FileWrapper(file_system.open(file_name, 'rb'), 512))
    response['Content_Type'] = 'application/octet-stream'
    response['Content-Disposition'] = 'attachment; filename=%s' % file_name
    return response

3. Using the sendfile Mechanism

When Django handles a file, it has to read the file content into memory and then send the in-memory content to the browser. Django’s ability to forward files is far below that of Nginx. A better approach is to use Django for the permission check and then let Nginx forward the file.

sendfile is a high-performance network IO approach. By using the operating system kernel’s sendfile call, it can push the file content straight to the NIC buffer, avoiding the overhead of reading and writing the file in the application layer. In Nginx, the X-Accel-Redirect feature can be used to implement file handling through the sendfile mechanism.

Nginx configuration

1
2
3
4
location /media {
    internal;
    alias /var/www/my_django_project_root/media;
}

views.py

1
2
3
4
5
6
7
8
9
from django.conf import settings
from django.http import HttpResponse

def download_file_from_nginx(request):
    file_name = 'test.zip'
    response['Content_Type']='application/octet-stream'
    response['Content-Disposition'] = 'attachment;filename=%s' % (file_name)
    response['X-Accel-Redirect'] = '/media/%s' % file_name
    return response

4. References


WeChat Official Account
WRITTEN BY
WeChat Official Account