Rent my time and knowledge so I can help with your project!


Ricardo Pascal

Usually a Python programmer, sometimes an Angular programmer, a physics dropou planning a triumphal come back, currently working as bioinformatician and team leader in a biotech company in south Brazil.

Enjoy expose/consume data from rest API's and teach/evangelize python newcomers.

Serving large and small files (or anything in between) with Django

Since the beginning of times (1981) a common use of web applications has been serving files, django is capable of that, but is usually not considered the best alternative. But why? The honest answer is: because there are servers that will do this by a fraction of the resources that django takes, but this doesn't mean that you can't do it at all (as long as you know the limitations that you will face).

To be fair, small applications can handle files through django views without much complication, but as your app usage grows, problems will arise. Let's explore three ways of serving a file in a django application.

Load file in memory and serve with django

This first approach is probably the most common, but not much economic, since you load the entire file into memory.

The drawbacks

  • can't handle files larger than ram
  • can't serve more files that in size sum more than the available ram.
  • django server will have threads busy until the transference is over.

Load file chunks and serve with django

The drawbacks

  • much more CPU cycles and disk I/O will be used
  • django server will have threads busy until the transference is concluded.

Both examples work in small scale, but imagine that you have a lot of concurrent users and many big files to handle, you simply can't afford that much memory and cores.

Let apache (or another webserver) deal with files

If you let a webserver handle the files for you, with their optimised code designed to serve data of granular size, your hardware will use much less memory and CPU cycles. A common approach (with apache) is to use the x-send-file module.

It handles the file transference directly and can be triggered by a django view as following:

You will need to install and enable the x-send-module, in ubuntu systems (tested on 14.04) just type:

apt-get install apache2-mod-xsendfile

a2enmod xsendfile

you also have to add the XSendFile and XSendFilePath directives in your virtual host configuration:

<VirtualHost *:80>
    DocumentRoot /var/www/html

    ErrorLog ${APACHE_LOG_DIR}/error.log
    CustomLog ${APACHE_LOG_DIR}/access.log combined
    WSGIScriptAlias / /srv/

    XSendFile On
    XSendFilePath /tmp

If everything goes right, each file requested to your View that match a file in the XSendFilePath will be handled by apache, and you will still be able to programmatically apply any restriction to define who can access the given file.

by Ricardo Pascal on Nov. 1, 2014

comments powered by Disqus