Nginx & Django on Webfaction – Part 3

Now that we have a stable and resilient environment (See Part 1a, Part 1b, and Part 2) I’m going to show you how we can hand off some of the work of file uploads to nginx and even get a nice upload progress bar into the bargain :)
Take a deep breath as this is quite a long one…. :)

First of all we will need to configure our django application, and add some models and views to handle the uploads (I’ve given some basic examples below but you will want to roll your own!).
(I’m assuming you already have a suitable project and app outline setup!)

models.py:

import datetime
from django.db import models
 
class Upload(models.Model):
    """Uploaded files."""
    file = models.FileField(upload_to='uploads',)
    timestamp = models.DateTimeField(default=datetime.datetime.now)
    notes = models.CharField(max_length=255, blank=True)
 
    class Meta:
        ordering = ['-timestamp',]
 
    def __unicode__(self):
        return u"%s" % (self.file)
 
    @property
    def size(self):
        return filesizeformat(self.file.size)

forms.py:

from django import forms
from models import Upload
 
class UploadForm(forms.models.ModelForm):
    class Meta:
        model = Upload
        exclude = ('timestamp',)

views.py:

from forms import UploadForm
from models import Upload
 
import datetime
from django.forms import save_instance
from django.shortcuts import render_to_response
from django.template import RequestContext
from django.http import HttpResponseRedirect
from django.core.urlresolvers import reverse
 
#Our initial page
def initial(request):
	data = {
		'form': UploadForm(),
	}
	return render_to_response('upload.html', data, RequestContext(request))
 
 
#Our file upload handler that the form will post to
def upload(request):
    if request.method == 'POST':
        form = UploadForm(request.POST, request.FILES)
        if form.is_valid():
            upload = Upload()
            upload.timestamp = datetime.datetime.now()
            save_instance(form, upload)
 
    return HttpResponseRedirect(reverse('initial'))

urls.py:

from django.conf.urls.defaults import *
from django.conf import settings
from django.contrib import admin
admin.autodiscover()
 
urlpatterns = patterns('',
    (r'^admin/', include(admin.site.urls)),
    (r'^media/(?P<path>.*)$', 'django.views.static.serve', {'document_root': settings.MEDIA_ROOT}),
)
 
urlpatterns += patterns('app.views',
    url(r'^$','initial', name='initial'),
    url(r'^upload/$','upload', name="upload"),
)

Next we’ll add a couple of templates to display our upload form.
base.html:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html>
    <head>
        <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
        <title>Upload Test</title>
        <link rel="stylesheet" href="{{MEDIA_URL|default:"/media/"}}css/screen.css" type="text/css" media="screen" charset="utf-8"/>
    </head>
    <body>
    {% block main %}{% endblock %}
    {% block js %}{% endblock %}
    </body>
</html>

upload.html:

{% extends "base.html" %}
{% block main %}
<form id="upload_form" action="{% url upload %}" method="post" enctype="multipart/form-data" accept-charset="utf-8" class="steps">
    <h3>Upload a new file</h3>
    <p><span class="step">1:</span>{{form.file}}</p>
    {% if form.file.errors %}<p>{{form.file.errors}}</p>{% endif %}
    <p><span class="step">2:</span>{{form.notes}}<span class="instruction"> Add a note...</span></p>
    {% if form.file.errors %}<p>{{form.notes.errors}}</p>{% endif %}
    <p><span class="step">3:</span><input type="submit" value="Upload" id="submit" /></p>
</form>
{% endblock %}

OK now we have a working uploader with the advantage that our nginx server will cache the file until it is fully uploaded and then pass the uploaded file to our django instance locally. What happens if we are sending large files that may take some time to upload? (e.g. mp3 or video) – we need some feedback to show that the upload is progressing.

Let’s add some code to our nginx.conf file to use the upload progress module we compiled in part 1a. This module will track the total uploaded bytes for the upload and allow us to query the status and progress.

First we need to set up an upload “zone” and reserve some server memory for tracking uploads by using the following directive in the http section of our nginx.conf:

upload_progress <zone_name> <memory>;

next we need to tell nginx to track uploads to Django under this zone by adding the following directive to our django location:

track_uploads <zone_name> <timeout>;

and finally we need to add a new location to poll for our upload status:

location ^~ /upload/progress {
      report_uploads <zone_name>;
    }

for a zone called uploadtracker our nginx.conf should now look something like this (Note the client_max_body_size is now set to allow 50Mb uploads!):

daemon off;
worker_processes 1;
 
events {
    worker_connections 1024;
}
 
http {
    include             mime.types;
    default_type        application/octet-stream;	
    upload_progress     uploadtracker 1m;	
 
    server {
        listen  55615;
        server_name *.mydomain.com;
        client_max_body_size 50m;
 
        location ^~ /upload/progress {
            report_uploads uploadtracker;
        }
 
        location /media/ {
           alias /upload/media/;
        }
 
        location ~* /favicon.ico {
           root /upload/media/img/;
           access_log off;
        }
 
        location / {
            proxy_set_header  	        X-Real-IP  $remote_addr;
            proxy_set_header            X-Forwarded-For $proxy_add_x_forwarded_for;
            fastcgi_pass                unix:/upload/django.sock;
            fastcgi_pass_header         Authorization;          
            fastcgi_hide_header         X-Accel-Redirect;
            fastcgi_hide_header         X-Sendfile;
            fastcgi_intercept_errors    off;
            include                     fastcgi_params;
            track_uploads uploadtracker 30s;
        }
    }
}

You’ll need to restart nginx with ./sbin/nginx -s reload for it to pick up the changes or if you are using supervisor you can use supervisorctl restart all to restart the django and nginx instances.

Now we need to add some java-script (and html) to poll the server and update a progress bar on the client side. I’ll use MooTools and the excellent dwProgressBar from david walsh to do the heavy lifting for this although it should be easy to re-write in the framework of your choice.

first lets change the upload.html template and add in some html and some css for the progress bar:
upload.html:

{% extends "base.html" %}
{% block main %}
<form id="upload_form" action="{% url upload %}" method="post" enctype="multipart/form-data" accept-charset="utf-8" class="steps">
    <h3>Upload a new file</h3>
    <p><span class="step">1:</span>{{form.file}}</p>
    {% if form.file.errors %}<p>{{form.file.errors}}</p>{% endif %}
    <p><span class="step">2:</span>{{form.notes}}<span class="instruction"> Add a note...</span></p>
    {% if form.file.errors %}<p>{{form.notes.errors}}</p>{% endif %}
    <p><span class="step">3:</span><input type="submit" value="Upload" id="submit" /></p>
</form>
<div id="progress_container">
    <div id="progress_filename">Please Choose a File...</div>
    <div id="put-bar-here"></div>
</div>
{% endblock %}
{% block js %}
<script type="text/javascript" src="{{MEDIA_URL|default:"/media/"}}js/mootools-1.2.4-core.js"></script>
<script type="text/javascript" src="{{MEDIA_URL|default:"/media/"}}js/mootools-1.2.4.1-more.js"></script>
<script type="text/javascript" src="{{MEDIA_URL|default:"/media/"}}js/dwProgressBar.js"></script>
{% endblock %}

screen.css:

/* Progress Bar */
#progress_container {font-size: .9em;width: 400px;height: 80px;margin:0;background-color:#fff;padding:10px;}
#box2 {background:url(../img/progress-back.png) right center no-repeat; width:400px; height:18px;float:left;}
#perc2 {background:url(../img/progress.png) right center no-repeat; height:18px;}
#text {font-family:tahoma, arial, sans-serif;font-size:11px;color:#000;float:left;padding:3px 0 0 10px;}
#progress_filename {font-size:.9em;width:100%;line-height:1.2em;padding:0 0 10px 0;color:#000;}

Now we can add our javascript – first we will create our progress bar – add this after the last <script> tag in the upload.html template

<script type="text/javascript" charset="utf-8">
pb2 = new dwProgressBar({
    container: $('put-bar-here'),
    startPercentage: 0,
    speed:1000,
    boxID: 'box2',
    percentageID: 'perc2',
    displayID: 'text',
    displayText: false
});
</script>

Next we need to create a unique id for our upload:

var uuid = ""
for (i = 0; i < 32; i++) {
    uuid += Math.floor(Math.random() * 16).toString(16);
}

The next bit is where it gets a little tricky we are going to use MooTools to create a request to our nginx progress location, we will use Request.Periodical to send a request every second. The nginx server will return a JSON response containing the status of the upload, along with some other data such as total bytes and uploaded bytes depending on the status.
Note: for this to work in WebKit based browsers we need to set async=false on the request due to this bug https://bugs.webkit.org/show_bug.cgi?id=23933

var req = new Request({
    method: 'get',
    headers: {'X-Progress-ID': uuid},
    url: '/upload/progress/',
    initialDelay: 500,
    delay: 1000,
    limit: 10000,
    async: false,
    onSuccess: function(reply) {
        test = JSON.decode(reply);
        switch(test.state) {
            case "uploading": 
                percent = 0.00 + parseFloat(Math.floor((test.received / test.size)*1000)/10);
                $('progress_filename').set('html','Uploading ' + filename + ' ...' + percent + '%');
                pb2.set(percent); 
                break;
            case "starting":
                $('progress_filename').set('html','Starting Upload... '); 
                break;
            case "error":
                $('progress_filename').set('html','Upload Error... ' + test.status);
                break;
            case "done":
                $('progress_filename').set('html','Upload Finished...');
                req.stopTimer();
                break;
            default:
                console.debug("Oooops!");
                break;  
        }
    },
});

Finally we hook it all together by adding an onClick handler to the submit button to change the action on our form, get the filename, and start our periodical requests.

$('submit').addEvent( 'click', function(evt){
    filename = $("id_file").get('value').split(/[\/\\]/).pop();
    $("progress_filename").set('html','Uploading ' + filename + ' ...');
    $("upload_form").set('action', "/upload/?X-Progress-ID=" + uuid);
    req.startTimer('X-Progress-ID=' + uuid);
});

The final upload.html template looks like this:

{% extends "base.html" %}
 
{% block main %}
<form id="upload_form" action="{% url upload %}" method="post" enctype="multipart/form-data" accept-charset="utf-8" class="steps">
    <h3>Upload a new file</h3>
    <p><span class="step">1:</span>{{form.file}}</p>
    {% if form.file.errors %}<p>{{form.file.errors}}</p>{% endif %}
    <p><span class="step">2:</span>{{form.notes}}<span class="instruction"> Add a note...</span></p>
    {% if form.file.errors %}<p>{{form.notes.errors}}</p>{% endif %}
    <p><span class="step">3:</span><input type="submit" value="Upload" id="submit" /></p>
</form>
<div id="progress_container">
    <div id="progress_filename">Please Choose a File...</div>
    <div id="put-bar-here"></div>
</div>
{% endblock %}
 
{% block js %}
<script type="text/javascript" src="{{MEDIA_URL|default:"/media/"}}js/mootools-1.2.4-core.js"></script>
<script type="text/javascript" src="{{MEDIA_URL|default:"/media/"}}js/mootools-1.2.4.1-more.js"></script>
<script type="text/javascript" src="{{MEDIA_URL|default:"/media/"}}js/dwProgressBar.js"></script>
<script type="text/javascript" charset="utf-8">
    var uuid = ""
    pb2 = new dwProgressBar({
        container: $('put-bar-here'),
        startPercentage: 0,
        speed:1000,
        boxID: 'box2',
        percentageID: 'perc2',
        displayID: 'text',
        displayText: false
    });
 
    for (i = 0; i < 32; i++) {
        uuid += Math.floor(Math.random() * 16).toString(16);
    }
 
    var req = new Request({
        method: 'get',
        headers: {'X-Progress-ID': uuid},
        url: '/upload/progress/',
        initialDelay: 500,
        delay: 1000,
        limit: 10000,
        onSuccess: function(reply) {
            test = JSON.decode(reply);
            switch(test.state) {
                case "uploading": 
                    percent = 0.00 + parseFloat(Math.floor((test.received / test.size)*1000)/10);
                    $('progress_filename').set('html','Uploading ' + filename + ' ...' + percent + '%');
                    pb2.set(percent); 
                    break;
                case "starting":
                    $('progress_filename').set('html','Starting Upload... '); 
                    break;
                case "error":
                    $('progress_filename').set('html','Upload Error... ' + test.status);
                    break;
                case "done":
                    $('progress_filename').set('html','Upload Finished...');
                    req.stopTimer();
                    break;
                default:
                    console.debug("Oooops!");
                    break;  
            }
        },
    })
 
    $('submit').addEvent( 'click', function(evt){
        filename = $("id_file").get('value').split(/[\/\\]/).pop();
        $("progress_filename").set('html','Uploading ' + filename + ' ...');
        $("upload_form").set('action', "{% url upload %}?X-Progress-ID=" + uuid);
        req.startTimer('X-Progress-ID=' + uuid);
	} );
</script>
{% endblock %}

You can download all the completed files from here – ZipFile.

11 Responses to “Nginx & Django on Webfaction – Part 3”

  1. Mauricio Q. Says:
    October 30th, 2009 at 22:00

    Great Article. As a minor comment, you HTML title text says “dajngo” instead of “django” that probably affects search engine indexing bla bla…

    Thanks again for this post.

  2. Richard Cooper Says:
    October 30th, 2009 at 22:11

    Thanks,
    I’ve updated the title, and I’m glad you like the post.

    Regards,
    Richard.

  3. Daniel Gollás Says:
    January 8th, 2010 at 03:40

    I spent a day implementing a custom UploadHandler in django only to find out that it won’t really work with an nginx proxy since it will not stream the upload but instead cache it until it is complete… so is nginx progress module the only way to get progress? Do you know if there is a way to get nginx to actually send me (django upload handler) chunks of data as they arrive?

    The progress module is great for progress reporting but what if I want to do something a little more customized to my data as it arrives? I think it falls a little short.

  4. Richard Cooper Says:
    January 8th, 2010 at 10:19

    Nginx was specifically designed to move work away from the back end process and thus the core Nginx sever does this caching. I believe that the author Igor is now considering changing this and adding a server directive to allow you to bypass the proxy/caching of uploads. In the meantime you could write (or get someone to write) an upload passthrough module. I suggest you ask this question on the Nginx mailing list to find out a timeline for this from Igor.

    If this is complete blocker for you then I suggest you stick with apache or consider something like lighttpd or cherokee

  5. Ian Schenck Says:
    January 11th, 2010 at 07:13

    Actually, I just had to move away from Cherokee because it both caches the upload and doesn’t have any replacement such as Nginx’s upload progress module.

    There is an open ticket and the Cherokee guys are aware of it. If it wasn’t for this issue (admittedly, combined with some laziness), I’d still be using it.

  6. Gordon Says:
    February 22nd, 2010 at 07:36

    I’m trying to setup an upload progress bar using Nginx with Passenger and Rails on WebFaction. The problem I’m having is that it reports as “starting” for the duration of the upload, then jumps the progress bar to 100% for a split second at the end of the upload. I’ve tried asking the WebFaction guys for help, also Stack Overflow and googling around, but did not get anywhere. I realize that it may be the different technologies in use that are causing my problems eg not using fastcgi, passenger, etc. but i was just wondering if you had thoughts on if there might be any salient details I might be missing.

    This is the conf file I have currently:
    http://pastie.org/836127

    Any advice would be much appreciate!

  7. Passy Says:
    June 17th, 2010 at 12:24

    Excellent post, exactly what I was searching for. Thanks for sharing!

  8. Andres Bottone Says:
    November 14th, 2010 at 21:23

    Search engine skips the duplicate content site and treat it as SPAM

  9. Dolly Serrano Says:
    December 25th, 2010 at 22:49

    Great Article. As a minor comment, you HTML title text says “dajngo” instead of “django” that probably affects search engine indexing bla bla… Thanks again for this post.

  10. Marylin Spraque Says:
    May 30th, 2011 at 00:00

    Keep working, nice post! This was the stuff I needed to get.

  11. Barbra Says:
    September 13th, 2012 at 08:44

    Hi mates, its wonderful paragraph regarding teachingand
    completely explained, keep it up all the time.

Leave a Reply