Nginx & Django on Webfaction – Part 3

Richard Cooper — Fri, 30 Oct 2009 16:52:11 +0000

Now that we have a stable and resilient environment (See Part 1a, Part 1b, and Part 2) I’m going to show you how we can hand off some of the work of file uploads to nginx and even get a nice upload progress bar into the bargain :)
Take a deep breath as this is quite a long one…. :)

First of all we will need to configure our django application, and add some models and views to handle the uploads (I’ve given some basic examples below but you will want to roll your own!).
(I’m assuming you already have a suitable project and app outline setup!)

models.py:

import datetime
from django.db import models
 
class Upload(models.Model):
    """Uploaded files."""
    file = models.FileField(upload_to='uploads',)
    timestamp = models.DateTimeField(default=datetime.datetime.now)
    notes = models.CharField(max_length=255, blank=True)
 
    class Meta:
        ordering = ['-timestamp',]
 
    def __unicode__(self):
        return u"%s" % (self.file)
 
    @property
    def size(self):
        return filesizeformat(self.file.size)

forms.py:

from django import forms
from models import Upload
 
class UploadForm(forms.models.ModelForm):
    class Meta:
        model = Upload
        exclude = ('timestamp',)

views.py:

from forms import UploadForm
from models import Upload
 
import datetime
from django.forms import save_instance
from django.shortcuts import render_to_response
from django.template import RequestContext
from django.http import HttpResponseRedirect
from django.core.urlresolvers import reverse
 
#Our initial page
def initial(request):
	data = {
		'form': UploadForm(),
	}
	return render_to_response('upload.html', data, RequestContext(request))
 
 
#Our file upload handler that the form will post to
def upload(request):
    if request.method == 'POST':
        form = UploadForm(request.POST, request.FILES)
        if form.is_valid():
            upload = Upload()
            upload.timestamp = datetime.datetime.now()
            save_instance(form, upload)
 
    return HttpResponseRedirect(reverse('initial'))

urls.py:

from django.conf.urls.defaults import *
from django.conf import settings
from django.contrib import admin
admin.autodiscover()
 
urlpatterns = patterns('',
    (r'^admin/', include(admin.site.urls)),
    (r'^media/(?P.*)$', 'django.views.static.serve', {'document_root': settings.MEDIA_ROOT}),
)
 
urlpatterns += patterns('app.views',
    url(r'^$','initial', name='initial'),
    url(r'^upload/$','upload', name="upload"),
)

Next we’ll add a couple of templates to display our upload form.
base.html:


<html>
    <head>
        <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
        <title>Upload Test</title>
        <link rel="stylesheet" href="{{MEDIA_URL|default:"/media/"}}css/screen.css" type="text/css" media="screen" charset="utf-8"/>
    </head>
    <body>
    {% block main %}{% endblock %}
    {% block js %}{% endblock %}
    </body>
</html>

upload.html:

{% extends "base.html" %}
{% block main %}
<form id="upload_form" action="{% url upload %}" method="post" enctype="multipart/form-data" accept-charset="utf-8" class="steps">
    <h3>Upload a new file</h3>
    <p><span class="step">1:</span>{{form.file}}</p>
    {% if form.file.errors %}<p>{{form.file.errors}}</p>{% endif %}
    <p><span class="step">2:</span>{{form.notes}}<span class="instruction"> Add a note...</span></p>
    {% if form.file.errors %}<p>{{form.notes.errors}}</p>{% endif %}
    <p><span class="step">3:</span><input type="submit" value="Upload" id="submit" /></p>
</form>
{% endblock %}

OK now we have a working uploader with the advantage that our nginx server will cache the file until it is fully uploaded and then pass the uploaded file to our django instance locally. What happens if we are sending large files that may take some time to upload? (e.g. mp3 or video) – we need some feedback to show that the upload is progressing.

Let’s add some code to our nginx.conf file to use the upload progress module we compiled in part 1a. This module will track the total uploaded bytes for the upload and allow us to query the status and progress.

First we need to set up an upload “zone” and reserve some server memory for tracking uploads by using the following directive in the http section of our nginx.conf:

upload_progress <zone_name> <memory>;

next we need to tell nginx to track uploads to Django under this zone by adding the following directive to our django location:

track_uploads <zone_name> <timeout>;

and finally we need to add a new location to poll for our upload status:

location ^~ /upload/progress {
      report_uploads <zone_name>;
    }

for a zone called uploadtracker our nginx.conf should now look something like this (Note the client_max_body_size is now set to allow 50Mb uploads!):

daemon off;
worker_processes 1;
 
events {
    worker_connections 1024;
}
 
http {
    include             mime.types;
    default_type        application/octet-stream;	
    upload_progress     uploadtracker 1m;	
 
    server {
        listen  55615;
        server_name *.mydomain.com;
        client_max_body_size 50m;
 
        location ^~ /upload/progress {
            report_uploads uploadtracker;
        }
 
        location /media/ {
           alias /upload/media/;
        }
 
        location ~* /favicon.ico {
           root /upload/media/img/;
           access_log off;
        }
 
        location / {
            proxy_set_header  	        X-Real-IP  $remote_addr;
            proxy_set_header            X-Forwarded-For $proxy_add_x_forwarded_for;
            fastcgi_pass                unix:/upload/django.sock;
            fastcgi_pass_header         Authorization;          
            fastcgi_hide_header         X-Accel-Redirect;
            fastcgi_hide_header         X-Sendfile;
            fastcgi_intercept_errors    off;
            include                     fastcgi_params;
            track_uploads uploadtracker 30s;
        }
    }
}

You’ll need to restart nginx with ./sbin/nginx -s reload for it to pick up the changes or if you are using supervisor you can use supervisorctl restart all to restart the django and nginx instances.

Now we need to add some java-script (and html) to poll the server and update a progress bar on the client side. I’ll use MooTools and the excellent dwProgressBar from david walsh to do the heavy lifting for this although it should be easy to re-write in the framework of your choice.

first lets change the upload.html template and add in some html and some css for the progress bar:
upload.html:

{% extends "base.html" %}
{% block main %}
<form id="upload_form" action="{% url upload %}" method="post" enctype="multipart/form-data" accept-charset="utf-8" class="steps">
    <h3>Upload a new file</h3>
    <p><span class="step">1:</span>{{form.file}}</p>
    {% if form.file.errors %}<p>{{form.file.errors}}</p>{% endif %}
    <p><span class="step">2:</span>{{form.notes}}<span class="instruction"> Add a note...</span></p>
    {% if form.file.errors %}<p>{{form.notes.errors}}</p>{% endif %}
    <p><span class="step">3:</span><input type="submit" value="Upload" id="submit" /></p>
</form>
<div id="progress_container">
    <div id="progress_filename">Please Choose a File...</div>
    <div id="put-bar-here"></div>
</div>
{% endblock %}
{% block js %}
<script type="text/javascript" src="{{MEDIA_URL|default:"/media/"}}js/mootools-1.2.4-core.js"></script>
<script type="text/javascript" src="{{MEDIA_URL|default:"/media/"}}js/mootools-1.2.4.1-more.js"></script>
<script type="text/javascript" src="{{MEDIA_URL|default:"/media/"}}js/dwProgressBar.js"></script>
{% endblock %}

screen.css:

/* Progress Bar */
#progress_container {font-size: .9em;width: 400px;height: 80px;margin:0;background-color:#fff;padding:10px;}
#box2 {background:url(../img/progress-back.png) right center no-repeat; width:400px; height:18px;float:left;}
#perc2 {background:url(../img/progress.png) right center no-repeat; height:18px;}
#text {font-family:tahoma, arial, sans-serif;font-size:11px;color:#000;float:left;padding:3px 0 0 10px;}
#progress_filename {font-size:.9em;width:100%;line-height:1.2em;padding:0 0 10px 0;color:#000;}

Now we can add our javascript – first we will create our progress bar – add this after the last

Fighting Rabbits » Random

Nginx & Django on Webfaction – Part 3