Fighting Rabbits

Nginx & Django on Webfaction – Part 3

Posted by Richard Cooper | Filed under Random

Now that we have a stable and resilient environment (See Part 1a, Part 1b, and Part 2) I’m going to show you how we can hand off some of the work of file uploads to nginx and even get a nice upload progress bar into the bargain :)
Take a deep breath as this is quite a long one…. :)

First of all we will need to configure our django application, and add some models and views to handle the uploads (I’ve given some basic examples below but you will want to roll your own!).
(I’m assuming you already have a suitable project and app outline setup!)

models.py:

import datetime
from django.db import models
 
class Upload(models.Model):
    """Uploaded files."""
    file = models.FileField(upload_to='uploads',)
    timestamp = models.DateTimeField(default=datetime.datetime.now)
    notes = models.CharField(max_length=255, blank=True)
 
    class Meta:
        ordering = ['-timestamp',]
 
    def __unicode__(self):
        return u"%s" % (self.file)
 
    @property
    def size(self):
        return filesizeformat(self.file.size)

forms.py:

from django import forms
from models import Upload
 
class UploadForm(forms.models.ModelForm):
    class Meta:
        model = Upload
        exclude = ('timestamp',)

views.py:

from forms import UploadForm
from models import Upload
 
import datetime
from django.forms import save_instance
from django.shortcuts import render_to_response
from django.template import RequestContext
from django.http import HttpResponseRedirect
from django.core.urlresolvers import reverse
 
#Our initial page
def initial(request):
	data = {
		'form': UploadForm(),
	}
	return render_to_response('upload.html', data, RequestContext(request))
 
 
#Our file upload handler that the form will post to
def upload(request):
    if request.method == 'POST':
        form = UploadForm(request.POST, request.FILES)
        if form.is_valid():
            upload = Upload()
            upload.timestamp = datetime.datetime.now()
            save_instance(form, upload)
 
    return HttpResponseRedirect(reverse('initial'))

urls.py:

from django.conf.urls.defaults import *
from django.conf import settings
from django.contrib import admin
admin.autodiscover()
 
urlpatterns = patterns('',
    (r'^admin/', include(admin.site.urls)),
    (r'^media/(?P<path>.*)$', 'django.views.static.serve', {'document_root': settings.MEDIA_ROOT}),
)
 
urlpatterns += patterns('app.views',
    url(r'^$','initial', name='initial'),
    url(r'^upload/$','upload', name="upload"),
)

Next we’ll add a couple of templates to display our upload form.
base.html:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html>
    <head>
        <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
        <title>Upload Test</title>
        <link rel="stylesheet" href="{{MEDIA_URL|default:"/media/"}}css/screen.css" type="text/css" media="screen" charset="utf-8"/>
    </head>
    <body>
    {% block main %}{% endblock %}
    {% block js %}{% endblock %}
    </body>
</html>

upload.html:

{% extends "base.html" %}
{% block main %}
<form id="upload_form" action="{% url upload %}" method="post" enctype="multipart/form-data" accept-charset="utf-8" class="steps">
    <h3>Upload a new file</h3>
    <p><span class="step">1:</span>{{form.file}}</p>
    {% if form.file.errors %}<p>{{form.file.errors}}</p>{% endif %}
    <p><span class="step">2:</span>{{form.notes}}<span class="instruction"> Add a note...</span></p>
    {% if form.file.errors %}<p>{{form.notes.errors}}</p>{% endif %}
    <p><span class="step">3:</span><input type="submit" value="Upload" id="submit" /></p>
</form>
{% endblock %}

OK now we have a working uploader with the advantage that our nginx server will cache the file until it is fully uploaded and then pass the uploaded file to our django instance locally. What happens if we are sending large files that may take some time to upload? (e.g. mp3 or video) – we need some feedback to show that the upload is progressing.

Let’s add some code to our nginx.conf file to use the upload progress module we compiled in part 1a. This module will track the total uploaded bytes for the upload and allow us to query the status and progress.

First we need to set up an upload “zone” and reserve some server memory for tracking uploads by using the following directive in the http section of our nginx.conf:

upload_progress <zone_name> <memory>;

next we need to tell nginx to track uploads to Django under this zone by adding the following directive to our django location:

track_uploads <zone_name> <timeout>;

and finally we need to add a new location to poll for our upload status:

location ^~ /upload/progress {
      report_uploads <zone_name>;
    }

for a zone called uploadtracker our nginx.conf should now look something like this (Note the client_max_body_size is now set to allow 50Mb uploads!):

daemon off;
worker_processes 1;
 
events {
    worker_connections 1024;
}
 
http {
    include             mime.types;
    default_type        application/octet-stream;	
    upload_progress     uploadtracker 1m;	
 
    server {
        listen  55615;
        server_name *.mydomain.com;
        client_max_body_size 50m;
 
        location ^~ /upload/progress {
            report_uploads uploadtracker;
        }
 
        location /media/ {
           alias /upload/media/;
        }
 
        location ~* /favicon.ico {
           root /upload/media/img/;
           access_log off;
        }
 
        location / {
            proxy_set_header  	        X-Real-IP  $remote_addr;
            proxy_set_header            X-Forwarded-For $proxy_add_x_forwarded_for;
            fastcgi_pass                unix:/upload/django.sock;
            fastcgi_pass_header         Authorization;          
            fastcgi_hide_header         X-Accel-Redirect;
            fastcgi_hide_header         X-Sendfile;
            fastcgi_intercept_errors    off;
            include                     fastcgi_params;
            track_uploads uploadtracker 30s;
        }
    }
}

You’ll need to restart nginx with ./sbin/nginx -s reload for it to pick up the changes or if you are using supervisor you can use supervisorctl restart all to restart the django and nginx instances.

Now we need to add some java-script (and html) to poll the server and update a progress bar on the client side. I’ll use MooTools and the excellent dwProgressBar from david walsh to do the heavy lifting for this although it should be easy to re-write in the framework of your choice.

first lets change the upload.html template and add in some html and some css for the progress bar:
upload.html:

{% extends "base.html" %}
{% block main %}
<form id="upload_form" action="{% url upload %}" method="post" enctype="multipart/form-data" accept-charset="utf-8" class="steps">
    <h3>Upload a new file</h3>
    <p><span class="step">1:</span>{{form.file}}</p>
    {% if form.file.errors %}<p>{{form.file.errors}}</p>{% endif %}
    <p><span class="step">2:</span>{{form.notes}}<span class="instruction"> Add a note...</span></p>
    {% if form.file.errors %}<p>{{form.notes.errors}}</p>{% endif %}
    <p><span class="step">3:</span><input type="submit" value="Upload" id="submit" /></p>
</form>
<div id="progress_container">
    <div id="progress_filename">Please Choose a File...</div>
    <div id="put-bar-here"></div>
</div>
{% endblock %}
{% block js %}
<script type="text/javascript" src="{{MEDIA_URL|default:"/media/"}}js/mootools-1.2.4-core.js"></script>
<script type="text/javascript" src="{{MEDIA_URL|default:"/media/"}}js/mootools-1.2.4.1-more.js"></script>
<script type="text/javascript" src="{{MEDIA_URL|default:"/media/"}}js/dwProgressBar.js"></script>
{% endblock %}

screen.css:

/* Progress Bar */
#progress_container {font-size: .9em;width: 400px;height: 80px;margin:0;background-color:#fff;padding:10px;}
#box2 {background:url(../img/progress-back.png) right center no-repeat; width:400px; height:18px;float:left;}
#perc2 {background:url(../img/progress.png) right center no-repeat; height:18px;}
#text {font-family:tahoma, arial, sans-serif;font-size:11px;color:#000;float:left;padding:3px 0 0 10px;}
#progress_filename {font-size:.9em;width:100%;line-height:1.2em;padding:0 0 10px 0;color:#000;}

Now we can add our javascript – first we will create our progress bar – add this after the last <script> tag in the upload.html template

<script type="text/javascript" charset="utf-8">
pb2 = new dwProgressBar({
    container: $('put-bar-here'),
    startPercentage: 0,
    speed:1000,
    boxID: 'box2',
    percentageID: 'perc2',
    displayID: 'text',
    displayText: false
});
</script>

Next we need to create a unique id for our upload:

var uuid = ""
for (i = 0; i < 32; i++) {
    uuid += Math.floor(Math.random() * 16).toString(16);
}

The next bit is where it gets a little tricky we are going to use MooTools to create a request to our nginx progress location, we will use Request.Periodical to send a request every second. The nginx server will return a JSON response containing the status of the upload, along with some other data such as total bytes and uploaded bytes depending on the status.
Note: for this to work in WebKit based browsers we need to set async=false on the request due to this bug https://bugs.webkit.org/show_bug.cgi?id=23933

var req = new Request({
    method: 'get',
    headers: {'X-Progress-ID': uuid},
    url: '/upload/progress/',
    initialDelay: 500,
    delay: 1000,
    limit: 10000,
    async: false,
    onSuccess: function(reply) {
        test = JSON.decode(reply);
        switch(test.state) {
            case "uploading": 
                percent = 0.00 + parseFloat(Math.floor((test.received / test.size)*1000)/10);
                $('progress_filename').set('html','Uploading ' + filename + ' ...' + percent + '%');
                pb2.set(percent); 
                break;
            case "starting":
                $('progress_filename').set('html','Starting Upload... '); 
                break;
            case "error":
                $('progress_filename').set('html','Upload Error... ' + test.status);
                break;
            case "done":
                $('progress_filename').set('html','Upload Finished...');
                req.stopTimer();
                break;
            default:
                console.debug("Oooops!");
                break;  
        }
    },
});

Finally we hook it all together by adding an onClick handler to the submit button to change the action on our form, get the filename, and start our periodical requests.

$('submit').addEvent( 'click', function(evt){
    filename = $("id_file").get('value').split(/[\/\\]/).pop();
    $("progress_filename").set('html','Uploading ' + filename + ' ...');
    $("upload_form").set('action', "/upload/?X-Progress-ID=" + uuid);
    req.startTimer('X-Progress-ID=' + uuid);
});

The final upload.html template looks like this:

{% extends "base.html" %}
 
{% block main %}
<form id="upload_form" action="{% url upload %}" method="post" enctype="multipart/form-data" accept-charset="utf-8" class="steps">
    <h3>Upload a new file</h3>
    <p><span class="step">1:</span>{{form.file}}</p>
    {% if form.file.errors %}<p>{{form.file.errors}}</p>{% endif %}
    <p><span class="step">2:</span>{{form.notes}}<span class="instruction"> Add a note...</span></p>
    {% if form.file.errors %}<p>{{form.notes.errors}}</p>{% endif %}
    <p><span class="step">3:</span><input type="submit" value="Upload" id="submit" /></p>
</form>
<div id="progress_container">
    <div id="progress_filename">Please Choose a File...</div>
    <div id="put-bar-here"></div>
</div>
{% endblock %}
 
{% block js %}
<script type="text/javascript" src="{{MEDIA_URL|default:"/media/"}}js/mootools-1.2.4-core.js"></script>
<script type="text/javascript" src="{{MEDIA_URL|default:"/media/"}}js/mootools-1.2.4.1-more.js"></script>
<script type="text/javascript" src="{{MEDIA_URL|default:"/media/"}}js/dwProgressBar.js"></script>
<script type="text/javascript" charset="utf-8">
    var uuid = ""
    pb2 = new dwProgressBar({
        container: $('put-bar-here'),
        startPercentage: 0,
        speed:1000,
        boxID: 'box2',
        percentageID: 'perc2',
        displayID: 'text',
        displayText: false
    });
 
    for (i = 0; i < 32; i++) {
        uuid += Math.floor(Math.random() * 16).toString(16);
    }
 
    var req = new Request({
        method: 'get',
        headers: {'X-Progress-ID': uuid},
        url: '/upload/progress/',
        initialDelay: 500,
        delay: 1000,
        limit: 10000,
        onSuccess: function(reply) {
            test = JSON.decode(reply);
            switch(test.state) {
                case "uploading": 
                    percent = 0.00 + parseFloat(Math.floor((test.received / test.size)*1000)/10);
                    $('progress_filename').set('html','Uploading ' + filename + ' ...' + percent + '%');
                    pb2.set(percent); 
                    break;
                case "starting":
                    $('progress_filename').set('html','Starting Upload... '); 
                    break;
                case "error":
                    $('progress_filename').set('html','Upload Error... ' + test.status);
                    break;
                case "done":
                    $('progress_filename').set('html','Upload Finished...');
                    req.stopTimer();
                    break;
                default:
                    console.debug("Oooops!");
                    break;  
            }
        },
    })
 
    $('submit').addEvent( 'click', function(evt){
        filename = $("id_file").get('value').split(/[\/\\]/).pop();
        $("progress_filename").set('html','Uploading ' + filename + ' ...');
        $("upload_form").set('action', "{% url upload %}?X-Progress-ID=" + uuid);
        req.startTimer('X-Progress-ID=' + uuid);
	} );
</script>
{% endblock %}

You can download all the completed files from here – ZipFile.

Django on Snow Leopard

Posted by Richard Cooper | Filed under Coding, Setup

Having just been given an new MacBook Pro as an early birthday present, it didn’t take me long to try and install Django on it. Rather than use the MacPorts solution I decided to use the native python 2.6 and build the rest, apart from MySQL (use the Mac OS X 10.5 (x86_64) package from here)

Before proceeding make sure that you have installed XCode (It’s on the Snow Leopard install disc) as we need it to install libjpeg and PIL (all of my Django projects use this!)

Download libjpeg then:

tar -xzf jpegsrc-1.v7.tar
cd jpegsrc-1.v7
ln -s `which glibtool` ./libtool
set env MACOSX_DEPLOYMENT_TARGET 10.6
./configure --enable-shared && make && sudo make install

Now we can download the source and build PIL

tar -xzf Imaging-1.1.6.tar.gz
cd Imaging-1.1.6
sudo python setup.py install

Last thing to do before we install Django is to install the MySqlDb connector – you will need to do add the following to your .bash_profile

PATH="${PATH}:/usr/local/mysql/bin"

Then:

sudo easy_install mysql-python

Now we can easy_install Django:

sudo easy_install django

and we are all set to go :)

Thanks to these guys for the help in figuring this out:
http://dryan.com/
http://www.brambraakman.com/blog/
http://colbypalmer.com/
http://www.agapow.net/

Tags: django, easy_install, os x, snow leopard, Software., tools

Mounting Supervisor on a WebFaction subdomain

Posted by Richard Cooper | Filed under Setup, webfaction

In a previous post I mentioned that you could mount your supervisor process on a sub domain so that you could view and manage your Django and Nginx processes from a web browser. In this post I’m going to show you how to do this.
Read the rest of this entry…

Tags: django, hosting, nginx, tools, webfaction

« Older Entries

Fighting Rabbits

Nginx & Django on Webfaction – Part 3

Django on Snow Leopard

Mounting Supervisor on a WebFaction subdomain

Now Tweeting

Tags

Blogroll

Social Media