Fighting Rabbits

Install java on an Ubuntu Oneiric instance in EC2

Richard Cooper — Tue, 25 Oct 2011 20:24:28 +0000

Just ran into a small problem installing openjdk in an Ubuntu Oneiric instance on EC2. It took a while to diagnose so I’m posting this here so I remember next time I have to do it.

The problem is with the x11-common package needing a temp directory which is missing – so running the following command will give us a /tmp directory:

sudo mkdir /tmp

You should then be able to install x11-common and therefore openjdk without problems.

Long time, no post….

Richard Cooper — Thu, 02 Dec 2010 15:30:42 +0000

Hmmm, been missing a while from this blog… will start to post again soon as I have some interesting updates to share with you.

Please Support Them

Richard Cooper — Wed, 23 Jun 2010 12:28:04 +0000

A couple of friends of mine are cycling across America – 3500 miles in 30 Days! – to raise money for 3 Cancer charities – Please help them…

I feel that I have I winged a bit today. I don’t want to sound like a moaner, I am just trying to get a cross that this really is a challenge in so many ways. In fact, the cycling is probably the easy bit. This has been far tougher than I had imagined.

I do realise that what I am going through is nothing compared to what someone fighting cancer goes through. And maybe that is just another reason why today has been emotional. Realising, why I am here and why I am doing this.

CycleAcrossAmerica2010.com

Lightweight Django Thumbnails

Richard Cooper — Thu, 18 Feb 2010 20:07:53 +0000

I needed a good thumbnail solution for a recent Django project but didn’t want the overhead of using sorl – I just needed a simple template tag that would let me generate the correct size of thumbnail from the main picture.

I found a solution on djangosnippets.org but it needed a little tweeking to suit my needs.

The filter is applied directly to an image field using a template tag such as

{{ object.image|make_thumbnail:'100x200' }}

and supposing the image filename is “image.jpg”, it checks if there is a file called “image_100x200.jpg”, if the file isn’t there, it resizes and saves the original image, finally it returns the proper url to the resized image.

The updated filter is given below (you will need PIL installed):

import os
import Image
 
from django.template import Library, Node
from django.template import Variable, resolve_variable
from django.core.urlresolvers import reverse
from django.contrib.contenttypes.models import ContentType
 
register = Library()
 
def make_thumbnail(file, size='200x200'):
    """
    Example:
    
    
    
 
    The filter is applied to a image field (not the image url get from
    get_image_url method of the model), supposing the image filename is
    "image.jpg", it checks if there is a file called "image_200x200.jpg" or
    "image_200x300.jpg" on the second case, if the file isn't there, it resizes
    the original image, finally it returns the proper url to the resized image.
    """
 
    if file:
        # defining the size
        x, y = [int(x) for x in size.split('x')]
        # defining the filename and the miniature filename
        basename, format = file.path.rsplit('.', 1)
        baseurl, _format = file.url.rsplit('.', 1)
        miniature_filename = basename + '_' + size + '.' +  format
        miniature_url = baseurl + '_' + size + '.' +  format
 
        # if the image wasn't already resized, resize it
        if not os.path.exists(miniature_filename):
            image = Image.open(file.path)
            pw = image.size[0]
            ph = image.size[1]
            nw = x
            nh = y
            pr = float(pw) / float(ph)
            nr = float(nw) / float(nh)
 
            if image.mode not in ('L', 'RGB'):
                image = image.convert('RGB')
 
            if pw == nw and ph == nh:
                pass
            else:
                if pr > nr:
                    # image aspect is wider than destination ratio
                    tw = int(round(nh * pr))
                    image = image.resize((tw, nh), Image.ANTIALIAS)
                    l = int(round(( tw - nw ) / 2.0))
                    image = image.crop((l, 0, l + nw, nh))
                elif pr < nr:
                    # image aspect is taller than destination ratio
                    th = int(round(nw / pr))
                    image = image.resize((nw, th), Image.ANTIALIAS)
                    t = int(round(( th - nh ) / 2.0))
                    image = image.crop((0, t, nw, t + nh))
                else:
                    # image aspect matches the destination ratio
                    image = image.resize(size, Image.ANTIALIAS)
            image.save(miniature_filename, image.format)
        return miniature_url
 
    return ''
 
register.filter(make_thumbnail)

Save it as something like thumbs_filter.py in a templatetag folder in one of your django apps and then use it like this:

{% load thumbs_filter %}
{{ object.image|make_thumbnail:'100x200' }}

Nginx & Django on Webfaction – Part 3

Richard Cooper — Fri, 30 Oct 2009 16:52:11 +0000

Now that we have a stable and resilient environment (See Part 1a, Part 1b, and Part 2) I’m going to show you how we can hand off some of the work of file uploads to nginx and even get a nice upload progress bar into the bargain :)
Take a deep breath as this is quite a long one…. :)

First of all we will need to configure our django application, and add some models and views to handle the uploads (I’ve given some basic examples below but you will want to roll your own!).
(I’m assuming you already have a suitable project and app outline setup!)

models.py:

import datetime
from django.db import models
 
class Upload(models.Model):
    """Uploaded files."""
    file = models.FileField(upload_to='uploads',)
    timestamp = models.DateTimeField(default=datetime.datetime.now)
    notes = models.CharField(max_length=255, blank=True)
 
    class Meta:
        ordering = ['-timestamp',]
 
    def __unicode__(self):
        return u"%s" % (self.file)
 
    @property
    def size(self):
        return filesizeformat(self.file.size)

forms.py:

from django import forms
from models import Upload
 
class UploadForm(forms.models.ModelForm):
    class Meta:
        model = Upload
        exclude = ('timestamp',)

views.py:

from forms import UploadForm
from models import Upload
 
import datetime
from django.forms import save_instance
from django.shortcuts import render_to_response
from django.template import RequestContext
from django.http import HttpResponseRedirect
from django.core.urlresolvers import reverse
 
#Our initial page
def initial(request):
	data = {
		'form': UploadForm(),
	}
	return render_to_response('upload.html', data, RequestContext(request))
 
 
#Our file upload handler that the form will post to
def upload(request):
    if request.method == 'POST':
        form = UploadForm(request.POST, request.FILES)
        if form.is_valid():
            upload = Upload()
            upload.timestamp = datetime.datetime.now()
            save_instance(form, upload)
 
    return HttpResponseRedirect(reverse('initial'))

urls.py:

from django.conf.urls.defaults import *
from django.conf import settings
from django.contrib import admin
admin.autodiscover()
 
urlpatterns = patterns('',
    (r'^admin/', include(admin.site.urls)),
    (r'^media/(?P.*)$', 'django.views.static.serve', {'document_root': settings.MEDIA_ROOT}),
)
 
urlpatterns += patterns('app.views',
    url(r'^$','initial', name='initial'),
    url(r'^upload/$','upload', name="upload"),
)

Next we’ll add a couple of templates to display our upload form.
base.html:


<html>
    <head>
        <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
        <title>Upload Test</title>
        <link rel="stylesheet" href="{{MEDIA_URL|default:"/media/"}}css/screen.css" type="text/css" media="screen" charset="utf-8"/>
    </head>
    <body>
    {% block main %}{% endblock %}
    {% block js %}{% endblock %}
    </body>
</html>

upload.html:

{% extends "base.html" %}
{% block main %}
<form id="upload_form" action="{% url upload %}" method="post" enctype="multipart/form-data" accept-charset="utf-8" class="steps">
    <h3>Upload a new file</h3>
    <p><span class="step">1:</span>{{form.file}}</p>
    {% if form.file.errors %}<p>{{form.file.errors}}</p>{% endif %}
    <p><span class="step">2:</span>{{form.notes}}<span class="instruction"> Add a note...</span></p>
    {% if form.file.errors %}<p>{{form.notes.errors}}</p>{% endif %}
    <p><span class="step">3:</span><input type="submit" value="Upload" id="submit" /></p>
</form>
{% endblock %}

OK now we have a working uploader with the advantage that our nginx server will cache the file until it is fully uploaded and then pass the uploaded file to our django instance locally. What happens if we are sending large files that may take some time to upload? (e.g. mp3 or video) – we need some feedback to show that the upload is progressing.

Let’s add some code to our nginx.conf file to use the upload progress module we compiled in part 1a. This module will track the total uploaded bytes for the upload and allow us to query the status and progress.

First we need to set up an upload “zone” and reserve some server memory for tracking uploads by using the following directive in the http section of our nginx.conf:

upload_progress <zone_name> <memory>;

next we need to tell nginx to track uploads to Django under this zone by adding the following directive to our django location:

track_uploads <zone_name> <timeout>;

and finally we need to add a new location to poll for our upload status:

location ^~ /upload/progress {
      report_uploads <zone_name>;
    }

for a zone called uploadtracker our nginx.conf should now look something like this (Note the client_max_body_size is now set to allow 50Mb uploads!):

daemon off;
worker_processes 1;
 
events {
    worker_connections 1024;
}
 
http {
    include             mime.types;
    default_type        application/octet-stream;	
    upload_progress     uploadtracker 1m;	
 
    server {
        listen  55615;
        server_name *.mydomain.com;
        client_max_body_size 50m;
 
        location ^~ /upload/progress {
            report_uploads uploadtracker;
        }
 
        location /media/ {
           alias /upload/media/;
        }
 
        location ~* /favicon.ico {
           root /upload/media/img/;
           access_log off;
        }
 
        location / {
            proxy_set_header  	        X-Real-IP  $remote_addr;
            proxy_set_header            X-Forwarded-For $proxy_add_x_forwarded_for;
            fastcgi_pass                unix:/upload/django.sock;
            fastcgi_pass_header         Authorization;          
            fastcgi_hide_header         X-Accel-Redirect;
            fastcgi_hide_header         X-Sendfile;
            fastcgi_intercept_errors    off;
            include                     fastcgi_params;
            track_uploads uploadtracker 30s;
        }
    }
}

You’ll need to restart nginx with ./sbin/nginx -s reload for it to pick up the changes or if you are using supervisor you can use supervisorctl restart all to restart the django and nginx instances.

Now we need to add some java-script (and html) to poll the server and update a progress bar on the client side. I’ll use MooTools and the excellent dwProgressBar from david walsh to do the heavy lifting for this although it should be easy to re-write in the framework of your choice.

first lets change the upload.html template and add in some html and some css for the progress bar:
upload.html:

{% extends "base.html" %}
{% block main %}
<form id="upload_form" action="{% url upload %}" method="post" enctype="multipart/form-data" accept-charset="utf-8" class="steps">
    <h3>Upload a new file</h3>
    <p><span class="step">1:</span>{{form.file}}</p>
    {% if form.file.errors %}<p>{{form.file.errors}}</p>{% endif %}
    <p><span class="step">2:</span>{{form.notes}}<span class="instruction"> Add a note...</span></p>
    {% if form.file.errors %}<p>{{form.notes.errors}}</p>{% endif %}
    <p><span class="step">3:</span><input type="submit" value="Upload" id="submit" /></p>
</form>
<div id="progress_container">
    <div id="progress_filename">Please Choose a File...</div>
    <div id="put-bar-here"></div>
</div>
{% endblock %}
{% block js %}
<script type="text/javascript" src="{{MEDIA_URL|default:"/media/"}}js/mootools-1.2.4-core.js"></script>
<script type="text/javascript" src="{{MEDIA_URL|default:"/media/"}}js/mootools-1.2.4.1-more.js"></script>
<script type="text/javascript" src="{{MEDIA_URL|default:"/media/"}}js/dwProgressBar.js"></script>
{% endblock %}

screen.css:

/* Progress Bar */
#progress_container {font-size: .9em;width: 400px;height: 80px;margin:0;background-color:#fff;padding:10px;}
#box2 {background:url(../img/progress-back.png) right center no-repeat; width:400px; height:18px;float:left;}
#perc2 {background:url(../img/progress.png) right center no-repeat; height:18px;}
#text {font-family:tahoma, arial, sans-serif;font-size:11px;color:#000;float:left;padding:3px 0 0 10px;}
#progress_filename {font-size:.9em;width:100%;line-height:1.2em;padding:0 0 10px 0;color:#000;}

Now we can add our javascript – first we will create our progress bar – add this after the last