Nginx & Django on Webfaction – Part 2

Posted by Richard Cooper | Filed under Random

Hopefully after finishing Part 1a and Part 1b you have a working Nginx + Django install on Webfaction. In this installment I’m going to show you how to keep your site running through app failures and server reboots.

One of the problems with having our own web stack  running is what happens if the Webfaction support team ever need to reboot the server or if our Django App crashes? We could solve this by adding a cron job to restart our stack every few minutes but because Nginx doesn’t spawn our FCGI Django process it gets quite complex, and we have no easy way to monitor the status.

The solution as suggested here  is to use supervisor to control our server processes and then make sure that we have a cron job to check and restart supervisor.

Please don’t try this on a production site if you don’t know what you are doing unless you can afford some down time whilst you get it working!

First we need to create a new app listening on port to reserve ourselves a port for supervisor to run on:
Create A new App
And make a note of the port number for later:
Make a note of the Port Number
First we need to create a start script for Django so that supervisor can monitor it, use your favourite editor to create a file called runserver.py in your django project directory (/home/<myuser>/webapps/<myapp>/<myproject>/runserver.py) with the following content:

#!/usr/local/bin/python2.5
import sys
import os
sys.path.append('/home/<myuser>/webapps/<myapp>/<myproject>/')
sys.path.append('/home/<myuser>/webapps/<myapp>/lib/python2.5/django/bin')
sys.path.append('/home/<myuser>/webapps/<myapp>/lib/python2.5/django/')
 
if __name__ == '__main__':
    from flup.server.fcgi_fork import WSGIServer
    from django.core.handlers.wsgi import WSGIHandler
    WSGIServer(WSGIHandler(),maxSpare=1, maxChildren=1, debug=False).run()

Remember to change the <myuser> etc to the correct values, save the file and make it executable using:

chmod +x runserver.py

We also need to change our django settings.py file and change the ROOT_URLCONF to:

ROOT_URLCONF = 'urls'

Because supervisor can only interact with non daemonized processes we need to make a minor change to our nginx.conf file and add the following line right at the begining:

daemon off;

Next we need to install supervisor:

easy_install-2.5 supervisor

Then create a file called supervisord.conf in your home directory with the following content (remember to replace <portnumber> with the number from earlier in this post):

; Webfaction supervisor config file.
 
[inet_http_server]          				; inet (TCP) server setings
port=127.0.0.1:<portnumber>	    			; (ip_address:port specifier, *:port for all iface)
username=<username>           				; (default is no username (open server))
password=<password>   	   			           ; (default is no password (open server))
 
[supervisord]
logfile=/home/<myuser>/logs/user/supervisor/supervisord.log 		; (main log file;default $CWD/supervisord.log)
logfile_maxbytes=50MB       						; (max main logfile bytes b4 rotation;default 50MB)
logfile_backups=10          						; (num of main logfile rotation backups;default 10)
loglevel=debug               						; (log level;default info; others: debug,warn,trace)
pidfile=/home/<myuser>/supervisord.pid                          		; (supervisord pidfile;default supervisord.pid)
nodaemon=false              						; (start in foreground if true;default false)
minfds=1024                 						; (min. avail startup file descriptors;default 1024)
minprocs=200                						; (min. avail process descriptors;default 200)
 
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
 
[supervisorctl]
serverurl=http://127.0.0.1:<portnumber>
username=<username>
password=<password>
 
[program:nginx]
command=/home/<myuser>/webapps/<myapp>/sbin/nginx -c /home/<myuser>/webapps/<myapp>/conf/nginx.conf
autostart=true
autorestart=true
redirect_stderr=true
exitcodes=0
 
[fcgi-program:django]
socket=unix:///home/<myuser>/webapps/<myapp>/<myproject>/django.sock
process_name=%(program_name)s_%(process_num)02d
numprocs=1
command = /home/<myuser>/webapps/<myapp>/<myproject>/runserver.py  	
environment=DJANGO_SETTINGS_MODULE=settings
autostart=true
autorestart=true
redirect_stderr=true

Replace the <username> and <password> in the file above with ones of your choosing (not your webfaction account ones!) Assuming that all the <xxx> bits have been replaced properly we could now start supervisor – however we would still suffer from the problem of supervisor not being started on a reboot. Let’s fix that – create a file called start_supervisor.sh in your home directory with the following content:

#! /bin/bash 
 
NAME=supervisord
SUPERVISORD=/home/<myuser>/bin/supervisord
SUPERVISORCTL=/home/<myuser>/bin/supervisorctl
PIDFILE=/home/<myuser>/supervisord.pid
OPTS="-c /home/<myuser>/supervisord.conf"
PS=$NAME
TRUE=1
FALSE=0
 
test -x $SUPERVISORD || exit 0
 
export PATH="${PATH:+$PATH:}/usr/local/bin:/usr/sbin:/sbin:/home/<myuser>/bin:"
 
isRunning(){
    pidof_daemon
    PID=$?
 
    if [ $PID -gt 0 ]; then
	return 1
    else
        return 0
    fi
}
 
pidof_daemon() {
    PIDS=`pidof -x $PS` || true
 
    [ -e $PIDFILE ] && PIDS2=`cat $PIDFILE`
 
    for i in $PIDS; do
        if [ "$i" = "$PIDS2" ]; then
            return 1
        fi
    done
    return 0
}
 
start () {
    echo "Starting Supervisor daemon manager..."
    isRunning
    isAlive=$?
 
    if [ "${isAlive}" -eq $TRUE ]; then
        echo "Supervisor is already running."
    else
        $SUPERVISORD $OPTS || echo "Failed...!"
        echo "OK"
    fi
}
 
stop () {
    echo "Stopping Supervisor daemon manager..."
    $SUPERVISORCTL shutdown ||  echo "Failed...!"
    echo "OK"
}
 
case "$1" in
  start)
    start
	;;
 
  stop)
    stop
	;;
 
  restart|reload|force-reload)
    stop
    start
    ;;
 
esac
 
exit 0

we’ll use this script to start and stop supervisor so we need to make it executable. Now we need to add it to a cron job to make sure that it runs every few minutes (the script checks that supervisor is running so won’t start multiple instances of it) this will allow us to be back up and running a few minutes after the server comes back up.

EDITOR=nano crontab -e
*/10 * * * * cd ~/; ./start_supervisor.sh

This will run our script every 10 mins. Now if you stop your Django and Nginx processes, cd to your home directory, and run

./start_supervisor.sh

You should now be running on supervisor and your site should be up. If you’d like to be able to see that status of your processes then mount the app we created at the start on a subdomain such as status.mydomain.com and then browse to that url (you’ll need to enter the user and password that you put into the supervisor.conf file)

Supervisor Website

Supervisor Website


Enjoy!

Permalink | Comments (12) | Sep 21, 2009

12 Responses to “Nginx & Django on Webfaction – Part 2”

  1. Randy Pena Says:
    September 21st, 2009 at 19:28

    Where did you get your blog layout from? I’d like to get one like it for my blog.

  2. Richard Cooper Says:
    September 21st, 2009 at 20:32

    It’s called VectorLover you can get it from here http://www.themelab.com/2008/08/04/vectorlover-free-wordpress-theme/

    :)

  3. tim Says:
    September 27th, 2009 at 21:11

    Which folder do you install supervisor in? Also can you explain how to mount the process of supervisor on a subdomain?

  4. Richard Cooper Says:
    September 27th, 2009 at 21:39

    @tim

    If you use “easy-install-2.5 supervisor” on webfaction it will install it in the correct directory (if you are worried then cd ~/ to take you to your home folder first.)

    I’ll post something in the morning (UK time) about mounting the supervisor on a sub domain. But basically you have to create a new website record in the control panel to link the supervisor App (created at the beginning of this post) to the sub-domain.

  5. Hernan Says:
    November 13th, 2009 at 05:05

    shouldn’t the crontab be like:

    */10 * * * * cd ~/; ./start_supervisor.sh start

    and

    ./start_supervisor.sh start

    Other than that great post!

  6. Hernan Says:
    November 13th, 2009 at 06:21

    Also found out that

    command = /home/mystartp/webapps/fcgi/accuputt/runserver.py

    was not working for me, I had to do

    command = python2.5 /home/mystartp/webapps/fcgi/accuputt/runserver.py

  7. Andy Baker Says:
    February 6th, 2010 at 16:29

    What’s the cause of this step?:

    ROOT_URLCONF = ‘urls’

    The reason I’m asking is I’m getting a bunch of other import errors in an app that worked fine without nginx and that line seems to imply some funkiness with my Python path.

  8. Richard Cooper Says:
    February 7th, 2010 at 15:04

    probably something to do with the sys.path.append lines in runserver.py
    I’d look there first if you have path issues. :)

    Let me know if you get it fixed and I’ll update the post with any changes….

  9. Andy Baker Says:
    February 8th, 2010 at 17:49

    Yep. That was it. I’ve still got the bad habits picked up from the Django tutorial and all my projects are called ‘myproject’ ;-)

    Assuming a project called ‘myproject’ then as well as:
    sys.path.append(‘/home//webapps//myproject/’)

    if you add:
    sys.path.append(‘/home//webapps//’)

    then that you can have imports like either:
    import myproject.something
    or just:
    import something

    and the urls line can stay as:
    ROOT_URLCONF = ‘myproject.urls’

    It’s would be better to fix my code so that I avoided explicitly referencing ‘myproject’ but quite a few tutorials do this and so does manage.py’s

    It’s better not to do either of these things but I’ve fallen into bad habits and I’m sure others have too!

  10. Richard Cooper Says:
    February 8th, 2010 at 20:27

    Hi Andy,
    I was just looking at this today as I’m working on a script to enable this to be setup via the Webfaction control panel using their API. I’ll update the post with this in a day or so and give you the credit :)

    Thanks for your interest.
    Richard.

  11. Andy Baker Says:
    February 15th, 2010 at 18:19

    Your easy_install supervisor line won’t necessarily put supervisor into the correct directory as it specify the same lib and bin locations as you did in part 1 of the tutorial.

  12. JONES Says:
    April 26th, 2010 at 11:01

    Nice approach. Thank you for your insight.

Leave a Reply




Switch to our mobile site