TurboGears Deployment :: Supervisor
Now we don't want to have to manually start our app all the time by running ./start-avwsystems.py all the time. We want to setup something to make sure it starts up when the computer does, and make sure that if our app dies, that it gets restarted automatically.
To do this we're going to use Supervisor2. It will start itself up as a daemon and launch our application for us. If our application dies Supervisor will restart it for us. It will also give us a nice control panel for stopping our application or even restarting it. We might need to restart it when we update the application to the changes are applied.
First I installed supervisor using the instructions here: http://paddyland.serveblog.net/article/13
Once it is installed we need to setup our application to launch like we want. In my case I want my application to run as my web user (I'm used to apache/php in this manner). We also want to make sure that our application runs in the path we expect it to. This will make sure things like file paths don't bite us. For instance...just for kicks...let's say you defined you application log file to be server.log. Since it's relatively defined we need to make sure our app is run from `/home/rharding/src/tg-avw` where the `www-data` user has permissions. Otherwise it might try to create the file where the `www-data` user does not have rights and supervisor will be happy to choke on starting our application. Yep, this happened to me. [edit: so I've been corrected that launching apps from a non root directory is not a good thing to do.]
We'll setup our launch command for Supervisor2:
/home/rharding/src/tg-avw/app_run.sh
oops, that file doesn't exist yet so let's create it.
Bash Launch Scipt:
#!/bin/bash cd "${0%/*}" exec ./start-avwsystems.py ${1:-prod.cfg}
With that we can work on setting up supervisor to launch our application. Below is a sample `/etc/supervisord.conf` file.
/etc/supervisord.conf
[supervisord]
;http_port=/tmp/supervisor.sock ; (default is to run a UNIX domain socket server)
http_port=127.0.0.1:9001 ; (alternately, ip_address:port specifies AF_INET)
;sockchmod=0700 ; AF_UNIX socketmode (AF_INET ignore, default 0700)
;sockchown=nobody.nogroup ; AF_UNIX socket uid.gid owner (AF_INET ignores)
;umask=022 ; (process file creation umask;default 022)
logfile=/var/log/supervisord.log ; (main log file;default $CWD/supervisord.log)
logfile_maxbytes=50MB ; (max main logfile bytes b4 rotation;default 50MB)
logfile_backups=10 ; (num of main logfile rotation backups;default 10)
loglevel=trace ; (logging level;default info; others: debug,warn)
pidfile=/var/run/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
nodaemon=false ; (start in foreground if true;default false)
minfds=1024 ; (min. avail startup file descriptors;default 1024)
minprocs=200 ; (min. avail process descriptors;default 200)
;nocleanup=true ; (don't clean up tempfiles at start;default false)
http_username=username ; (default is no username (open system))
http_password=password ; (default is no password (open system))
;childlogdir=/tmp ; ('AUTO' child log dir, default $TEMP)
user=root ; (default is current user, required if root)
directory=/ ; (default is not to cd during start)
;environment=KEY=value ; (key value pairs to add to environment)
[supervisorctl]
;serverurl=unix:///tmp/supervisor.sock ; use a unix:// URL for a unix socket
serverurl=http://127.0.0.1:9001 ; use an http:// url to specify an inet socket
username=username ; should be same as http_username if set
password=password ; should be same as http_password if set
;prompt=mysupervisor ; cmd line prompt (default "supervisor")
; The below sample program section shows all possible program subsection values,
; create one or more 'real' program: sections to be able to control them under
; supervisor.
#[program:myproject]
#; the program (relative uses PATH, can take args)
#command=/home/foo/share/tg_apps/MyProject/run
#priority=999 ; the relative start priority (default 999)
#autostart=true ; start at supervisord start (default: true)
#autorestart=true ; retstart at unexpected quit (default: true)
#startsecs=10 ; number of secs prog must stay running (def. 10)
#startretries=3 ; max # of serial start failures (default 3)
#;exitcodes=0,2 ; 'expected' exit codes for process (default 0,2)
#stopsignal=TERM ; signal used to kill process (default TERM)
#;stopwaitsecs=10 ; max num secs to wait before SIGKILL (default 10)
#user=foo ; setuid to this UNIX account to run the program
#log_stdout=true ; if true, log program stdout (default true)
#log_stderr=true ; if true, log program stderr (def false)
#; child log path, use NONE for none; default AUTO
#logfile=/var/log/myproject-supervisor.log
#logfile_maxbytes=10MB ; max # logfile bytes b4 rotation (default 50MB)
#logfile_backups=10 ; # of logfile backups (default 10)
[program:tg-avw]
command=/home/rharding/src/tg-avw/app_run.sh
autorestart=true
autostart=true
user=www-data
logfile=/var/log/tg-avw-supervisor.log
logfile_maxbytes=50MB
logfile_backups=10
With this file all setup and ready to go (notice we have the command coded to call our bash script we created earlier) we can try to launch supervisor. To start out try to launch supervisor in non-daemon mode so we can get feedback.
sudo supervisord -n
If it starts up and runs we should see something like:
2007-02-16 17:55:01,055 CRIT Set uid to user 0
2007-02-16 17:55:01,073 INFO 127.0.0.1:Medusa (V1.1.1.1) started at Fri Feb 16 17:55:01 2007
Hostname: localhost
Port:9001
2007-02-16 17:55:01,146 INFO supervisord started with pid 6969
2007-02-16 17:55:01,153 INFO spawned: 'tg-avw' with pid 6970
2007-02-16 17:55:02,153 INFO success: tg-avw entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Now if we launch our browser and test things out our application should be running/working. Just launch our url `http://tg-avw/`
With that working we can visit our web control panel. If you used my config from above you can launch it by going to the url
http://localhost:9001/
login with http auth settings from the `supervisord.conf`. Here's a screenshot of the tool.
There should be our little application running. From this web interface we can restart/stop our application. Go ahead, stop it and then try to visit the application url. You'll get an apache error like so:
Bad Gateway
The proxy server received an invalid response from an upstream server.
Ok, so it works in non-daemon mode. The next step is to kill our running `supervisord` and start it without the -n flag to get it to run in daemon mode.
Go ahead, start supervisor back up (make sure you killed the one we were running above ^^) and test things out. If all went well the application is humming along nicely. From here you can look at adding supervisord to your systems startup scripts. If you followed the instructions from the blog entry above supervisor will be setup to run from init.d on start http://mitechie.com/uploads/supervisor-web.pngup of the system.
Now I've found that while starting will work from init.d commands, stopping doesn't work.
Other things that need work: The application should be launched from / instead of from the application directory. In order to do this I still need to work on the code in Turbogears to detect what directory the file is in and use that to find the path to things like my server.log example.