Tags

, , , , , , , , , ,

When an application has live traffic then the application is marked critical. Downtime incident to that service is not tolerable by users. So as an engineer we should design the application infrastructure more reliable and resilient. Among lots, one of the ways is to add service supervision.

Out of many supervisors, we have chosen runit as our best suitable service supervisor. Runit provides us following benefits:
1. Runit guarantees each service a clean process state
2. The runsv program provides a reliable logging facility for the service daemon
3. Fast system boot up and shutdown
4. Small code size

Installation

Install runit

sudo apt-get install runit

Let’s check whether runit is installed properly or not

ps -ef | grep runsvdir

Now, runit has been installed. Since the instance/machine has a different user to perform different activities. So let’s say runit will be supervising each user assigned service/server. For example, assume a deploy user which takes care of background worker named ‘Sidekiq‘.

Note: You don’t need to create nested directories if you want to be simple.

So, as a root user create a required directory based on username:

sudo mkdir -p /etc/sv/user
sudo mkdir -p /etc/sv/user/deploy

Runit supervisor looks for a run file to execute, so let’s create a run file.

sudo vi /etc/sv/user/deploy/run

and add this content:

#!/bin/sh -e
exec 2>&1
exec chpst -udeploy runsvdir /home/deploy/service

Next, change the permission of run script so that its executable.

sudo chmod +x run

Since runit search run file under folder/etc/service. So let’s symlink

sudo ln -sf /etc/sv/user/deploy/ /etc/service/deploy

So, as a deploy user let’s, create required directory as mentioned in /etc/sv/user/deploy/run script

mkdir /home/deploy/service

As deploy user task is to run background worker, let’s create directory for that

mkdir /home/deploy/service/sidekiq

and add this content to run the script inside the /home/deploy/service/sidekiq directory

#!/bin/bash
exec 2>&1 > /tmp/runit_sidekiq.log
NUM=0
APP= path/to/app_folder
source ~/.rvm/scripts/rvm
rvm use 2.3.1

SIDEKIQ_CONFIG='-C config/sidekiq.yml -c 6'
RAILS_ENV=production

cd $APP/current
RAILS_ENV=$RAILS_ENV bundle exec sidekiq \
         $SIDEKIQ_CONFIG \
         --index $NUM \
         --pidfile $APP/shared/tmp/pids/sidekiq-$NUM.pid \
         --environment $RAILS_ENV \
         --logfile $APP/shared/log/sidekiq-$NUM.log

Now, change the permission of run script so that its executable.

sudo chmod +x run

Now when you check process list, you should be able to view sidekiq running under runsv sidekiq.

ps aux | grep sidekiq
deploy    1255  0.0  0.0    168    32 ?        Ss   Jan22   0:00 runsv sidekiq-0
deploy   11188 36.5 13.6 1076644 525272 ?      Sl   05:10   1:15 sidekiq 4.2.10 applicationme [4 of 6 busy]
deploy   11775  0.0  0.0  10460   916 pts/0    S+   05:13   0:00 grep sidekiq

Woo, now sidekiq service is resilient.
Happy learning!!