A RSS Feed For Your Crontabs

The crontab system is an embodiment of several facets of the “Unix Way”. One of the famous rules of this philosophy is the “Rule of Silence”, meaning that unless something alarming or unexpected has happened, it should remain silent. By design, the scripts you are running in your crontab should produce no output, they are simply meant to run at the assigned time and do a job.

Sadly, our software is not perfect and our scripts are fallible. Servers go offline, software updates change syntax, our old assumptions may not always be valid. So we need to find some sort of happy medium between running invisibly and constantly being in our face.

Case Study - A Backup System That Could Use Some Improvement

We are starting with a very basic backup routine. Data on our file server is stored in /storage and we want to clone it nightly to a removable USB drive mounted at /backups. This is accomplished by the following script.

We run this script nightly from crontab as follows.

So this works pretty well, our backup job executes and since the v switch is being used we exploit a side effect of a crontab - any output is emailed to the cron job owner. We end up with a nice output of the backup job, if everything runs well we see a verbose list of files synced, if something goes wrong we see that output as well.

This couldn’t be simpler.

But there is a problem. After a few days of smooth sailing it will be very tempting to skim the daily reports and just assume that everything is going well. Then as more time passes we will simple tune them out and delete these reports. The challenge we are facing is that we still want some feed back from our backup system, but not too much that we ignore it. Essentially, we want to know:

  • Is the system even running?

  • Did something bad happen?

Luckily, with just a little instrumentation added to our backup script, we can use this to create an RSS feed that we can follow in our favorite feed reader to see the health of our system at a glance.

Modifying Our Backup Script

For our added instrumentation we will be exploiting another side effect of the “Unix Way”. Specifically, the return value of the command. A “well behaved” program will return an exit code of 0 meaning “everything ran as expected”. Any other code will mean “something went wrong”.

We, aren’t going to dive into the different return codes and what they mean went wrong, we just want to know if it went as expected or not.

Let’s review the changes we have made:

  • #!/usr/bin/env bash - Switch from sh to bash since sh might be linked or another shell that is not compatible with the variables and if statements we are using

  • RSYNC_RESULT=$? - This tells the script to capture the return code of the last command executed. As we’ve discussed, 0 is good, anything else means “something went wrong”

  • if [ $RSYNC_RESULT ] - If the result is 0 execute the following command, otherwise execute the later

  • echo “Rsync to backup drive success, date” » /storage/BACKUPS/jobstatus.txt - The results was good so we will append this new status to jobstatus.txt

  • echo “Rsync to backup drive failed, date” » /storage/BACKUPS/jobstatus.txt - The results was no so good, append this new status to jobstatus.txt

Using more complicated decision structures and Boolean logic tests we could create very complicated, multi-step tests, but for now let’s stick with something simple.

Now that we have our backup script added updates the jobstatus.txt the final step is to take this primitive database and use to to create an RSS feed.

Generating the RSS Feed

We aren’t going to get too deep into the explanation of this script (If you are new to Python, please check out the excellent guide, “Learn Python The Hard Way”).

Our “database” is set in the DATALOC, the script opens this file and iterates through each line. It reads in the line and assigns the first half (before the comma) to the description and assigns the date to the second half. If you want to, feel free to create a third field (separated by another comma) for the title of the RSS item.

After the script has parsed each RSS item it generates a feed our of all of these items and writes it to FEEDLOC. This location should be in accessible by a web-server running on this server. From there it’s just a matter of your feed reader picking up this update.

Below is a sample of our new crontab, first running the backup script, then running our RSS generator:

Our Daily Status Report

And with a few minutes of work we can now monitor our daily scripts and jobs from a simple glance at our RSS feed. This is definitely not suited for a real-time status, but it does well for daily or weekly tasks.

There are many improvements that could be made, better checking of status codes to report the exact nature of the error if the status is not 0, more bullet-proof scripts to begin with, but this is a good start to looking at and thinking about your repeating jobs in a different way. Keeping the status brief and to the point it can easily be glanced over and is less likely to be forgotten.

Erlang OTP Template

Still working on a few lengthier blog posts, but I wanted to share my template for Erlang OTP applications.  I’ve gotten tired of looking this up every time I start a new Erlang app and having to remember where I found this and that so I decided to just make a git repo for this and piece together the different tips and tricks I come across.

Presently, this is a very bare-bones OTP layout. I am still looking for a good “make” scheme to really automate building and cleanup, and other goodies like generating edocs.

For now I give you my Erlang OTP Template.

Please feel free to share your suggestions in the comments, or at https://github.com/zpeters/ErlangOtpTemplate

Coming soon...

Hey everyone, just checking in with a quick yes-i’m-still-alive-sorry-i-haven’t-been-blogging-much-I-promise-something-really-soon post.

Here’s what I’ve been working on, in no particular order:

  • “Git Setup” - Where I describe the basic setup of a Linux based git server using git and ssh and a Windows “client” using Tortoisegit

  • Using Git as a “Poor Man’s” Time Machine – Part Three - The third installment of my far to stretched out implementation of a Time Machine like system using git and some batch files

  • “The Power of Text” - a discussion on the power of text based formats for configuration and communication

  • “Netsh traces” - a post on the usefulness of the “traces” facility in Windows 7/Server 2008 for network troubleshooting

  • “Erlang Weather Server” - where i describe a simple weather server written in Erlang that pulls data from noaa

Thanks for the patience, I hope to finishing a few of these up soon.

In the meantime, if you have any suggestions of topics you’d like to discuss or questions you have please leave feedback in the comments, or feel free to email me at: [email protected]

Happy Hacking,