Monitor cron with Icinga

cron

I’m a big Icinga fan and have written about it a couple of times. Check out this awesome series for Debian. Now people have asked a better way to monitor cron (jobs). Can we integrate this into Icinga?

I’ve looked at a couple or solutions that are out there, but the most obvious is to monitor crons with Icinga (since I’m already using it). After some searching around it proves to be not too difficult.

Cronjob monitoring is done by inspecting exit code and job interval. Icinga reads the output and acts upon this. First we’re going to setup the check_exit_code.pl plugin as found here. Old, but still works with a very small edit:

On Debian change the shebang to #!/usr/bin/perl -w

Add the code and make it executable with these commands on your Icinga master:

# vi /usr/lib/nagios/plugins/check_exit_code.pl
# chmod +x /usr/lib/nagios/plugins/check_exit_code.pl

This plugin doesn’t have an Icinga command definition so we have to add it to /etc/icinga2/zones.d/global-templates/commands.conf like so:

object CheckCommand "exit_code" {
      import "ipv4-or-ipv6"

      command = [ PluginDir + "/check_exit_code.pl" ]

      arguments = {
         "-f" = {
                value = "$exit_code_logfile$"
                description = "The logfile to check for the exit code."
         }
         "-t" = {
                value = "$exit_code_time$"
                description = "Time in minutes which a log file can be unmodified before raising alert."
         }
     }
}

So far so good. Now a little preparation on the to be monitored system. By default cronjobs will output and email everything to a designated email address. Since people don’t want a lot of email, often cronjob output gets rerouted to /dev/null. We’re going to change this.

Login, create the log directory and log for the cronjob:

# mkdir /var/log/cronjobs
# touch /var/log/cronjobs/yourcron.log
# chown -R root:nagios /var/log/cronjobs

Then we have to edit the existing cronjob to actually use the log. For example this:

/usr/bin/php /home/user/cronjobs/yourcron.php >/dev/null 2>&1

Becomes this:

/usr/bin/php /home/user/cronjobs/yourcron.php 2>&1 > /var/log/cronlogs/yourcron.log; echo "Exit code: $?" >> /var/log/cronlogs/yourcron.log

At every cronjob run this will redirect all output to the log and append the exit code. Next run the log will be overwritten with the output and another exit code.

Back to our Icinga master! Add the below code to your client services config at /etc/icinga2/zones.d/master/yourclient.domain.com/services.conf:

apply Service for (exit_code => config in host.vars.exit_code) {
import "generic-service"
check_command = "exit_code"
vars += config
command_endpoint = host.vars.client_endpoint
assign where host.name == "yourclient.domain.com"
}

And to your hosts.conf in the same directory:

vars.exit_code["cron yourcron"] = {
exit_code_logfile = "/var/log/cronjobs/yourcron.log"
exit_code_time = "60"

This will alert you when the cron doesn’t exit with a status 0 or when the cron hasn’t run for 60 minutes (change to your needs of course).

Check and restart Icinga:

# icinga2 daemon -C
# systemctl restart icinga2

You can add as many crons as you like.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.