a Basic setup for monitoring with ICINGA 2

This is the second installment of a series I am doing about setting up monitoring with ICINGA 2.

In Part 1 I described how to set up very basic monitoring. In this part, I will cover how to set up notifications, as, like we all know, notifications are an important part of system monitoring. I mean, what is the use of a system monitoring all the services on a network, but you not having an idea of it if something goes wrong? The nice thing about the notifications is that, once they are set up, we do not have to keep ICINGA open the whole time and manually checking that there is no problem. So let’s dive in, shall we?

Notifications are no different, although it took me quite some time to realize this. a Notification object uses a NotificationCommand Object to do the necessary notifications.

NOTE: By itself, ICINGA does not send the notifications. Instead, when the notification is called, it calls a script to do the notification.

Because of this, there are a lot of possibilities. Currently, my setup calls a custom built email script. Yes, I used PHP for it. No, I am not ashamed. It works well, it does not eat up resources (I mean it is quite small) and using the PHPMailer library, I did not have to reinvent the wheel just to do something that has been done over, and over, and over again, but it would be very easy to make it do something else. Just write the script, put it in a notification, and viola! So let’s dive in:

The first step is to define a NotificationCommand object. This contains the command, along with the arguments ICINGA 2 will execute when the notification fires.

The format for this:

object NotificationCommand "command-name" {
    import "plugin-notification-command"
    command = [ "<PreFix>", SysconfDir + "/Path/to/script" ]
    arguments = {
   	 "-a" = "$All$"
   	 "-r" = "$Arguments$"
   	 "-t" = "$To$"
   	 "-b" = "$Be$"
   	 "-p" = "$Passed$"
    }
}

 

On the first line you tell ICINGA 2 that you are defining a NotificationCommand object with the name “command-name”, as specified in the part between quotation marks.

Next we need to import the template for it. This is necessary for just about any NotificationCommand. We do this with the line import "plugin-notification-command". I have not yet come across a case where it is not necessary, but if necessary then the functionality is there.

Next, on line 3, we have the command that is to be executed. Once agains, as with the CheckCommand this is an array. The format is exactly the same as it is still a command (system command) that is to be executed. Once again, this could be any script to run when a notification is fired, whether to send a mail, or log something to the company ticket/task management system.

NOTE: In this definition I am using a constant (PluginDir). There are many constants and you can define your own. See ‘constants.conf’ for more information.

When finished, the command to be executed will look something like this:

<PreFix> /Path/ToScript/To/Execute

Once again, <PreFix> could be anything, my command uses php as prefix, so then the command looks something like this:

php /Path/ToScript/To/Execute

You could also use something else, for example sudo as prefix for a command.You can even use multiple prefix commands. Just put them in the correct order for exefcution. For example, if I wanted to add sudo to the above command definition, it would look like this:

command = [ “sudo”, "<PreFix>", SysconfDir + "/Path/to/script" ]

The resulting command execution will look something like this:

sudo php /Path/ToScript/To/Execute <argumments>

NOTE: As with the CheckCommand, when using sudo, the necessary permissions need to be in place.

You will notice that there are arguments to be passed to the script. They are picked up from the Check’s custom variables array vars. So when defining a check, keep the need for such things in mind, they might need to be passed to a NotificationCommand, or, in the same way. to a CheckCommand.

Also remember that the command defined in a *Command definition must be able to be executed directly on the system as the user ICINGA runs as (usually the user nagios - it is that for backwards compatibility).

It is also quite easy to write your own scripts for NotificationCommand or for CheckCommands. I have done both now, and it is indeed not rocket science. For the CheckCommands you just have to exit with the appropriate exit code and for the notification you don’t even need to do that.

So, as you can see, it is not hard at all to set up notifications. Go on and try it for yourself and you’ll see, after a while, as with any system, it comes naturally.