I’ll insert a post here that explains a few lessons learned (usually quite some time down the road) so you can avoid things I did wrong and figured out a lot later.

make sure everything you set up is working properly before plowing on

Particularly, check the History section if you icinga2web to see if you got unexpected amounts of state changes (check flapping as nagios used to call it) and notifications (if you have set up some already). If you don’t and you have some issue there that is only a monitoring issue not some host or service with an actual problem, all your historical data will be moot. So doing overviews and reports “something for the boss” later on over that data will not produce anything useful.

decive how you name your hosts and services and stick to it

For one renaming hosts also messes with statistics you might wand to do later on, also you want to keep things tidy and orderly. Even worse if you add your data to some other tool for long-term statistics, then even the display name might get used.

You might want to ponder a bit if you use IPs, Hostnames or fully-qualified DNS-Names for your “Host address”-field, “Hostname” (intenal identifier of the host) and/or Display name.

Some general “state the obvious” about that

• display name shoud have some description added if your hostnames are not descriptive enough to tell you at a glance what that host is/what’s running on that host
• using DNS names for “Host address” obviously will fail if your DNS is broken. so you need to use some IPs for very basic infrastructure like your DNS.

My take was to use FQDNS for most things, sometimes add info to that for a Hostname. And have basic network devices, virtualization servers, storage etc. with IP as “Host address” so I can see what is broken if my DNS fails.

I ended up with a bit of a mix though, and at least in parts cleaning it up would make my historical data useless so currently i’m not sure if I ever will.

Feel free to comment on that strategy 😀

The first thing i had to figure out was how to go about installing the whole thing.

Our envorionment is basically completely virtualized, so creating a new monitoring-VM from my favourite debian template only took a few minutes.

Luckily I figured out really soon there is such a thing as director for icinga (see https://github.com/Icinga/icingaweb2-module-director) and that

a) it seems a good idea to use it

b) it’s probably best do do everything with it, so no manual configuration at all

After looking a bit I found an install guide that explained (most importantly for me) how to clean up the default icinga2 config so you can actually start with director with a clean canvas. It’s for cent OS but that hardly matters, you only need to add the apt source (deb http://packages.icinga.com/debian icinga-stretch main) and install icinga2 and icinga2web packages. Then pick up the guide with the cleanup of the stock config and adding the director.

I’ve been using icinga2 with icinga director for over a year now to rebuild our monitoring. Old one was nagios3, but i didn’t bother migrating anything old since we were moving over to new server hardware and in the process re-creating basically all VMs in use, partly for license reasons partly just to clean up and upgrade installations.

So i set up icinga2 and kept adding the new things to it as we build them, slowly moving away from the old nagios3.

I want to start a small series of blog entries about how i went with my setup since icinga guides especially when it comes to using director are rather sparse, so other ppl might find it useful.

Stay tuned.