icinga2 #2 – some lessons learned

I’ll insert a post here that explains a few lessons learned (usually quite some time down the road) so you can avoid things I did wrong and figured out a lot later.

make sure everything you set up is working properly before plowing on

Particularly, check the History section if you icinga2web to see if you got unexpected amounts of state changes (check flapping as nagios used to call it) and notifications (if you have set up some already). If you don’t and you have some issue there that is only a monitoring issue not some host or service with an actual problem, all your historical data will be moot. So doing overviews and reports “something for the boss” later on over that data will not produce anything useful.

decive how you name your hosts and services and stick to it

For one renaming hosts also messes with statistics you might wand to do later on, also you want to keep things tidy and orderly. Even worse if you add your data to some other tool for long-term statistics, then even the display name might get used.

You might want to ponder a bit if you use IPs, Hostnames or fully-qualified DNS-Names for your “Host address”-field, “Hostname” (intenal identifier of the host) and/or Display name.

Some general “state the obvious” about that

  • display name shoud have some description added if your hostnames are not descriptive enough to tell you at a glance what that host is/what’s running on that host
  • using DNS names for “Host address” obviously will fail if your DNS is broken. so you need to use some IPs for very basic infrastructure like your DNS.

My take was to use FQDNS for most things, sometimes add info to that for a Hostname. And have basic network devices, virtualization servers, storage etc. with IP as “Host address” so I can see what is broken if my DNS fails.

I ended up with a bit of a mix though, and at least in parts cleaning it up would make my historical data useless so currently i’m not sure if I ever will.

Feel free to comment on that strategy 😀


Leave a Reply

Your email address will not be published. Required fields are marked *