Zabbix is an enterprise-class open source software for monitoring and tracking the status of different network servers and other hardware. The first version of this free software was released to the public in 2001 and since then, the Zabbix community, along with the tool, have grown by leaps and bounds.
We have two ways to integrate with Zabbix: the Plugin method, which requires you to download and install a set of scripts, or the API method, which uses our alert ingestion API and requires more manual configuration. (You can read more details about the API method here.)
In this post, we’ll detail how to get your Zabbix integration configured using the plugin method.
- Configuration elements
- Escalation routing
- Configure Ack-Back to Zabbix
- How notifications are handled
VictorOps’ Zabbix plugin is a set of scripts that queue trigger notifications and forward them to VictorOps servers for processing. Queuing of notifications is done in-band as Zabbix runs the notification script, and forwarding is done out-of-band by a long-running process.
The forwarding process can retry sending the notification, and if the send fails for any reason, for an extended period of time (1 minute by default), it will fall back to sending an email to a configured address.
The package is available as an RPM, DEB or tarball, and can be installed via yum or apt-get by adding VictorOps to your package sources.
PHP >= 5.3.0 is required on the server, which is also a requirement of the Zabbix front-end, and so it is likely to already be installed.
curl is also required on the server
The main package files are installed to /opt/victorops/zabbix_plugin
Notifications are queued in /var/victorops/zabbix-alerts.
Logging is written in /var/log/victorops.
Symbolic links are created in the Zabbix alert scripts and external scripts directories (probably /usr/lib/zabbix/alertscripts/ and /usr/lib/zabbix/externalscripts/) for some of the files in /opt/victorops/zabbix_plugin/bin.
Install with apt-get:
Add the following lines to /etc/apt/sources.list
Note that we do not yet sign our deb package, so you will have to answer “Y” to the “Install these packages without verification” prompt.
Install with yum:
Download http://software.victorops.com/VictorOps-Public.repo and place it in your /etc/yum.repos.d/ directory.
Note that we do not yet sign our rpm package, so there is no GPG key to import at this time.
If you install from the DEB or RPM packages, the installer will put the plugin files in their location in /opt/victorops/zabbix_plugin. Additionally, two directories will be created, /var/victorops/zabbix-alerts and /var/log/victorops, for notification queuing and logging, respectively.
Once installed, the package will need configuration by running thescript. The configure script will connect to the Zabbix server, so to use it you will need to provide a Zabbix admin user ID and password. The script will prompt you for these.
Additionally, you will need to provide your organization “slug” and zabbix alert key to thescript. These are found on your Zabbix integration page at https://portal.victorops.com/.
You may also provide an email address as the fallback contact in the unlikely event that sending a notification to VictorOps fails.
The configure script connects to the Zabbix server and, using the Zabbix API, adds several elements required to forward notifications to VictorOps. These are:
– A Zabbix user “VictorOps Alert”
– A Zabbix user group “VictorOps Users”
– A Zabbix media type “Notify VictorOps”
– A Zabbix action “Notify VictorOps”. The action includes the specific message content required for processing, and is invoked by Zabbix for any trigger in problem status, including recovery notifications.
– A Zabbix item “VictorOps forwarder” to the host “Zabbix Server” for the VictorOps forwarder process. This item is an “external check” type. When Zabbix runs a check of this item, it will start the forwarder process if necessary.
Any incidents created by notifications sent to VictorOps will be processed on the default route. To change the routing, you modify the “media” for the “VictorOps Alert” user and put your VictorOps routing key in the “Send to” field:
To create multiple or complex routing schemes, you may need to create new Zabbix actions, users and or media types for the routing keys involved.
To enable the ack-back feature for Zabbix, do the following:
Copy /opt/victorops/zabbix_plugin/conf/local.zabbix.conf.example to /opt/victorops/zabbix_plugin/conf/local.zabbix.conf.
Open /opt/victorops/zabbix_plugin/conf/local.zabbix.conf in an editor and set a user and password allowed to log in to the Zabbix console with “acknowledge” permissions
You may wish to change the owner of the file to the zabbix user, and set permissions to 600.
On the zabbix console, navigate to the “Items” list for your zabbix server and enable the “VictorOps Command Poll” item.
The action to notify VictorOps saves the notification details to a file on disk. Within a couple seconds, the forwarder process will send the details to VictorOps, where the normal alert processing will be performed.
If forwarding the details is unsuccessful, the timestamp of the details is checked, and if the details are old (1 minute by default), the details are instead sent to the backup email address you provided at configuration time.