Roxy-WI Checker 5 | Distributed monitoring and alerts

About Checker

Checker monitors HAProxy, NGINX and Apache services as well as HAProxy backends and UDP listeners. It publishes status changes to Roxy-WI, which stores incident history and sends notifications through Socket, Telegram, Slack, email, PagerDuty or Mattermost.

If you installed services manually, read how to configure them here.

Checker 5 is one standalone, long-running distributed worker. Roxy-WI publishes desired assignments through RabbitMQ, and Checker stores them in its own database, performs checks continuously and publishes heartbeats and status changes back to Roxy-WI.

There is no checker_master.py and no per-target subprocess.
Assignments survive restarts and are protected by leases so another replica can recover them.
Roxy-WI stores incident history and sends notifications; Checker does not access the Roxy-WI database.
Worker version, heartbeat and assignment count are displayed on Overview and Admin area => Tools.

Multiple Checker replicas share one command queue and one Checker-owned MySQL/MariaDB database. A single replica may use SQLite.

Checker Features

HAProxy service, backend and Maxconn threshold checks;
NGINX and Apache service checks;
UDP listener checks;
durable status-change events and incident history in Roxy-WI;
notifications through Socket, Telegram, Slack, email, PagerDuty and Mattermost channels configured in Roxy-WI.

Remote Keepalived checks require a target-side probe agent or credential provider. Checker deliberately reports them as unsupported instead of sending SSH passwords or private keys through RabbitMQ.

Installation

# EL/RHEL-compatible systems
sudo dnf install roxy-wi-checker

# Debian/Ubuntu
sudo apt update
sudo apt install roxy-wi-checker

# Run after checker.env is configured on a standalone host
sudo systemctl enable --now roxy-wi-checker

The package does not require Roxy-WI on the same host. Its default auto mode reads the connection from a local Roxy-WI installation when available. On a worker-only host, edit /etc/roxy-wi/checker.env, set CHECKER_USE_ROXYWI_RABBITMQ_SETTINGS=0, provide all CHECKER_RABBITMQ_* connection values and then start the unit. Keep the package's default CHECKER_DATABASE_URL for one replica or configure a shared Checker database for HA.

Create a root-readable /etc/roxy-wi/checker.env for a container:

CHECKER_DATABASE_URL=sqlite:////var/lib/roxy-checker/checker-state.db
CHECKER_RABBITMQ_HOST=rabbitmq.internal
CHECKER_RABBITMQ_PORT=5672
CHECKER_RABBITMQ_VHOST=/roxy-wi
CHECKER_RABBITMQ_USER=roxy-wi
CHECKER_RABBITMQ_PASSWORD=GENERATED_STRONG_PASSWORD
CHECKER_RABBITMQ_QUEUE_TYPE=classic
CHECKER_CAPACITY=20

sudo chmod 600 /etc/roxy-wi/checker.env
docker volume create roxy-wi-checker-data
docker run -d --name roxy-wi-checker --restart unless-stopped --stop-timeout 45 \
  --env-file /etc/roxy-wi/checker.env \
  --mount source=roxy-wi-checker-data,target=/var/lib/roxy-checker \
  ghcr.io/roxy-wi/roxy-wi-checker:5.0.0

For two or more replicas, replace SQLite with one shared Checker-owned MySQL/MariaDB database. See the distributed-services installation notes.

Telegram parameters

Telegram notifications can be configured for NGINX, HAProxy and Apache server events. To set up notifications via Telegram, go to the Monitoring => Сhecker: Settings section. Then, in the section for the desired server, select the Telegram channel from the drop-down list where you want to receive notifications.

To add a new channel, go to the Monitoring => Channels section and click Add.

Specify the following parameters in the pop-up window:

Parameter:	Description:
Token	Token received from @father_bot.
Channel	Name of your channel. The channel must be public.
Group	You can specify one of the existing groups. A group is equivalent to a namespace. Members of one group cannot see the servers of the other groups.

Slack parameters

Slack notifications are available for each server starting with version 6.1.0. Setting up notifications for Slack is quite the same as for Telegram.

To add a new Slack channel, go to the Monitoring => Channels section and click Add.

Specify the following parameters in the pop-up window:

Parameter:	Description:
Token	Token received via the Slack API.
Channel	Name of your channel. The channel must be public.
Group	You can specify one of the existing groups. A group is equivalent to a namespace. Members of one group cannot see the servers of the other groups.

Notification settings

Use the Monitoring => Checker settings section to manage notifications for your NGINX, HAProxy and Apache servers.

Select a channel from the drop-down list to enable notifications, or select Disabled to stop sending notifications. If you want to send email notifications to all group members, check the Email checkbox.

Email notifications are not sent to users in the Guest role.

If necessary, check the other checkboxes to be notified of the following information:

Service availability (the Service checkbox)
Changes in the server status (the Status checkbox)
HAProxy backend servers availability (the Backend checkbox)
The Maxconn parameter value for HAProxy; e.g. when the value is about to reach the specified threshold, а notification is sent (the Maxconn checkbox)

Logs

Checker incident events remain available in Monitoring => Checker history. Runtime logs are read on the worker host:

# Package
journalctl -u roxy-wi-checker -f

# Container
docker logs -f roxy-wi-checker

Use Overview or Admin area => Tools to check version 5.0.0, heartbeat, health and active assignment count.

Updating

Package-installed Checker can be updated with dnf upgrade roxy-wi-checker, apt install --only-upgrade roxy-wi-checker or from Admin area => Update.

Container and Kubernetes installations are updated by changing the pinned image tag and performing a rollout. Roxy-WI does not update remote container images. Follow the migration guide when replacing the legacy master.

Checker service description