librenms/librenms

# Dispatcher Service

> Status: Release Candidate

The LibreNMS dispatcher service (`librenms-service.py`) is a new method
of running the poller service at set times. It does not replace the php scripts,
just the cron entries running them.

## External Requirements

### A recent version of Python

The LibreNMS service requires Python 3 and some features require
behaviour only found in Python3.4+.

### Python modules

- PyMySQL is recommended as it requires no C compiler to
  install. MySQLclient can also be used, but does require compilation.
- python-dotenv .env loader
- redis-py 4.0+ and Redis 5.0+ server (if using distributed polling)
- psutil

These can be obtained from your OS package manager, or from PyPI with the below commands.

```bash
pip3 install -r requirements.txt
```

### Redis (distributed polling)

If you want to use distributed polling, you'll need a Redis instance
to coordinate the nodes. It's recommended that you do not share the
Redis database with any other system - by default, Redis supports up
to 16 databases (numbered 0-15). You can also use Redis on a single host if you want

It's strongly recommended that you deploy a resilient cluster of redis
systems, and use redis-sentinel.

You should not rely on the password for the security of your
system. See <https://redis.io/topics/security>

### Memcached (distributed polling)

LibreNMS can still use memcached as a locking mechanism when using
distributed polling.  So you can configure memcached for this purpose
unless you have updates disabled.

See `Locking Mechanisms` at
<https://docs.librenms.org/Extensions/Distributed-Poller/>

### MySQL

You should already have this, but the pollers do need access to the
SQL database. The LibreNMS service runs faster and more aggressively
than the standard poller, so keep an eye on the number of open
connections and other important health metrics.

## Configuration

Connection settings are required in `.env`. The `.env` file is
generated after composer install and `APP_KEY` and `NODE_ID` are set.
Remember that the APP_KEY value must be the same on all your pollers.

```dotenv
#APP_KEY=   #Required, generated by composer install
#NODE_ID=   #Required, generated by composer install

DB_HOST=localhost
DB_DATABASE=librenms
DB_USERNAME=librenms
DB_PASSWORD=
```

### Distributed Polling Configuration

Once you have your Redis database set up, configure it in the .env file on each node. Configure the redis cache driver for distributed locking.

There are a number of options - most of them are optional if your redis instance is standalone and unauthenticated (neither recommended).

```dotenv
##
## Standalone
##
REDIS_HOST=127.0.0.1
REDIS_PORT=6379
REDIS_DB=0
REDIS_TIMEOUT=60

# If requirepass is set in redis set everything above as well as: (recommended)
REDIS_PASSWORD=PasswordGoesHere

# If ACL's are in use, set everything above as well as: (highly recommended)
REDIS_USERNAME=UsernameGoesHere

##
## Sentinel
##
REDIS_SENTINEL=redis-001.example.org:26379,redis-002.example.org:26379,redis-003.example.org:26379
REDIS_SENTINEL_SERVICE=mymaster

# If requirepass is set in sentinel, set everything above as well as: (recommended)
REDIS_SENTINEL_PASSWORD=SentinelPasswordGoesHere

# If ACL's are in use, set everything above as well as: (highly recommended)
REDIS_SENTINEL_USERNAME=SentinelUsernameGoesHere
```

For more information on ACL's, see <https://redis.io/docs/management/security/acl/> 

Note that if you use Sentinel, you may still need `REDIS_PASSWORD`, `REDIS_USERNAME`, `REDIS_DB` and `REDIS_TIMEOUT` - Sentinel just provides the address of the instance currently accepting writes and manages failover. It's possible (and recommended) to have authentication both on Sentinel and the managed Redis instances.

### Basic Configuration

Additional configuration settings can be set in your config.

The defaults are shown here - it's recommended that you at least tune
the number of workers.

!!! setting "poller/distributed"
    ```bash
    lnms config:set service_poller_workers 24
    lnms config:set service_services_workers 8
    lnms config:set service_discovery_workers 16
    ```

Optional Settings

!!! setting "poller/distributed"
    ```bash
    lnms config:set service_poller_frequency 300
    lnms config:set service_services_frequency 300
    lnms config:set service_discovery_frequency 21600
    lnms config:set service_billing_frequency 300
    lnms config:set service_billing_calculate_frequency 60
    lnms config:set service_poller_down_retry 60
    lnms config:set service_loglevel INFO
    lnms config:set service_update_frequency 86400
    ```

There are also some SQL options, but these should be inherited from
your LibreNMS web UI configuration.

Logs are sent to the system logging service (usually `journald` or
`rsyslog`) - see
<https://docs.python.org/3/library/logging.html#logging-levels> for the
options available.

```php
$config['distributed_poller']                    = true;            # Set to true to enable distributed polling
$config['distributed_poller_name']               = php_uname('n');  # Uniquely identifies the poller instance
$config['distributed_poller_group']              = 0;               # Which group to poll
```
### Tuning the number of workers

See https://your_librenms_install/poller

You want to keep Consumed Worker Seconds comfortably below Maximum Worker Seconds. The closer the values are to each other, the flatter the CPU graph of the poller machine. Meaning that you are utilizing your CPU resources well. As long as Consumed WS stays below Maximum WS and Devices Pending is 0, you should be ok.

If Consumed WS is below Maximum WS and Devices Pending is > 0, your hardware is not up to the task.

Maximum WS equals the number of workers multiplied with the number of seconds in the polling period. (default 300)

## Fast Ping

The [fast ping](Fast-Ping-Check.md) scheduler is disabled by default.
You can enable it by setting the following:

```php
$config['service_ping_enabled'] = true;
```

## Watchdog

The watchdog scheduler is disabled by default. You can enable it by setting the following:

```php
$config['service_watchdog_enabled'] = true;
```

The watchdog scheduler will check that the poller log file has been written to within the last poll period. If there is no change to the log file since, the watchdog will restart the polling service. The poller log file is set by `$config['log_file']` and defaults to `./logs/librenms.log`

## Cron Scripts

Once the LibreNMS service is installed, the cron scripts used by LibreNMS to start alerting, polling, discovery and maintenance tasks are no longer required and must be disabled either by removing or commenting them out. The service handles these tasks when enabled. The only cron task enabled after switching to the dispatcher service should be the following:
```
*    *    * * *   librenms    cd /opt/librenms/ && php artisan schedule:run >> /dev/null 2>&1
```

## Service Installation

A systemd unit file is provided - You must adapt `ExecStart` and `WorkingDirectory` if you did not install librenms in `/opt/librenms`

The sysv and upstart init scripts could also be used with a little modification.

### systemd service

A systemd unit file can be found in `misc/librenms.service`. To
install run:
```bash
cp /opt/librenms/misc/librenms.service /etc/systemd/system/librenms.service && systemctl enable --now librenms.service
```



### systemd service with watchdog

This service file is an alternative to the above service file. It uses the systemd WatchdogSec= option to restart the service if it does not receive a keep-alive from the running process.

A systemd unit file can be found in `misc/librenms-watchdog.service`. To
install run:
```bash
cp /opt/librenms/misc/librenms-watchdog.service /etc/systemd/system/librenms.service && systemctl enable --now librenms.service
```

This requires: python3-systemd (or python-systemd on older systems)
or https://pypi.org/project/systemd-python/
If you run this systemd service without python3-systemd it will restart every 30 seconds.

### OS-Specific Instructions

#### RHEL/CentOS

To get the LibreNMS service running under python3.4+ on
RHEL-derivatives with minimal fuss, you can use the software
collections build:

First, enable SCL's on your system:

##### CentOS 7

```
# yum install centos-release-scl
```

##### RHEL 7

```
# subscription-manager repos --enable rhel-server-rhscl-7-rpms
```

Then install and configure the runtime and service:

```
# yum install gcc rh-python36 rh-python36-python-devel epel-release
# yum --enablerepo=remi install redis
# vi /opt/librenms/config.php
# vi /etc/redis.conf
# systemctl enable --now redis.service
# scl enable rh-python36 bash
# change directory to librenms (default /opt/librenms)
# pip3 install -r requirements.txt
# cp /opt/librenms/misc/librenms.service.scl /etc/systemd/system/librenms.service
# systemctl enable --now librenms.service
```

If you want to use another version of python 3, change `rh-python36`
in the unit file and the commands above to match the name of the
replacement scl.

#### Debian/Ubuntu

##### Debian 11 (Bullseye)

Warning: Bullseye provide PHP 7.4 that is too old to run LibreNMS.

##### Debian 12 (Bookworm)

Warning: Bookworm is not available as stable yet (as 2022 november).

Install dependancies
```
apt install python3 python3-mysqldb python3-dotenv
```

Add the `python3-systemd` package for service with watchdog.