netdata/netdata

<!--startmeta
custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/collectors/python.d.plugin/zscores/README.md"
meta_yaml: "https://github.com/netdata/netdata/edit/master/src/collectors/python.d.plugin/zscores/metadata.yaml"
sidebar_label: "python.d zscores"
learn_status: "Published"
learn_rel_path: "Collecting Metrics/Other"
most_popular: False
message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE COLLECTOR'S metadata.yaml FILE"
endmeta-->

# python.d zscores

Plugin: python.d.plugin
Module: zscores

<img src="https://img.shields.io/badge/maintained%20by-Netdata-%2300ab44" />

## Overview

By using smoothed, rolling [Z-Scores](https://en.wikipedia.org/wiki/Standard_score) for selected metrics or charts you can narrow down your focus and shorten root cause analysis.


This collector uses the [Netdata rest api](/src/web/api/README.md) to get the `mean` and `stddev`
for each dimension on specified charts over a time range (defined by `train_secs` and `offset_secs`).

For each dimension it will calculate a Z-Score as `z = (x - mean) / stddev` (clipped at `z_clip`). Scores are then smoothed over
time (`z_smooth_n`) and, if `mode: 'per_chart'`, aggregated across dimensions to a smoothed, rolling chart level Z-Score at each time step.


This collector is supported on all platforms.

This collector supports collecting metrics from multiple instances of this integration, including remote instances.


### Default Behavior

#### Auto-Detection

This integration doesn't support auto-detection.

#### Limits

The default configuration for this integration does not impose any limits on data collection.

#### Performance Impact

The default configuration for this integration is not expected to impose a significant performance impact on the system.


## Metrics

Metrics grouped by *scope*.

The scope defines the instance that the metric belongs to. An instance is uniquely identified by a set of labels.



### Per python.d zscores instance

These metrics refer to the entire monitored application.

This scope has no labels.

Metrics:

| Metric | Dimensions | Unit |
|:------|:----------|:----|
| zscores.z | a dimension per chart or dimension | z |
| zscores.3stddev | a dimension per chart or dimension | count |



## Alerts

There are no alerts configured by default for this integration.


## Setup

### Prerequisites

#### Python Requirements

This collector will only work with Python 3 and requires the below packages be installed.

```bash
# become netdata user
sudo su -s /bin/bash netdata
# install required packages
pip3 install numpy pandas requests netdata-pandas==0.0.38
```



### Configuration

#### File

The configuration file name for this integration is `python.d/zscores.conf`.


You can edit the configuration file using the `edit-config` script from the
Netdata [config directory](/docs/netdata-agent/configuration/README.md#the-netdata-config-directory).

```bash
cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata
sudo ./edit-config python.d/zscores.conf
```
#### Options

There are 2 sections:

* Global variables
* One or more JOBS that can define multiple different instances to monitor.

The following options can be defined globally: priority, penalty, autodetection_retry, update_every, but can also be defined per JOB to override the global values.

Additionally, the following collapsed table contains all the options that can be configured inside a JOB definition.

Every configuration JOB starts with a `job_name` value which will appear in the dashboard, unless a `name` parameter is specified.


<details><summary>Config options</summary>

| Name | Description | Default | Required |
|:----|:-----------|:-------|:--------:|
| charts_regex | what charts to pull data for - A regex like `system\..*/` or `system\..*/apps.cpu/apps.mem` etc. | system\..* | yes |
| train_secs | length of time (in seconds) to base calculations off for mean and stddev. | 14400 | yes |
| offset_secs | offset (in seconds) preceding latest data to ignore when calculating mean and stddev. | 300 | yes |
| train_every_n | recalculate the mean and stddev every n steps of the collector. | 900 | yes |
| z_smooth_n | smooth the z score (to reduce sensitivity to spikes) by averaging it over last n values. | 15 | yes |
| z_clip | cap absolute value of zscore (before smoothing) for better stability. | 10 | yes |
| z_abs | set z_abs: 'true' to make all zscores be absolute values only. | true | yes |
| burn_in | burn in period in which to initially calculate mean and stddev on every step. | 2 | yes |
| mode | mode can be to get a zscore 'per_dim' or 'per_chart'. | per_chart | yes |
| per_chart_agg | per_chart_agg is how you aggregate from dimension to chart when mode='per_chart'. | mean | yes |
| update_every | Sets the default data collection frequency. | 5 | no |
| priority | Controls the order of charts at the netdata dashboard. | 60000 | no |
| autodetection_retry | Sets the job re-check interval in seconds. | 0 | no |
| penalty | Indicates whether to apply penalty to update_every in case of failures. | yes | no |

</details>

#### Examples

##### Default

Default configuration.

```yaml
local:
  name: 'local'
  host: '127.0.0.1:19999'
  charts_regex: 'system\..*'
  charts_to_exclude: 'system.uptime'
  train_secs: 14400
  offset_secs: 300
  train_every_n: 900
  z_smooth_n: 15
  z_clip: 10
  z_abs: 'true'
  burn_in: 2
  mode: 'per_chart'
  per_chart_agg: 'mean'

```


## Troubleshooting

### Debug Mode

To troubleshoot issues with the `zscores` collector, run the `python.d.plugin` with the debug option enabled. The output
should give you clues as to why the collector isn't working.

- Navigate to the `plugins.d` directory, usually at `/usr/libexec/netdata/plugins.d/`. If that's not the case on
  your system, open `netdata.conf` and look for the `plugins` setting under `[directories]`.

  ```bash
  cd /usr/libexec/netdata/plugins.d/
  ```

- Switch to the `netdata` user.

  ```bash
  sudo -u netdata -s
  ```

- Run the `python.d.plugin` to debug the collector:

  ```bash
  ./python.d.plugin zscores debug trace
  ```