src/libnetdata/log/README.md
<!--
title: "Log"
custom_edit_url: https://github.com/netdata/netdata/edit/master/src/libnetdata/log/README.md
sidebar_label: "Log"
learn_status: "Published"
learn_topic_type: "Tasks"
learn_rel_path: "Developers/libnetdata"
-->
# Netdata Logging
This document describes how Netdata generates its own logs, not how Netdata manages and queries logs databases.
## Log sources
Netdata supports the following log sources:
1. **daemon**, logs generated by Netdata daemon.
2. **collector**, logs generated by Netdata collectors, including internal and external ones.
3. **access**, API requests received by Netdata
4. **health**, all alert transitions and notifications
## Log outputs
For each log source, Netdata supports the following output methods:
- **off**, to disable this log source
- **journal**, to send the logs to systemd-journal.
- **syslog**, to send the logs to syslog.
- **system**, to send the output to `stderr` or `stdout` depending on the log source.
- **stdout**, to write the logs to Netdata's `stdout`.
- **stderr**, to write the logs to Netdata's `stderr`.
- **filename**, to send the logs to a file.
For `daemon` and `collector` the default is `journal` when systemd-journal is available.
To decide if systemd-journal is available, Netdata checks:
1. `stderr` is connected to systemd-journald
2. `/run/systemd/journal/socket` exists
3. `/host/run/systemd/journal/socket` exists (`/host` is configurable in containers)
If any of the above is detected, Netdata will select `journal` for `daemon` and `collector` sources.
All other sources default to a file.
## Log formats
| Format | Description |
|---------|--------------------------------------------------------------------------------------------------------|
| journal | journald-specific log format. Automatically selected when logging to systemd-journal. |
| logfmt | logs data as a series of key/value pairs. The default when logging to any output other than `journal`. |
| json | logs data in JSON format. |
## Log levels
Each time Netdata logs, it assigns a priority to the log. It can be one of this (in order of importance):
| Level | Description |
|-----------|----------------------------------------------------------------------------------------|
| emergency | a fatal condition, Netdata will most likely exit immediately after. |
| alert | a very important issue that may affect how Netdata operates. |
| critical | a very important issue the user should know which, Netdata thinks it can survive. |
| error | an error condition indicating that Netdata is trying to do something, but it fails. |
| warning | something unexpected has happened that may or may not affect the operation of Netdata. |
| notice | something that does not affect the operation of Netdata, but the user should notice. |
| info | the default log level about information the user should know. |
| debug | these are more verbose logs that can be ignored. |
## Logs Configuration
In `netdata.conf`, there are the following settings:
```
[logs]
# logs to trigger flood protection = 1000
# logs flood protection period = 1m
# facility = daemon
# level = info
# daemon = journal
# collector = journal
# access = /var/log/netdata/access.log
# health = /var/log/netdata/health.log
```
- `logs to trigger flood protection` and `logs flood protection period` enable logs flood protection for `daemon` and `collector` sources. It can also be configured per log source.
- `facility` is used only when Netdata logs to syslog.
- `level` defines the minimum [log level](#log-levels) of logs that will be logged. This setting is applied only to `daemon` and `collector` sources. It can also be configured per source.
### Configuring log sources
Each for the sources (`daemon`, `collector`, `access`, `health`), accepts the following:
```
source = {FORMAT},level={LEVEL},protection={LOG}/{PERIOD}@{OUTPUT}
```
Where:
- `{FORMAT}`, is one of the [log formats](#log-formats),
- `{LEVEL}`, is the minimum [log level](#log-levels) to be logged,
- `{LOGS}` is the number of `logs to trigger flood protection` configured per output,
- `{PERIOD}` is the equivalent of `logs flood protection period` configured per output,
- `{OUTPUT}` is one of the `[log outputs](#log-outputs),
All parameters can be omitted, except `{OUTPUT}`. If `{OUTPUT}` is the only given parameter, `@` can be omitted.
### Logs rotation
Netdata comes with `logrotate` configuration to rotate its log files periodically.
The default is usually found in `/etc/logrotate.d/netdata`.
Sending a `SIGHUP` to Netdata, will instruct it to re-open all its log files.
## Log Fields
<details>
<summary>All fields exposed by Netdata</summary>
| journal | logfmt | json | Description |
|:--------------------------------------:|:------------------------------:|:------------------------------:|:---------------------------------------------------------------------------------------------------------:|
| `_SOURCE_REALTIME_TIMESTAMP` | `time` | `time` | the timestamp of the event |
| `SYSLOG_IDENTIFIER` | `comm` | `comm` | the program logging the event |
| `ND_LOG_SOURCE` | `source` | `source` | one of the [log sources](#log-sources) |
| `PRIORITY`<br/>numeric | `level`<br/>text | `level`<br/>numeric | one of the [log levels](#log-levels) |
| `ERRNO` | `errno` | `errno` | the numeric value of `errno` |
| `INVOCATION_ID` | - | - | a unique UUID of the Netdata session, reset on every Netdata restart, inherited by systemd when available |
| `CODE_LINE` | - | - | the line number of of the source code logging this event |
| `CODE_FILE` | - | - | the filename of the source code logging this event |
| `CODE_FUNCTION` | - | - | the function name of the source code logging this event |
| `TID` | `tid` | `tid` | the thread id of the thread logging this event |
| `THREAD_TAG` | `thread` | `thread` | the name of the thread logging this event |
| `MESSAGE_ID` | `msg_id` | `msg_id` | see [message IDs](#message-ids) |
| `ND_MODULE` | `module` | `module` | the Netdata module logging this event |
| `ND_NIDL_NODE` | `node` | `node` | the hostname of the node the event is related to |
| `ND_NIDL_INSTANCE` | `instance` | `instance` | the instance of the node the event is related to |
| `ND_NIDL_CONTEXT` | `context` | `context` | the context the event is related to (this is usually the chart name, as shown on netdata dashboards |
| `ND_NIDL_DIMENSION` | `dimension` | `dimension` | the dimension the event is related to |
| `ND_SRC_TRANSPORT` | `src_transport` | `src_transport` | when the event happened during a request, this is the request transport |
| `ND_SRC_IP` | `src_ip` | `src_ip` | when the event happened during an inbound request, this is the IP the request came from |
| `ND_SRC_PORT` | `src_port` | `src_port` | when the event happened during an inbound request, this is the port the request came from |
| `ND_SRC_FORWARDED_HOST` | `src_forwarded_host` | `src_forwarded_host` | the contents of the HTTP header `X-Forwarded-Host` |
| `ND_SRC_FORWARDED_FOR` | `src_forwarded_for` | `src_forwarded_for` | the contents of the HTTP header `X-Forwarded-For` |
| `ND_SRC_CAPABILITIES` | `src_capabilities` | `src_capabilities` | when the request came from a child, this is the communication capabilities of the child |
| `ND_DST_TRANSPORT` | `dst_transport` | `dst_transport` | when the event happened during an outbound request, this is the outbound request transport |
| `ND_DST_IP` | `dst_ip` | `dst_ip` | when the event happened during an outbound request, this is the IP the request destination |
| `ND_DST_PORT` | `dst_port` | `dst_port` | when the event happened during an outbound request, this is the port the request destination |
| `ND_DST_CAPABILITIES` | `dst_capabilities` | `dst_capabilities` | when the request goes to a parent, this is the communication capabilities of the parent |
| `ND_REQUEST_METHOD` | `req_method` | `req_method` | when the event happened during an inbound request, this is the method the request was received |
| `ND_RESPONSE_CODE` | `code` | `code` | when responding to a request, this this the response code |
| `ND_CONNECTION_ID` | `conn` | `conn` | when there is a connection id for an inbound connection, this is the connection id |
| `ND_TRANSACTION_ID` | `transaction` | `transaction` | the transaction id (UUID) of all API requests |
| `ND_RESPONSE_SENT_BYTES` | `sent_bytes` | `sent_bytes` | the bytes we sent to API responses |
| `ND_RESPONSE_SIZE_BYTES` | `size_bytes` | `size_bytes` | the uncompressed bytes of the API responses |
| `ND_RESPONSE_PREP_TIME_USEC` | `prep_ut` | `prep_ut` | the time needed to prepare a response |
| `ND_RESPONSE_SENT_TIME_USEC` | `sent_ut` | `sent_ut` | the time needed to send a response |
| `ND_RESPONSE_TOTAL_TIME_USEC` | `total_ut` | `total_ut` | the total time needed to complete a response |
| `ND_ALERT_ID` | `alert_id` | `alert_id` | the alert id this event is related to |
| `ND_ALERT_EVENT_ID` | `alert_event_id` | `alert_event_id` | a sequential number of the alert transition (per host) |
| `ND_ALERT_UNIQUE_ID` | `alert_unique_id` | `alert_unique_id` | a sequential number of the alert transition (per alert) |
| `ND_ALERT_TRANSITION_ID` | `alert_transition_id` | `alert_transition_id` | the unique UUID of this alert transition |
| `ND_ALERT_CONFIG` | `alert_config` | `alert_config` | the alert configuration hash (UUID) |
| `ND_ALERT_NAME` | `alert` | `alert` | the alert name |
| `ND_ALERT_CLASS` | `alert_class` | `alert_class` | the alert classification |
| `ND_ALERT_COMPONENT` | `alert_component` | `alert_component` | the alert component |
| `ND_ALERT_TYPE` | `alert_type` | `alert_type` | the alert type |
| `ND_ALERT_EXEC` | `alert_exec` | `alert_exec` | the alert notification program |
| `ND_ALERT_RECIPIENT` | `alert_recipient` | `alert_recipient` | the alert recipient(s) |
| `ND_ALERT_VALUE` | `alert_value` | `alert_value` | the current alert value |
| `ND_ALERT_VALUE_OLD` | `alert_value_old` | `alert_value_old` | the previous alert value |
| `ND_ALERT_STATUS` | `alert_status` | `alert_status` | the current alert status |
| `ND_ALERT_STATUS_OLD` | `alert_value_old` | `alert_value_old` | the previous alert value |
| `ND_ALERT_UNITS` | `alert_units` | `alert_units` | the units of the alert |
| `ND_ALERT_SUMMARY` | `alert_summary` | `alert_summary` | the summary text of the alert |
| `ND_ALERT_INFO` | `alert_info` | `alert_info` | the info text of the alert |
| `ND_ALERT_DURATION` | `alert_duration` | `alert_duration` | the duration the alert was in its previous state |
| `ND_ALERT_NOTIFICATION_TIMESTAMP_USEC` | `alert_notification_timestamp` | `alert_notification_timestamp` | the timestamp the notification delivery is scheduled |
| `ND_REQUEST` | `request` | `request` | the full request during which the event happened |
| `MESSAGE` | `msg` | `msg` | the event message |
</details>
### Message IDs
Netdata assigns specific message IDs to certain events:
- `ed4cdb8f1beb4ad3b57cb3cae2d162fa` when a Netdata child connects to this Netdata
- `6e2e3839067648968b646045dbf28d66` when this Netdata connects to a Netdata parent
- `9ce0cb58ab8b44df82c4bf1ad9ee22de` when alerts change state
- `6db0018e83e34320ae2a659d78019fb7` when notifications are sent
You can view these events using the Netdata systemd-journal.plugin at the `MESSAGE_ID` filter,
or using `journalctl` like this:
```bash
# query children connection
journalctl MESSAGE_ID=ed4cdb8f1beb4ad3b57cb3cae2d162fa
# query parent connection
journalctl MESSAGE_ID=6e2e3839067648968b646045dbf28d66
# query alert transitions
journalctl MESSAGE_ID=9ce0cb58ab8b44df82c4bf1ad9ee22de
# query alert notifications
journalctl MESSAGE_ID=6db0018e83e34320ae2a659d78019fb7
```
## Using journalctl to query Netdata logs
The Netdata service's processes execute within the `netdata` journal namespace. To view the Netdata logs, you should
specify the `--namespace=netdata` option.
```bash
# Netdata logs since the last time the service was started
journalctl _SYSTEMD_INVOCATION_ID="$(systemctl show --value --property=InvocationID netdata)" --namespace=netdata
# All netdata logs, the oldest entries are displayed first
journalctl -u netdata --namespace=netdata
# All netdata logs, the newest entries are displayed first
journalctl -u netdata --namespace=netdata -r
```