src/go/plugin/go.d/modules/pulsar/integrations/apache_pulsar.md
<!--startmeta
custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/go/plugin/go.d/modules/pulsar/README.md"
meta_yaml: "https://github.com/netdata/netdata/edit/master/src/go/plugin/go.d/modules/pulsar/metadata.yaml"
sidebar_label: "Apache Pulsar"
learn_status: "Published"
learn_rel_path: "Collecting Metrics/Message Brokers"
most_popular: True
message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE COLLECTOR'S metadata.yaml FILE"
endmeta-->
# Apache Pulsar
<img src="https://netdata.cloud/img/pulsar.svg" width="150"/>
Plugin: go.d.plugin
Module: pulsar
<img src="https://img.shields.io/badge/maintained%20by-Netdata-%2300ab44" />
## Overview
This collector monitors Pulsar servers.
It collects broker statistics using Pulsar's [Prometheus endpoint](https://pulsar.apache.org/docs/en/deploy-monitoring/#broker-stats).
This collector is supported on all platforms.
This collector supports collecting metrics from multiple instances of this integration, including remote instances.
### Default Behavior
#### Auto-Detection
By default, it detects Pulsar instances running on localhost.
#### Limits
The default configuration for this integration does not impose any limits on data collection.
#### Performance Impact
The default configuration for this integration is not expected to impose a significant performance impact on the system.
## Metrics
Metrics grouped by *scope*.
The scope defines the instance that the metric belongs to. An instance is uniquely identified by a set of labels.
- topic_* metrics are available when `exposeTopicLevelMetricsInPrometheus` is set to true.
- subscription_* and namespace_subscription metrics are available when `exposeTopicLevelMetricsInPrometheus` si set to true.
- replication_* and namespace_replication_* metrics are available when replication is configured and `replicationMetricsEnabled` is set to true.
### Per Apache Pulsar instance
These metrics refer to the entire monitored application.
This scope has no labels.
Metrics:
| Metric | Dimensions | Unit |
|:------|:----------|:----|
| pulsar.broker_components | namespaces, topics, subscriptions, producers, consumers | components |
| pulsar.messages_rate | publish, dispatch | messages/s |
| pulsar.throughput_rate | publish, dispatch | KiB/s |
| pulsar.storage_size | used | KiB |
| pulsar.storage_operations_rate | read, write | message batches/s |
| pulsar.msg_backlog | backlog | messages |
| pulsar.storage_write_latency | <=0.5ms, <=1ms, <=5ms, =10ms, <=20ms, <=50ms, <=100ms, <=200ms, <=1s, >1s | entries/s |
| pulsar.entry_size | <=128B, <=512B, <=1KB, <=2KB, <=4KB, <=16KB, <=100KB, <=1MB, >1MB | entries/s |
| pulsar.subscription_delayed | delayed | message batches |
| pulsar.subscription_msg_rate_redeliver | redelivered | messages/s |
| pulsar.subscription_blocked_on_unacked_messages | blocked | subscriptions |
| pulsar.replication_rate | in, out | messages/s |
| pulsar.replication_throughput_rate | in, out | KiB/s |
| pulsar.replication_backlog | backlog | messages |
### Per namespace
TBD
This scope has no labels.
Metrics:
| Metric | Dimensions | Unit |
|:------|:----------|:----|
| pulsar.namespace_broker_components | topics, subscriptions, producers, consumers | components |
| pulsar.namespace_messages_rate | publish, dispatch | messages/s |
| pulsar.namespace_throughput_rate | publish, dispatch | KiB/s |
| pulsar.namespace_storage_size | used | KiB |
| pulsar.namespace_storage_operations_rate | read, write | message batches/s |
| pulsar.namespace_msg_backlog | backlog | messages |
| pulsar.namespace_storage_write_latency | <=0.5ms, <=1ms, <=5ms, =10ms, <=20ms, <=50ms, <=100ms, <=200ms, <=1s, >1s | entries/s |
| pulsar.namespace_entry_size | <=128B, <=512B, <=1KB, <=2KB, <=4KB, <=16KB, <=100KB, <=1MB, >1MB | entries/s |
| pulsar.namespace_subscription_delayed | delayed | message batches |
| pulsar.namespace_subscription_msg_rate_redeliver | redelivered | messages/s |
| pulsar.namespace_subscription_blocked_on_unacked_messages | blocked | subscriptions |
| pulsar.namespace_replication_rate | in, out | messages/s |
| pulsar.namespace_replication_throughput_rate | in, out | KiB/s |
| pulsar.namespace_replication_backlog | backlog | messages |
| pulsar.topic_producers | a dimension per topic | producers |
| pulsar.topic_subscriptions | a dimension per topic | subscriptions |
| pulsar.topic_consumers | a dimension per topic | consumers |
| pulsar.topic_messages_rate_in | a dimension per topic | publishes/s |
| pulsar.topic_messages_rate_out | a dimension per topic | dispatches/s |
| pulsar.topic_throughput_rate_in | a dimension per topic | KiB/s |
| pulsar.topic_throughput_rate_out | a dimension per topic | KiB/s |
| pulsar.topic_storage_size | a dimension per topic | KiB |
| pulsar.topic_storage_read_rate | a dimension per topic | message batches/s |
| pulsar.topic_storage_write_rate | a dimension per topic | message batches/s |
| pulsar.topic_msg_backlog | a dimension per topic | messages |
| pulsar.topic_subscription_delayed | a dimension per topic | message batches |
| pulsar.topic_subscription_msg_rate_redeliver | a dimension per topic | messages/s |
| pulsar.topic_subscription_blocked_on_unacked_messages | a dimension per topic | blocked subscriptions |
| pulsar.topic_replication_rate_in | a dimension per topic | messages/s |
| pulsar.topic_replication_rate_out | a dimension per topic | messages/s |
| pulsar.topic_replication_throughput_rate_in | a dimension per topic | messages/s |
| pulsar.topic_replication_throughput_rate_out | a dimension per topic | messages/s |
| pulsar.topic_replication_backlog | a dimension per topic | messages |
## Alerts
There are no alerts configured by default for this integration.
## Setup
### Prerequisites
No action required.
### Configuration
#### File
The configuration file name for this integration is `go.d/pulsar.conf`.
You can edit the configuration file using the `edit-config` script from the
Netdata [config directory](/docs/netdata-agent/configuration/README.md#the-netdata-config-directory).
```bash
cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata
sudo ./edit-config go.d/pulsar.conf
```
#### Options
The following options can be defined globally: update_every, autodetection_retry.
<details open><summary>Config options</summary>
| Name | Description | Default | Required |
|:----|:-----------|:-------|:--------:|
| update_every | Data collection frequency. | 1 | no |
| autodetection_retry | Recheck interval in seconds. Zero means no recheck will be scheduled. | 0 | no |
| url | Server URL. | http://127.0.0.1:8080/metrics | yes |
| timeout | HTTP request timeout. | 1 | no |
| username | Username for basic HTTP authentication. | | no |
| password | Password for basic HTTP authentication. | | no |
| proxy_url | Proxy URL. | | no |
| proxy_username | Username for proxy basic HTTP authentication. | | no |
| proxy_password | Password for proxy basic HTTP authentication. | | no |
| method | HTTP request method. | GET | no |
| body | HTTP request body. | | no |
| headers | HTTP request headers. | | no |
| not_follow_redirects | Redirect handling policy. Controls whether the client follows redirects. | no | no |
| tls_skip_verify | Server certificate chain and hostname validation policy. Controls whether the client performs this check. | no | no |
| tls_ca | Certification authority that the client uses when verifying the server's certificates. | | no |
| tls_cert | Client TLS certificate. | | no |
| tls_key | Client TLS key. | | no |
</details>
#### Examples
##### Basic
A basic example configuration.
```yaml
jobs:
- name: local
url: http://127.0.0.1:8080/metrics
```
##### HTTP authentication
Basic HTTP authentication.
<details open><summary>Config</summary>
```yaml
jobs:
- name: local
url: http://127.0.0.1:8080/metrics
username: username
password: password
```
</details>
##### HTTPS with self-signed certificate
Do not validate server certificate chain and hostname.
<details open><summary>Config</summary>
```yaml
jobs:
- name: local
url: https://127.0.0.1:8080/metrics
tls_skip_verify: yes
```
</details>
##### Multi-instance
> **Note**: When you define multiple jobs, their names must be unique.
Collecting metrics from local and remote instances.
<details open><summary>Config</summary>
```yaml
jobs:
- name: local
url: http://127.0.0.1:8080/metrics
- name: remote
url: http://192.0.2.1:8080/metrics
```
</details>
## Troubleshooting
### Debug Mode
**Important**: Debug mode is not supported for data collection jobs created via the UI using the Dyncfg feature.
To troubleshoot issues with the `pulsar` collector, run the `go.d.plugin` with the debug option enabled. The output
should give you clues as to why the collector isn't working.
- Navigate to the `plugins.d` directory, usually at `/usr/libexec/netdata/plugins.d/`. If that's not the case on
your system, open `netdata.conf` and look for the `plugins` setting under `[directories]`.
```bash
cd /usr/libexec/netdata/plugins.d/
```
- Switch to the `netdata` user.
```bash
sudo -u netdata -s
```
- Run the `go.d.plugin` to debug the collector:
```bash
./go.d.plugin -d -m pulsar
```
### Getting Logs
If you're encountering problems with the `pulsar` collector, follow these steps to retrieve logs and identify potential issues:
- **Run the command** specific to your system (systemd, non-systemd, or Docker container).
- **Examine the output** for any warnings or error messages that might indicate issues. These messages should provide clues about the root cause of the problem.
#### System with systemd
Use the following command to view logs generated since the last Netdata service restart:
```bash
journalctl _SYSTEMD_INVOCATION_ID="$(systemctl show --value --property=InvocationID netdata)" --namespace=netdata --grep pulsar
```
#### System without systemd
Locate the collector log file, typically at `/var/log/netdata/collector.log`, and use `grep` to filter for collector's name:
```bash
grep pulsar /var/log/netdata/collector.log
```
**Note**: This method shows logs from all restarts. Focus on the **latest entries** for troubleshooting current issues.
#### Docker Container
If your Netdata runs in a Docker container named "netdata" (replace if different), use this command:
```bash
docker logs netdata 2>&1 | grep pulsar
```