# Introduction

Apache DolphinScheduler exports metrics for system observability. We use [Micrometer](https://micrometer.io/) as application metrics facade.
Currently, we only support `Prometheus Exporter` but more are coming soon.

## Quick Start

- We enable Apache DolphinScheduler to export metrics in `standalone` mode to help users get hands dirty easily.
- After triggering tasks in `standalone` mode, you could access metrics list by visiting url `http://localhost:12345/dolphinscheduler/actuator/metrics`.
- After triggering tasks in `standalone` mode, you could access `prometheus-format` metrics by visiting url `http://localhost:12345/dolphinscheduler/actuator/prometheus`.
- For a better experience with `Prometheus` and `Grafana`, we have prepared the out-of-the-box `Grafana` configurations for you, you could find the `Grafana` dashboards
  at `dolphinscheduler-meter/resources/grafana` and directly import these dashboards to your `Grafana` instance.
- If you want to try with `docker`, you can use the following command to start the out-of-the-box `Prometheus` and `Grafana`:

```shell
cd dolphinscheduler-meter/src/main/resources/grafana-demo
docker compose up
```

then access the `Grafana` by the url: `http://localhost:3001` for dashboards.

![image.png](../../../../img/metrics/metrics-master.png)
![image.png](../../../../img/metrics/metrics-worker.png)
![image.png](../../../../img/metrics/metrics-datasource.png)

- If you prefer to have some experiments in `cluster` mode, please refer to the [Configuration](#configuration) section below:

## Configuration

You could access the metrics by the url `http://ip:port/actuator/prometheus`.

The exporter port is the `server.port` defined in application.yaml, e.g: master: `server.port: 5679`, worker: `server.port: 1235`, alert: `server.port: 50053`, api: `server.port: 12345`.

For example, you can get the master metrics by `curl http://localhost:5679/actuator/prometheus`.

## Endpoint Security Authentication

Spring Boot Actuator provides a series of HTTP endpoints for monitoring and managing applications. By default, these endpoints may be exposed on public or internal networks, posing a risk of information leakage.

Therefore, we strongly recommend enabling endpoint security authentication in production environments. Currently, HTTP Basic Authentication is supported, and you can choose either of the following two configuration methods.

- You can configure it individually in the application.yaml file of each service:

```yaml
management:
  security:
    enabled: true
    username: username
    password: password
    exclude: [health,metrics]
```

- Alternatively, you can configure it globally via the dolphinscheduler_env.sh file:

```sh
export MANAGEMENT_SECURITY_ENABLED=true
export MANAGEMENT_SECURITY_USERNAME=username
export MANAGEMENT_SECURITY_PASSWORD=password
export MANAGEMENT_SECURITY_EXCLUDE="health,metrics"
```

- After enabling security authentication, you could access `prometheus-format` metrics using the following commands:

```sh
curl -u username:password 'http://localhost:12345/dolphinscheduler/actuator/prometheus'
curl -H 'Authorization: Basic xxxxx' 'http://localhost:12345/dolphinscheduler/actuator/prometheus' 
```

## Naming Convention & Mapping

- Naming of Apache DolphinScheduler metrics follows the officially-recommended approach by [Micrometer](https://github.com/micrometer-metrics/micrometer-docs/blob/main/src/docs/concepts/naming.adoc)
- `Micrometer` automatically maps the metrics names to suit the external metrics system you configured. Currently, we only support `Prometheus Exporter` but more are coming soon.

### Prometheus

- all dots mapped to underscores
- metric name starting with number added with prefix `m_`
- COUNTER: add `_total` suffix if not ending with it
- LONG_TASK_TIMER: `_timer_seconds` suffix added if not ending with them
- GAUGE: `_baseUnit` suffix added if not ending with it

## Dolphin Scheduler Metrics Cheatsheet

- We categorize metrics by dolphin scheduler components such as `master server`, `worker server`, `api server` and `alert server`.
- Although task / workflow related metrics exported by `master server` and `worker server`, we categorize them separately for users to query them more conveniently.

### Task Related Metrics

- ds.task.instance.count: (counter) the number of task instances, sliced by the tag `state`:
  - timeout: the number of timeout tasks
  - finish: the number of finished tasks, both succeeded and failed included
  - success: the number of successful tasks
  - fail: the number of failed tasks
  - stop: the number of stopped tasks
  - retry: the number of retried tasks
  - submit: the number of submitted tasks
  - failover: the number of task fail-overs
- ds.task.dispatch.count: (counter) the number of tasks dispatched to worker
- ds.task.dispatch.failure.count: (counter) the number of tasks failed to dispatch, retry failure included
- ds.task.dispatch.error.count: (counter) the number of task dispatch errors
- ds.task.execution.count.by.type: (counter) the number of task executions grouped by tag `task_type`
- ds.task.prepared: (gauge) the number of tasks prepared for task queue
- ds.task.execution.count: (counter) the number of executed tasks
- ds.task.execution.duration: (histogram) duration of task executions

### Workflow Related Metrics

- ds.workflow.create.command.count: (counter) the number of commands created and inserted by workflows
- ds.workflow.instance.submit.count: (counter) the number of submitted workflow instances
- ds.workflow.instance.running: (gauge) the number of running workflow instances
- ds.workflow.instance.count: (counter) the number of workflow instances, sliced by tags `process.definition.code` and `state`. To monitor a specific workflow, you could filter the metrics by tag `process.definition.code`, which refers to the definition code of your workflow. There are seven different states for workflow instances as follows:
  - submit: the number of submitted workflow instances
  - timeout: the number of timeout workflow instances
  - finish: the number of finished workflow instances, both successes and failures included
  - success: the number of successful workflow instances
  - fail: the number of failed workflow instances
  - stop: the number of stopped workflow instances
  - failover: the number of workflow instance fail-overs

### RPC Related Metrics

- ds.rpc.client.sync.request.exception.count: (counter) the number of exceptions occurred in sync rpc requests
- ds.rpc.client.sync.request.duration.time: (histogram) the time cost of sync rpc requests

### Master Server Metrics

- ds.master.overload.count: (counter) the number of times the master overloaded
- ds.master.consume.command.count: (counter) the number of commands consumed by master
- ds.master.scheduler.failover.check.count: (counter) the number of scheduler (master) fail-over checks
- ds.master.scheduler.failover.check.time: (histogram) the total time cost of scheduler (master) fail-over checks
- ds.master.quartz.job.executed: the total number of quartz jobs executed
- ds.master.quartz.job.execution.time: the total execution time of quartz jobs

### Worker Server Metrics

- ds.worker.overload.count: (counter) the number of times the worker overloaded
- ds.worker.task: (gauge) the number of tasks on the worker, including pending and running ones
- ds.worker.execute.queue.size: (gauge) the number of pending tasks on the worker
- ds.worker.active.execute.thread: (gauge) the number of running tasks on the worker
- ds.worker.memory.available: (gauge) the available physical memory of the worker (GB)
- ds.worker.cpu.usage: (gauge) the cpu usage percentage of the worker
- ds.worker.memory.usage: (gauge) the memory usage percentage of the worker
- ds.worker.full.submit.queue.count: (counter) the number of times the worker's submit queue being full
- ds.worker.resource.download.count: (counter) the number of downloaded resource files on workers, sliced by tag `status`
- ds.worker.resource.download.duration: (histogram) the time cost of resource download on workers
- ds.worker.resource.download.size: (histogram) the sizes of downloaded resource files on workers (bytes)

### Api Server Metrics

- ds.api.request.count: (counter) the number of requests received by the api server
- ds.api.response.count: (counter) the number of responses received by the api server, sliced by tag `code`
- ds.api.response.time: (timer) the response time distribution of the api server, sliced by tag `user_id`
- ds.api.resource.upload.size: (histogram) size distribution of resource files uploaded by the api server (bytes)
- ds.api.resource.download.size: (histogram) size distribution of resource files download by the api server (bytes)

### Alert Server Related

- ds.alert.send.count: (counter) the number of sent alerts, sliced by tag `status`
- ds.alert.pending: (gauge) the number of alerts waiting to get fired

**In each server, there are some default system-level metrics related to `database connection`, `JVM`, etc. We list them below for your reference:**

### Database Related Metrics (Default)

- hikaricp.connections: the total number of connections
- hikaricp.connections.creation: connection creation time (max, count, sum included)
- hikaricp.connections.acquire: connection acquirement time (max, count, sum included)
- hikaricp.connections.usage: connection usage time (max, count, sum included)
- hikaricp.connections.max: the max number of connections
- hikaricp.connections.min: the min number of connections
- hikaricp.connections.active: the number of active connections
- hikaricp.connections.idle: the number of idle connections
- hikaricp.connections.pending: the number of pending connections
- hikaricp.connections.timeout: the total number of timeout connections
- jdbc.connections.max: the max number of active connections that can be allocated at the same time
- jdbc.connections.min: the min number of idle connections in the pool
- jdbc.connections.idle: the number of established but idle connections
- jdbc.connections.active: the current number of active connections allocated from the data source

### JVM Related Metrics (Default)

- jvm.buffer.total.capacity: an estimate of the total capacity of the buffers in the pool
- jvm.buffer.count: an estimate of the number of buffers in the pool
- jvm.buffer.memory.used: an estimate of the memory that the JVM is using for this buffer pool
- jvm.memory.committed: the amount of memory in bytes committed for the JVM to use
- jvm.memory.max: the maximum amount of memory in bytes that can be used for memory management
- jvm.memory.used: the amount of used memory
- jvm.threads.peak: the peak live thread count since the JVM started or peak reset
- jvm.threads.states: the current number of threads having NEW state
- jvm.gc.memory.allocated: incremented for an increase in the size of the (young) heap memory pool after one GC to before the next
- jvm.gc.max.data.size: max size of long-lived heap memory pool
- jvm.gc.pause: time spent in GC pause (count, sum, max included)
- jvm.gc.live.data.size: the size of long-lived heap memory pool after reclamation
- jvm.gc.memory.promoted: the count of positive increases in the size of the old generation memory pool before GC to after GC.
- jvm.classes.loaded: the number of classes currently loaded in the JVM
- jvm.threads.live: the current number of live threads including both daemon and non-daemon threads
- jvm.threads.daemon: the current number of live daemon threads
- jvm.classes.unloaded: the total number of classes unloaded since the JVM started
- process.cpu.usage: the `recent cpu usage` for the JVM process
- process.start.time: start time of the process since unix epoch
- process.uptime: the uptime of the JVM

### Others (Default)

- jetty.threads.config.max: the max number of threads in the pool
- jetty.threads.config.min: the min number of threads in the pool
- jetty.threads.current: the total number of threads in the pool
- jetty.threads.idle: the number of idle threads in the pool
- jetty.threads.busy: the number of busy threads in the pool
- jetty.threads.jobs: number of queued jobs waiting for a thread
- process.files.max: the max number of file descriptors
- process.files.open: the number of open file descriptors
- system.cpu.usage: the recent cpu usage for the whole system
- system.cpu.count: the number of processors available to the JVM
- system.load.average.1m: the total number of runnable entities queued to available processors and runnable entities running on the available processors averaged over a period
- logback.events: the number of events that made it to the logs grouped by the tag `level`
- http.server.requests: total number of http requests

