Metrics
In this section, we will introduce the MetricsReporter
and HoodieMetrics
in Hudi. You can view the metrics-related configurations here.
MetricsReporter
MetricsReporter provides APIs for reporting HoodieMetrics
to user-specified backends. Currently, the implementations include InMemoryMetricsReporter, JmxMetricsReporter, MetricsGraphiteReporter and DatadogMetricsReporter. Since InMemoryMetricsReporter is only used for testing, we will introduce the other three implementations.
JmxMetricsReporter
JmxMetricsReporter is an implementation of JMX reporter, which used to report JMX metrics.
Configurations
The following is an example of JmxMetricsReporter
. More detailed configurations can be referenced here.
hoodie.metrics.on=true
hoodie.metrics.reporter.type=JMX
hoodie.metrics.jmx.host=192.168.0.106
hoodie.metrics.jmx.port=4001
Demo
As configured above, JmxMetricsReporter will started JMX server on port 4001. We can start a jconsole to connect to 192.168.0.106:4001. Below is an illustration of monitoring Hudi JMX metrics through jconsole.
MetricsGraphiteReporter
MetricsGraphiteReporter is an implementation of Graphite reporter, which connects to a Graphite server, and send HoodieMetrics
to it.
Configurations
The following is an example of MetricsGraphiteReporter
. More detaile configurations can be referenced here.
hoodie.metrics.on=true
hoodie.metrics.reporter.type=GRAPHITE
hoodie.metrics.graphite.host=192.168.0.106
hoodie.metrics.graphite.port=2003
hoodie.metrics.graphite.metric.prefix=<your metrics prefix>
Demo
As configured above, assuming a Graphite server is running on host 192.168.0.106 and port 2003, a running Hudi job will connect and report metrics data to it. Below is an illustration of monitoring hudi metrics through Graphite.
DatadogMetricsReporter
DatadogMetricsReporter is an implementation of Datadog reporter. A reporter which publishes metric values to Datadog monitoring service via Datadog HTTP API.
Configurations
The following is an example of DatadogMetricsReporter
. More detailed configurations can be referenced here.
hoodie.metrics.on=true
hoodie.metrics.reporter.type=DATADOG
hoodie.metrics.datadog.api.site=EU # or US
hoodie.metrics.datadog.api.key=<your api key>
hoodie.metrics.datadog.metric.prefix=<your metrics prefix>
hoodie.metrics.datadog.api.site
will set the Datadog API site, which determines whether the requests will be sent to api.datadoghq.eu (EU) or api.datadoghq.com (US). Set this according to your Datadog account settings.hoodie.metrics.datadog.api.key
will set the api key.hoodie.metrics.datadog.metric.prefix
will help segregate metrics by setting different prefixes for different jobs. Note that it will use.
to delimit the prefix and the metric name. For example, if the prefix is set tofoo
, thenfoo.
will be prepended to the metric name.