Description
Overview
Kafka Connect exposes various metrics via JMX. Those metrics can be exported i.e. by Prometheus JMX Exporter for further processing.
One of crucial attributes is connector's task status.
According to official Kafka docs, status is available as status attribute of following MBean:
kafka.connect:type=connector-task-metrics,connector="{connector}",task="{task}"status - The status of the connector task. One of 'unassigned', 'running', 'paused', 'failed', or 'destroyed'.
Issue
Generally connector-task-metrics are exposed propery for tasks in running status but not exposed at all if task is failed.
Failed Task appears properly with failed status when queried via REST API:
$ curl -X GET -u 'user:pass' http://kafka-connect.mydomain.com/connectors/customerconnector/status {"name":"customerconnector","connector":{"state":"RUNNING","worker_id":"kafka-connect.mydomain.com:8080"},"tasks":[{"id":0,"state":"FAILED","worker_id":"kafka-connect.mydomain.com:8080","trace":"org.apache.kafka.connect.errors.ConnectException: Received DML 'DELETE FROM mysql.rds_sysinfo ......"}],"type":"source"} $
Failed Task doesn't appear as bean with connector-task-metrics type when queried via JMX:
$ echo "beans -d kafka.connect" | java -jar target/jmxterm-1.1.0-SNAPSHOT-uber.jar -l localhost:8081 -n -v silent | grep connector=customerconnector
kafka.connect:connector=customerconnector,task=0,type=task-error-metricskafka.connect:connector=customerconnector,type=connector-metrics
$
Expected result
It is expected, that bean with connector-task-metrics type will appear also for tasks that failed.
Below is example of how beans are properly registered for tasks in Running state:
$ echo "get -b kafka.connect:connector=sinkConsentSubscription-1000,task=0,type=connector-task-metrics status" | java -jar target/jmxterm-1.1.0-SNAPSHOT-ube r.jar -l localhost:8081 -n -v silent
status = running;
$
Attachments
Issue Links
- relates to
-
KAFKA-10146 Backport KAFKA-9066 to 2.5 and 2.4 branches
-
- Resolved
-
- links to