Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-9066

Kafka Connect JMX : source & sink task metrics missing for tasks in failed state

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.1.1
    • Fix Version/s: 2.6.0, 2.4.2, 2.7.0, 2.5.2
    • Component/s: KafkaConnect
    • Labels:
      None

      Description

      Overview

      Kafka Connect exposes various metrics via JMX. Those metrics can be exported i.e. by Prometheus JMX Exporter for further processing.
      One of crucial attributes is connector's task status.

      According to official Kafka docs, status is available as status attribute of following MBean:

      kafka.connect:type=connector-task-metrics,connector="{connector}",task="{task}"status - The status of the connector task. One of 'unassigned', 'running', 'paused', 'failed', or 'destroyed'.

      Issue

      Generally connector-task-metrics are exposed propery for tasks in running status but not exposed at all if task is failed.

      Failed Task appears properly with failed status when queried via REST API:

       

      $ curl -X GET -u 'user:pass' http://kafka-connect.mydomain.com/connectors/customerconnector/status
      
      {"name":"customerconnector","connector":{"state":"RUNNING","worker_id":"kafka-connect.mydomain.com:8080"},"tasks":[{"id":0,"state":"FAILED","worker_id":"kafka-connect.mydomain.com:8080","trace":"org.apache.kafka.connect.errors.ConnectException: Received DML 'DELETE FROM mysql.rds_sysinfo ......"}],"type":"source"}
      $ 

       

      Failed Task doesn't appear as bean with connector-task-metrics type when queried via JMX:

       

      $ echo "beans -d kafka.connect" | java -jar target/jmxterm-1.1.0-SNAPSHOT-uber.jar -l localhost:8081 -n -v silent | grep connector=customerconnector
      kafka.connect:connector=customerconnector,task=0,type=task-error-metricskafka.connect:connector=customerconnector,type=connector-metrics
      $
      

      Expected result

      It is expected, that bean with connector-task-metrics type will appear also for tasks that failed.
      Below is example of how beans are properly registered for tasks in Running state:

       

      $ echo "get -b kafka.connect:connector=sinkConsentSubscription-1000,task=0,type=connector-task-metrics status" | java -jar target/jmxterm-1.1.0-SNAPSHOT-ube r.jar -l localhost:8081 -n -v silent
      status = running;
      $
      

       

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                ChrisEgerton Chris Egerton
                Reporter:
                mikouaj Mikołaj Stefaniak
                Reviewer:
                Randall Hauch
              • Votes:
                6 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: