Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-5061

client.id should be set for Connect producers/consumers

    XMLWordPrintableJSON

    Details

      Description

      In order to properly monitor individual tasks using the producer and consumer metrics, we need to have the framework disambiguate them. Currently when we create producers (https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/Worker.java#L362) and create consumers (https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java#L371-L394) the client ID is not being set. You can override it for the entire worker via worker-level producer/consumer overrides, but you can't get per-task metrics.

      There are a couple of things we might want to consider doing here:
      1. Provide default client IDs based on the worker group ID + task ID (providing uniqueness for multiple connect clusters up to the scope of the Kafka cluster they are operating on). This seems ideal since it's a good default; however it is a public-facing change and may need a KIP. Normally I would be less worried about this, but some folks may be relying on picking up metrics without this being set, in which case such a change would break their monitoring.
      2. Allow overriding client.id on a per-connector basis. I'm not sure if this will really be useful or not – it lets you differentiate between metrics for different connectors' tasks, but within a connector, all metrics would go to a single client.id. On the other hand, this makes the tasks act as a single group from the perspective of broker handling of client IDs.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              ewencp Ewen Cheslack-Postava
              Reviewer:
              Randall Hauch
            • Votes:
              3 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: