Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.9.0.0
    • Fix Version/s: 1.0.0
    • Component/s: KafkaConnect
    • Labels:

      Description

      Kafka Connect needs good metrics for monitoring since that will be the primary insight into the health of connectors as they copy data.

        Issue Links

          Activity

          Hide
          ewencp Ewen Cheslack-Postava added a comment -

          Not quite a direct comparison, but some helpful ideas about framework-level metrics we could add:

          One thing we found super helpful in samza was giving a metric that rolls up the percent of time getting records, time processing records, and time sending records to be able to debug these bottleneck issues. Technically you can kind of get this from the clients, but might be nice to roll that up to make it easy.

          We still will need to expose connector-specific metrics, but these are generic ones that can add a lot of value.

          Show
          ewencp Ewen Cheslack-Postava added a comment - Not quite a direct comparison, but some helpful ideas about framework-level metrics we could add: One thing we found super helpful in samza was giving a metric that rolls up the percent of time getting records, time processing records, and time sending records to be able to debug these bottleneck issues. Technically you can kind of get this from the clients, but might be nice to roll that up to make it easy. We still will need to expose connector-specific metrics, but these are generic ones that can add a lot of value.
          Hide
          criccomini Chris Riccomini added a comment -

          One thing we'd like to see here is the ability to hook into the Kafka Connect metrics from our connectors. That way we can add connector-specific metrics, and have them flow through the same pipeline.

          Show
          criccomini Chris Riccomini added a comment - One thing we'd like to see here is the ability to hook into the Kafka Connect metrics from our connectors. That way we can add connector-specific metrics, and have them flow through the same pipeline.
          Hide
          rhauch Randall Hauch added a comment -

          I agree with Chris Riccomini. It's be great to be able for connectors to add their own metrics using the same framework.

          Show
          rhauch Randall Hauch added a comment - I agree with Chris Riccomini . It's be great to be able for connectors to add their own metrics using the same framework.
          Hide
          alla@confluent.io Alla Tumarkin added a comment -

          In order to use KafkaConnect more broadly, it would be useful to have basic metrics around things like:

          • Write rate (meter)
            • by number of messages
            • by message size
          • Error rate (meter)
          • Task failure rate (meter)
          Show
          alla@confluent.io Alla Tumarkin added a comment - In order to use KafkaConnect more broadly, it would be useful to have basic metrics around things like: Write rate (meter) by number of messages by message size Error rate (meter) Task failure rate (meter)
          Hide
          rhauch Randall Hauch added a comment -

          Completed the first draft of KIP-196. Please discuss on the mailing list (preferably) or here.

          Show
          rhauch Randall Hauch added a comment - Completed the first draft of KIP-196 . Please discuss on the mailing list (preferably) or here.
          Hide
          wushujames James Cheng added a comment - - edited

          Randall Hauch: I'm not a Committer, obviously, but feel free to add me to any PRs if you want an extra pair of eyes.

          Show
          wushujames James Cheng added a comment - - edited Randall Hauch : I'm not a Committer, obviously, but feel free to add me to any PRs if you want an extra pair of eyes.
          Hide
          guozhang Guozhang Wang added a comment -

          Randall Hauch There are still a few subtasks for this JIRA, which is marked as blocker of 1.0.0, can they be completed in the next few days?

          Show
          guozhang Guozhang Wang added a comment - Randall Hauch There are still a few subtasks for this JIRA, which is marked as blocker of 1.0.0, can they be completed in the next few days?
          Hide
          rhauch Randall Hauch added a comment -

          Guozhang Wang, yes, they will be completed in the next few days.

          Show
          rhauch Randall Hauch added a comment - Guozhang Wang , yes, they will be completed in the next few days.
          Hide
          rhauch Randall Hauch added a comment -

          All of the subtasks have been completed, so marking this as fixed.

          Show
          rhauch Randall Hauch added a comment - All of the subtasks have been completed, so marking this as fixed.

            People

            • Assignee:
              rhauch Randall Hauch
              Reporter:
              ewencp Ewen Cheslack-Postava
            • Votes:
              1 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development