Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 0.9.0.0
    • Fix Version/s: None
    • Component/s: KafkaConnect
    • Labels:

      Description

      Copycat needs good metrics for monitoring since that will be the primary insight into the health of connectors as they copy data.

        Activity

        Hide
        ewencp Ewen Cheslack-Postava added a comment -

        Not quite a direct comparison, but some helpful ideas about framework-level metrics we could add:

        One thing we found super helpful in samza was giving a metric that rolls up the percent of time getting records, time processing records, and time sending records to be able to debug these bottleneck issues. Technically you can kind of get this from the clients, but might be nice to roll that up to make it easy.

        We still will need to expose connector-specific metrics, but these are generic ones that can add a lot of value.

        Show
        ewencp Ewen Cheslack-Postava added a comment - Not quite a direct comparison, but some helpful ideas about framework-level metrics we could add: One thing we found super helpful in samza was giving a metric that rolls up the percent of time getting records, time processing records, and time sending records to be able to debug these bottleneck issues. Technically you can kind of get this from the clients, but might be nice to roll that up to make it easy. We still will need to expose connector-specific metrics, but these are generic ones that can add a lot of value.
        Hide
        criccomini Chris Riccomini added a comment -

        One thing we'd like to see here is the ability to hook into the Kafka Connect metrics from our connectors. That way we can add connector-specific metrics, and have them flow through the same pipeline.

        Show
        criccomini Chris Riccomini added a comment - One thing we'd like to see here is the ability to hook into the Kafka Connect metrics from our connectors. That way we can add connector-specific metrics, and have them flow through the same pipeline.
        Hide
        rhauch Randall Hauch added a comment -

        I agree with Chris Riccomini. It's be great to be able for connectors to add their own metrics using the same framework.

        Show
        rhauch Randall Hauch added a comment - I agree with Chris Riccomini . It's be great to be able for connectors to add their own metrics using the same framework.
        Hide
        alla@confluent.io Alla Tumarkin added a comment -

        In order to use KafkaConnect more broadly, it would be useful to have basic metrics around things like:

        • Write rate (meter)
          • by number of messages
          • by message size
        • Error rate (meter)
        • Task failure rate (meter)
        Show
        alla@confluent.io Alla Tumarkin added a comment - In order to use KafkaConnect more broadly, it would be useful to have basic metrics around things like: Write rate (meter) by number of messages by message size Error rate (meter) Task failure rate (meter)

          People

          • Assignee:
            Unassigned
            Reporter:
            ewencp Ewen Cheslack-Postava
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:

              Development