Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: Impala 2.8.0
    • Fix Version/s: Impala 2.9.0
    • Component/s: Backend
    • Labels:
      None

      Description

      Expose the following Hadoop metrics:
      hedgedReadOps - the number of hedged reads that have occurred
      hedgeReadOpsWin - the number of times the hedged read returned faster than the original read

        Issue Links

          Activity

          Show
          sailesh Sailesh Mukil added a comment - https://github.com/apache/incubator-impala/commit/c826af6831719cfe5e967a56099493a103c68a90
          Hide
          sailesh Sailesh Mukil added a comment -

          Just seeing these comments.
          Silvius Rus https://issues.apache.org/jira/browse/HDFS-11531
          Richard Walshe 2 metrics have been added as mentioned in the commit.

          Show
          sailesh Sailesh Mukil added a comment - Just seeing these comments. Silvius Rus https://issues.apache.org/jira/browse/HDFS-11531 Richard Walshe 2 metrics have been added as mentioned in the commit.
          Hide
          richardatcloudera Richard Walshe added a comment -

          Would it be possible to add a metric for the current active hedged reads?

          Show
          richardatcloudera Richard Walshe added a comment - Would it be possible to add a metric for the current active hedged reads?
          Hide
          srus Silvius Rus added a comment -

          If HDFS adds this in their native client, I don't think the work on our side would be too hard, probably < 1 wk.

          Matthew Jacobs is there a Hadoop JIRA?

          Show
          srus Silvius Rus added a comment - If HDFS adds this in their native client, I don't think the work on our side would be too hard, probably < 1 wk. Matthew Jacobs is there a Hadoop JIRA?
          Hide
          mjacobs Matthew Jacobs added a comment -

          The hedged read metrics (read ops, read wins) should be exposed via the native client. Right now it's only available in Java.

          The metrics are exposed on the client (process wide for impala) rather than per file as they are for the other hdfs stats we collect, so we would collect the metrics and they would be exposed per process (e.g. metrics page) rather than what we have for other hdfs metrics which are per file and show up in the profile. How helpful that is, I don't know, but that's the only way we'd be able to gather and expose them.

          If HDFS adds this in their native client, I don't think the work on our side would be too hard, probably < 1 wk.

          Show
          mjacobs Matthew Jacobs added a comment - The hedged read metrics (read ops, read wins) should be exposed via the native client. Right now it's only available in Java. The metrics are exposed on the client (process wide for impala) rather than per file as they are for the other hdfs stats we collect, so we would collect the metrics and they would be exposed per process (e.g. metrics page) rather than what we have for other hdfs metrics which are per file and show up in the profile. How helpful that is, I don't know, but that's the only way we'd be able to gather and expose them. If HDFS adds this in their native client, I don't think the work on our side would be too hard, probably < 1 wk.

            People

            • Assignee:
              sailesh Sailesh Mukil
              Reporter:
              srus Silvius Rus
            • Votes:
              1 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development