HBase
  1. HBase
  2. HBASE-3869

RegionServer metrics - add read and write byte-transfer statistics

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: metrics
    • Labels:
      None

      Description

      It would be beneficial to have the "data transfer weight" of reads and writes per region server.

      HBASE-3647 split out the read/write metric requests from the uber-"request" metric - which is great.

      But there isn't a notion of "data transfer weight" and this is why it's important: the read metrics are effectively RPC-based. Thus, with a scan caching of 500, there is 1 RPC call every 500 rows read (and 1 'read' metric increment). And this metric doesn't indicate how much data is being transferred (e.g., a read with 50 attributes will probably cost a lot more than a read with 5 attributes).

        Issue Links

          Activity

          Hide
          Lars Hofhansl added a comment -

          There is a global metric that accumulates the size of all KVs read by a region server, so it's not what you meant, I guess.

          Show
          Lars Hofhansl added a comment - There is a global metric that accumulates the size of all KVs read by a region server, so it's not what you meant, I guess.
          Hide
          Doug Meil added a comment -

          Specifically, are the "KV-size metrics" for every KV transferred between client and RS? (e.g., read vs. write on table1:cf1:a1, table1:cf1:ca2, etc.) That's certainly more than what I asking for, but that would do it.

          Show
          Doug Meil added a comment - Specifically, are the "KV-size metrics" for every KV transferred between client and RS? (e.g., read vs. write on table1:cf1:a1, table1:cf1:ca2, etc.) That's certainly more than what I asking for, but that would do it.
          Hide
          Doug Meil added a comment -

          What's a specific example "KV-size type metrics"?

          Show
          Doug Meil added a comment - What's a specific example "KV-size type metrics"?
          Hide
          Lars Hofhansl added a comment -

          We now have KV-size type metrics. Is that good enough?

          Show
          Lars Hofhansl added a comment - We now have KV-size type metrics. Is that good enough?
          Hide
          Doug Meil added a comment -

          These read/write RS-client transfer metrics will be the most meaningful to developers in terms of measuring their cluster's level of HBase activity. The other metrics (e.g., read ops, write ops, etc.) are useful, but supplementary to these.

          Show
          Doug Meil added a comment - These read/write RS-client transfer metrics will be the most meaningful to developers in terms of measuring their cluster's level of HBase activity. The other metrics (e.g., read ops, write ops, etc.) are useful, but supplementary to these.
          Hide
          Doug Meil added a comment -

          re: "try to account for the amount of data being sent/received by the region server to the client?"

          That's pretty much what I'm looking for, actually.

          Show
          Doug Meil added a comment - re: "try to account for the amount of data being sent/received by the region server to the client?" That's pretty much what I'm looking for, actually.
          Hide
          Jean-Daniel Cryans added a comment -

          I agree that not all puts and scans were created equal. Do you think we should try to account for the amount of data being sent/received by the region server to the client? IMO this wouldn't be hard to do since we have the size of everything so along with incrementing counters we would also increment size counters using the objects' heapSize().

          Show
          Jean-Daniel Cryans added a comment - I agree that not all puts and scans were created equal. Do you think we should try to account for the amount of data being sent/received by the region server to the client? IMO this wouldn't be hard to do since we have the size of everything so along with incrementing counters we would also increment size counters using the objects' heapSize().

            People

            • Assignee:
              Unassigned
              Reporter:
              Doug Meil
            • Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:

                Development