HBase
  1. HBase
  2. HBASE-1177

Delay when client is located on the same node as the regionserver

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Invalid
    • Affects Version/s: 0.19.0
    • Fix Version/s: None
    • Component/s: Performance
    • Labels:
      None
    • Environment:

      Linux 2.6.25 x86_64

      Description

      During testing of HBASE-80, we uncovered a strange 40ms delay for random reads. We ran a series of tests and found that it only happens when the client is on the same node as the RS and for a certain range of payloads (not specifically related to number of columns or size of them, only total payload). It appears to be precisely 40ms every time.

      Unsure if this is particular to our architecture, but it does happen on all nodes we've tried. Issue completely goes away with very large payloads or moving the client.

      Will post a test program tomorrow if anyone can test on a different architecture.

      Making a blocker for 0.20. Since this happens when you have an MR task running local to the RS, and this is what we try to do, might also consider making this a blocker for 0.19.1.

      1. RowResults.readFields zoomed.jpg
        101 kB
        Jim Kellerman
      2. Elapsed time of RowResults.readFields.jpg
        105 kB
        Jim Kellerman
      3. zoom of columns vs round-trip blowup.jpg
        94 kB
        Jim Kellerman
      4. getRow + round-trip vs # columns.jpg
        111 kB
        Jim Kellerman
      5. Contribution of getClosest to getRow time.jpg
        166 kB
        Jim Kellerman
      6. Contribution of next to getRow time.jpg
        128 kB
        Jim Kellerman
      7. Contribution of seekTo to getClosest time.jpg
        139 kB
        Jim Kellerman
      8. getRow times.jpg
        130 kB
        Jim Kellerman
      9. screenshot-4.jpg
        132 kB
        Jim Kellerman
      10. screenshot-3.jpg
        82 kB
        Jonathan Gray
      11. screenshot-2.jpg
        119 kB
        Jim Kellerman
      12. screenshot-1.jpg
        141 kB
        Jim Kellerman
      13. ReadDelayTest.java
        4 kB
        Jonathan Gray

        Issue Links

          Activity

          No work has yet been logged on this issue.

            People

            • Assignee:
              Unassigned
              Reporter:
              Jonathan Gray
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development