Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-1177

Delay when client is located on the same node as the regionserver

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Invalid
    • Affects Version/s: 0.19.0
    • Fix Version/s: None
    • Component/s: Performance
    • Labels:
      None
    • Environment:

      Linux 2.6.25 x86_64

      Description

      During testing of HBASE-80, we uncovered a strange 40ms delay for random reads. We ran a series of tests and found that it only happens when the client is on the same node as the RS and for a certain range of payloads (not specifically related to number of columns or size of them, only total payload). It appears to be precisely 40ms every time.

      Unsure if this is particular to our architecture, but it does happen on all nodes we've tried. Issue completely goes away with very large payloads or moving the client.

      Will post a test program tomorrow if anyone can test on a different architecture.

      Making a blocker for 0.20. Since this happens when you have an MR task running local to the RS, and this is what we try to do, might also consider making this a blocker for 0.19.1.

        Attachments

        1. Contribution of getClosest to getRow time.jpg
          166 kB
          Jim Kellerman
        2. Contribution of next to getRow time.jpg
          128 kB
          Jim Kellerman
        3. Contribution of seekTo to getClosest time.jpg
          139 kB
          Jim Kellerman
        4. Elapsed time of RowResults.readFields.jpg
          105 kB
          Jim Kellerman
        5. getRow + round-trip vs # columns.jpg
          111 kB
          Jim Kellerman
        6. getRow times.jpg
          130 kB
          Jim Kellerman
        7. ReadDelayTest.java
          4 kB
          Jonathan Gray
        8. RowResults.readFields zoomed.jpg
          101 kB
          Jim Kellerman
        9. screenshot-1.jpg
          141 kB
          Jim Kellerman
        10. screenshot-2.jpg
          119 kB
          Jim Kellerman
        11. screenshot-3.jpg
          82 kB
          Jonathan Gray
        12. screenshot-4.jpg
          132 kB
          Jim Kellerman
        13. zoom of columns vs round-trip blowup.jpg
          94 kB
          Jim Kellerman

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                streamy Jonathan Gray
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: