Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-1177

Delay when client is located on the same node as the regionserver

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Invalid
    • 0.19.0
    • None
    • Performance
    • None

    • Linux 2.6.25 x86_64

    Description

      During testing of HBASE-80, we uncovered a strange 40ms delay for random reads. We ran a series of tests and found that it only happens when the client is on the same node as the RS and for a certain range of payloads (not specifically related to number of columns or size of them, only total payload). It appears to be precisely 40ms every time.

      Unsure if this is particular to our architecture, but it does happen on all nodes we've tried. Issue completely goes away with very large payloads or moving the client.

      Will post a test program tomorrow if anyone can test on a different architecture.

      Making a blocker for 0.20. Since this happens when you have an MR task running local to the RS, and this is what we try to do, might also consider making this a blocker for 0.19.1.

      Attachments

        1. zoom of columns vs round-trip blowup.jpg
          94 kB
          Jim Kellerman
        2. screenshot-4.jpg
          132 kB
          Jim Kellerman
        3. screenshot-3.jpg
          82 kB
          Jonathan Gray
        4. screenshot-2.jpg
          119 kB
          Jim Kellerman
        5. screenshot-1.jpg
          141 kB
          Jim Kellerman
        6. RowResults.readFields zoomed.jpg
          101 kB
          Jim Kellerman
        7. ReadDelayTest.java
          4 kB
          Jonathan Gray
        8. getRow times.jpg
          130 kB
          Jim Kellerman
        9. getRow + round-trip vs # columns.jpg
          111 kB
          Jim Kellerman
        10. Elapsed time of RowResults.readFields.jpg
          105 kB
          Jim Kellerman
        11. Contribution of seekTo to getClosest time.jpg
          139 kB
          Jim Kellerman
        12. Contribution of next to getRow time.jpg
          128 kB
          Jim Kellerman
        13. Contribution of getClosest to getRow time.jpg
          166 kB
          Jim Kellerman

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. HBASE-2125 tcpnodelay is false by default; it should be true?

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-8069 Enable TCP_NODELAY by default for IPC

          • Major - Major loss of function.
          • Closed

          Activity

            People

              Unassigned Unassigned
              streamy Jonathan Gray
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: