HBase
  1. HBase
  2. HBASE-1394

Uploads sometimes fall to 0 requests/second (Binding up on HLog#append?)

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.20.0
    • Component/s: None
    • Labels:
      None

      Description

      Trying to figure why rate sometimes goes to zero.

      Studying the reginoserver, HLog#append looks like a possible culprit.

      "IPC Server handler 7 on 60021" daemon prio=10 tid=0x000000004057dc00 nid=0x1bc4 in Object.wait() [0x0000000043393000..0x0000000043393b80]
         java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:485)
              at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.writeChunk(DFSClient.java:2964)
              - locked <0x00007f9e3e449ff0> (a java.util.LinkedList)
              - locked <0x00007f9e3e449e18> (a org.apache.hadoop.hdfs.DFSClient$DFSOutputStream)
              at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:150)
              at org.apache.hadoop.fs.FSOutputSummer.write1(FSOutputSummer.java:100)
              at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:86)
              - locked <0x00007f9e3e449e18> (a org.apache.hadoop.hdfs.DFSClient$DFSOutputStream)
              at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49)
              at java.io.DataOutputStream.write(DataOutputStream.java:90)
              - locked <0x00007f9e434e5588> (a org.apache.hadoop.fs.FSDataOutputStream)
              at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1020)
              - locked <0x00007f9e434e55c0> (a org.apache.hadoop.io.SequenceFile$Writer)
              at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:984)
              - locked <0x00007f9e434e55c0> (a org.apache.hadoop.io.SequenceFile$Writer)
              at org.apache.hadoop.hbase.regionserver.HLog.doWrite(HLog.java:565)
              at org.apache.hadoop.hbase.regionserver.HLog.append(HLog.java:521)
              - locked <0x00007f9dfa376f70> (a java.lang.Object)
              at org.apache.hadoop.hbase.regionserver.HRegion.update(HRegion.java:1777)
              at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1348)
              at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1289)
              at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdates(HRegionServer.java:1727)
              at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
              at java.lang.reflect.Method.invoke(Method.java:597)
              at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:642)
              at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:911)
      
      1. 1394.patch
        21 kB
        stack
      2. loghowlongittookappendinghlog.patch
        0.9 kB
        stack

        Issue Links

          Activity

          Gavin made changes -
          Link This issue is related to HBASE-7216 [ HBASE-7216 ]
          Gavin made changes -
          Link This issue is related to HBASE-7216 [ HBASE-7216 ]
          binlijin made changes -
          Link This issue is related too HBASE-7216 [ HBASE-7216 ]
          stack made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          stack made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          stack made changes -
          Attachment 1394.patch [ 12408074 ]
          Andrew Purtell made changes -
          Link This issue relates to HBASE-1413 [ HBASE-1413 ]
          stack made changes -
          Fix Version/s 0.20.0 [ 12313474 ]
          stack made changes -
          Priority Major [ 3 ] Critical [ 2 ]
          stack made changes -
          Field Original Value New Value
          Attachment loghowlongittookappendinghlog.patch [ 12407706 ]
          stack created issue -

            People

            • Assignee:
              Unassigned
              Reporter:
              stack
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development