Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-2077

NullPointerException with an open scanner that expired causing an immediate region server shutdown

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 0.20.2, 0.20.3
    • Fix Version/s: 0.90.4
    • Component/s: regionserver
    • Labels:
      None
    • Environment:

      Hadoop 0.20.0, Mac OS X, Java 6

    • Hadoop Flags:
      Reviewed
    • Release Note:
      Removes lease from lease monitor while operation is running inside the server.

      Description

      2009-12-29 18:05:55,432 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner -4250070597157694417 lease expired
      2009-12-29 18:05:55,443 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer:
      java.lang.NullPointerException
      at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1310)
      at org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:136)
      at org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:127)
      at org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:117)
      at java.util.PriorityQueue.siftDownUsingComparator(PriorityQueue.java:641)
      at java.util.PriorityQueue.siftDown(PriorityQueue.java:612)
      at java.util.PriorityQueue.poll(PriorityQueue.java:523)
      at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:113)
      at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:1776)
      at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:1719)
      at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1944)
      at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648)
      at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
      2009-12-29 18:05:55,446 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 7 on 55260, call next(-4250070597157694417, 10000) from 192.168.1.90:54011: error: java.io.IOException: java.lang.NullPointerException
      java.io.IOException: java.lang.NullPointerException
      at org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:869)
      at org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:859)
      at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1965)
      at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648)
      at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
      Caused by: java.lang.NullPointerException
      at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1310)
      at org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:136)
      at org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:127)
      at org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:117)
      at java.util.PriorityQueue.siftDownUsingComparator(PriorityQueue.java:641)
      at java.util.PriorityQueue.siftDown(PriorityQueue.java:612)
      at java.util.PriorityQueue.poll(PriorityQueue.java:523)
      at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:113)
      at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:1776)
      at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:1719)
      at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1944)
      ... 5 more
      2009-12-29 18:05:55,447 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server Responder, call next(-4250070597157694417, 10000) from 192.168.1.90:54011: output error
      2009-12-29 18:05:55,448 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 7 on 55260 caught: java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:126)
      at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
      at org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1125)
      at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:615)
      at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:679)
      at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:943)

      2009-12-29 18:05:56,322 INFO org.apache.hadoop.ipc.HBaseServer: Stopping server on 55260
      2009-12-29 18:05:56,322 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC Server listener on 55260

        Attachments

        1. [Bug_HBASE-2077]_Fixes_a_very_rare_race_condition_between_lease_expiration_and_renewal.patch
          1 kB
          Sam Pullara
        2. HBASE-2077-redux.patch
          2 kB
          Jean-Daniel Cryans
        3. HBASE-2077-3.patch
          5 kB
          Jean-Daniel Cryans
        4. 2077-suggestion.txt
          4 kB
          stack
        5. 2077-v4.txt
          6 kB
          stack

          Issue Links

            Activity

              People

              • Assignee:
                spullara Sam Pullara
                Reporter:
                spullara Sam Pullara
              • Votes:
                2 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - 1h
                  1h
                  Remaining:
                  Remaining Estimate - 1h
                  1h
                  Logged:
                  Time Spent - Not Specified
                  Not Specified