HBase
  1. HBase
  2. HBASE-2077

NullPointerException with an open scanner that expired causing an immediate region server shutdown

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.20.2, 0.20.3
    • Fix Version/s: 0.90.4
    • Component/s: regionserver
    • Labels:
      None
    • Environment:

      Hadoop 0.20.0, Mac OS X, Java 6

    • Hadoop Flags:
      Reviewed
    • Release Note:
      Removes lease from lease monitor while operation is running inside the server.

      Description

      2009-12-29 18:05:55,432 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner -4250070597157694417 lease expired
      2009-12-29 18:05:55,443 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer:
      java.lang.NullPointerException
      at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1310)
      at org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:136)
      at org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:127)
      at org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:117)
      at java.util.PriorityQueue.siftDownUsingComparator(PriorityQueue.java:641)
      at java.util.PriorityQueue.siftDown(PriorityQueue.java:612)
      at java.util.PriorityQueue.poll(PriorityQueue.java:523)
      at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:113)
      at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:1776)
      at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:1719)
      at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1944)
      at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648)
      at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
      2009-12-29 18:05:55,446 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 7 on 55260, call next(-4250070597157694417, 10000) from 192.168.1.90:54011: error: java.io.IOException: java.lang.NullPointerException
      java.io.IOException: java.lang.NullPointerException
      at org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:869)
      at org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:859)
      at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1965)
      at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648)
      at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
      Caused by: java.lang.NullPointerException
      at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1310)
      at org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:136)
      at org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:127)
      at org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:117)
      at java.util.PriorityQueue.siftDownUsingComparator(PriorityQueue.java:641)
      at java.util.PriorityQueue.siftDown(PriorityQueue.java:612)
      at java.util.PriorityQueue.poll(PriorityQueue.java:523)
      at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:113)
      at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:1776)
      at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:1719)
      at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1944)
      ... 5 more
      2009-12-29 18:05:55,447 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server Responder, call next(-4250070597157694417, 10000) from 192.168.1.90:54011: output error
      2009-12-29 18:05:55,448 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 7 on 55260 caught: java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:126)
      at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
      at org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1125)
      at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:615)
      at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:679)
      at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:943)

      2009-12-29 18:05:56,322 INFO org.apache.hadoop.ipc.HBaseServer: Stopping server on 55260
      2009-12-29 18:05:56,322 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC Server listener on 55260

      1. [Bug_HBASE-2077]_Fixes_a_very_rare_race_condition_between_lease_expiration_and_renewal.patch
        1 kB
        Sam Pullara
      2. HBASE-2077-redux.patch
        2 kB
        Jean-Daniel Cryans
      3. HBASE-2077-3.patch
        5 kB
        Jean-Daniel Cryans
      4. 2077-suggestion.txt
        4 kB
        stack
      5. 2077-v4.txt
        6 kB
        stack

        Issue Links

          Activity

          Sam Pullara created issue -
          Sam Pullara made changes -
          Field Original Value New Value
          Summary NullPointException with an open scanner that expired causing an immediate region server shutdown NullPointerException with an open scanner that expired causing an immediate region server shutdown
          Sam Pullara made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Jean-Daniel Cryans made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Assignee Sam Pullara [ spullara ]
          Fix Version/s 0.21.0 [ 12313607 ]
          Resolution Fixed [ 1 ]
          Jean-Daniel Cryans made changes -
          Resolution Fixed [ 1 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          Jean-Daniel Cryans made changes -
          Attachment HBASE-2077-redux.patch [ 12436666 ]
          Jean-Daniel Cryans made changes -
          Fix Version/s 0.20.4 [ 12314496 ]
          Fix Version/s 0.21.0 [ 12313607 ]
          Fix Version/s 0.20.3 [ 12314357 ]
          Affects Version/s 0.20.3 [ 12314357 ]
          Priority Major [ 3 ] Critical [ 2 ]
          Jean-Daniel Cryans made changes -
          Status Reopened [ 4 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Jean-Daniel Cryans made changes -
          Resolution Fixed [ 1 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          Jean-Daniel Cryans made changes -
          Attachment HBASE-2077-3.patch [ 12440402 ]
          Jean-Daniel Cryans made changes -
          Fix Version/s 0.20.5 [ 12314800 ]
          Fix Version/s 0.21.0 [ 12313607 ]
          Fix Version/s 0.20.4 [ 12314496 ]
          Jean-Daniel Cryans made changes -
          Link This issue relates to HBASE-2503 [ HBASE-2503 ]
          stack made changes -
          Fix Version/s 0.20.5 [ 12314800 ]
          ryan rawson made changes -
          Fix Version/s 0.90.0 [ 12313607 ]
          Fix Version/s 0.92.0 [ 12314223 ]
          stack made changes -
          Attachment 2077-suggestion.txt [ 12479956 ]
          stack made changes -
          Attachment 2077-v4.txt [ 12479982 ]
          stack made changes -
          Status Reopened [ 4 ] Resolved [ 5 ]
          Fix Version/s 0.90.4 [ 12316406 ]
          Fix Version/s 0.92.0 [ 12314223 ]
          Resolution Fixed [ 1 ]
          stack made changes -
          Release Note Removes lease from lease monitor while operation is running inside the server.

            People

            • Assignee:
              Sam Pullara
              Reporter:
              Sam Pullara
            • Votes:
              2 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 1h
                1h
                Remaining:
                Remaining Estimate - 1h
                1h
                Logged:
                Time Spent - Not Specified
                Not Specified

                  Development