[HBASE-2077] NullPointerException with an open scanner that expired causing an immediate region server shutdown

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.20.2, 0.20.3
• Fix Version/s: 0.90.4
• Component/s: regionserver

Environment

Hadoop 0.20.0, Mac OS X, Java 6

• Hadoop Flags: Reviewed
• Release Note: Removes lease from lease monitor while operation is running inside the server.
• Assignee: Sam Pullara
• Reporter: Sam Pullara
• Created: 30/Dec/09 07:08
• Updated: 30/Sep/16 09:01
• Resolved: 21/Jun/11 20:58

Description

2009-12-29 18:05:55,432 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner -4250070597157694417 lease expired
2009-12-29 18:05:55,443 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer:
java.lang.NullPointerException
at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1310)
at org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:136)
at org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:127)
at org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:117)
at java.util.PriorityQueue.siftDownUsingComparator(PriorityQueue.java:641)
at java.util.PriorityQueue.siftDown(PriorityQueue.java:612)
at java.util.PriorityQueue.poll(PriorityQueue.java:523)
at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:113)
at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:1776)
at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:1719)
at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1944)
at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2009-12-29 18:05:55,446 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 7 on 55260, call next(-4250070597157694417, 10000) from 192.168.1.90:54011: error: java.io.IOException: java.lang.NullPointerException
java.io.IOException: java.lang.NullPointerException
at org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:869)
at org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:859)
at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1965)
at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
Caused by: java.lang.NullPointerException
at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1310)
at org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:136)
at org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:127)
at org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:117)
at java.util.PriorityQueue.siftDownUsingComparator(PriorityQueue.java:641)
at java.util.PriorityQueue.siftDown(PriorityQueue.java:612)
at java.util.PriorityQueue.poll(PriorityQueue.java:523)
at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:113)
at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:1776)
at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:1719)
at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1944)
... 5 more
2009-12-29 18:05:55,447 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server Responder, call next(-4250070597157694417, 10000) from 192.168.1.90:54011: output error
2009-12-29 18:05:55,448 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 7 on 55260 caught: java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:126)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
at org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1125)
at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:615)
at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:679)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:943)

2009-12-29 18:05:56,322 INFO org.apache.hadoop.ipc.HBaseServer: Stopping server on 55260
2009-12-29 18:05:56,322 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC Server listener on 55260

Issue Links

• relates to: HBASE-2503 PriorityQueue isn't thread safe, KeyValueHeap uses it that way
• relates to: HBASE-16737 NPE during close of RegionScanner