[HBASE-2073] IllegalArgumentException causing regionserver failure

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Won't Fix
• Affects Version/s: 0.20.2
• Component/s: regionserver

Environment

Ubuntu 8.10, Java 1.6.0_10, HBase 0.20.2

• Assignee: Unassigned
• Reporter: Greg Lu
• Created: 29/Dec/09 17:41
• Updated: 11/Jun/22 23:18
• Resolved: 24/Feb/12 20:44

Description

After a regionserver went down last night, I checked its logs and found the following exception:

2009-12-29 00:17:27,663 INFO org.apache.hadoop.hbase.regionserver.HLog: Roll /hbase/amsterdam_factory/.logs/factory05.lab.mtl,60020,1262042255724/hlog.dat.1262060247637, entries=1830, calcsize=22946017, filesize=22758899. New hlog /hbase/amsterdam_factory/.logs/factory05.lab.mtl,60020,1262042255724/hlog.dat.1262063847659
2009-12-29 00:34:36,210 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer:
java.lang.IllegalArgumentException
at java.nio.Buffer.position(Buffer.java:218)
at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:1114)
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:58)
at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:79)
at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:189)
at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:106)
at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:1776)
at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:1719)
at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1944)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2009-12-29 00:34:36,214 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 0 on 60020, call next(4170645244799815171, 1) from 192.168.1.108:53401: error: java.io.IOException: java.lang.IllegalArgumentException
java.io.IOException: java.lang.IllegalArgumentException
at org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:869)
at org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:859)
at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1965)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
Caused by: java.lang.IllegalArgumentException
at java.nio.Buffer.position(Buffer.java:218)
at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:1114)
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:58)
at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:79)
at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:189)
at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:106)
at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:1776)
at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:1719)
at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1944)
... 5 more

Looks like this bug was encountered before at https://issues.apache.org/jira/browse/HBASE-1495 and spanned a few JIRAs. It's supposed to be resolved as of 0.20.0, but we're running 0.20.2 and it took down one of our regionservers.

I'm also attaching more of the log.