HBase
  1. HBase
  2. HBASE-1809

NPE thrown in BoundedRangeFileInputStream

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.20.1
    • Component/s: io
    • Labels:
      None
    • Environment:

      All

    • Hadoop Flags:
      Reviewed

      Description

      NPE is thrown in BoundedRangeFileInputStream.read when attempting to synchronize on 'in' (line 97).

      This probably means the BRFIS was created with a null FSDIS.

      1. HFile-returnDecompressor.patch
        2 kB
        Mathias Herberts
      2. 1809.patch
        2 kB
        stack
      3. 1809-v2.patch
        2 kB
        stack

        Activity

        Hide
        stack added a comment -

        I just ran into this on an 0.20.0RC2 hbase:

        2009-09-24 23:14:45,578 DEBUG org.apache.hadoop.hbase.regionserver.Store: Completed compaction of info; new storefile is hdfs://XX.XX.XX.XX:10000/hbase/coral_hbase/.META./1028785192/info/1652111193973935565; store size is 752.9k
        2009-09-24 23:14:45,581 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region .META.,,1 in 0sec
        2009-09-24 23:14:45,607 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: 
        java.lang.NullPointerException
                at org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(BoundedRangeFileInputStream.java:97)
                at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:100)
                at org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:979)
                at org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:936)
                at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.loadBlock(HFile.java:1258)
                at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.seekTo(HFile.java:1141)
                at org.apache.hadoop.hbase.regionserver.StoreFileGetScan.getStoreFile(StoreFileGetScan.java:80)
                at org.apache.hadoop.hbase.regionserver.StoreFileGetScan.get(StoreFileGetScan.java:65)
                at org.apache.hadoop.hbase.regionserver.Store.get(Store.java:1442)
                at org.apache.hadoop.hbase.regionserver.HRegion.getClosestRowBefore(HRegion.java:1043)
                at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1755)
                at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
                at java.lang.reflect.Method.invoke(Unknown Source)
                at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
                at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
        2009-09-24 23:14:45,607 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: 
        java.io.IOException: Stream closed
                at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1705)
                at java.io.DataInputStream.read(Unknown Source)
                at org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(BoundedRangeFileInputStream.java:99)
                at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:100)
                at org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:979)
                at org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:936)
                at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.loadBlock(HFile.java:1258)
                at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.seekTo(HFile.java:1141)
                at org.apache.hadoop.hbase.regionserver.StoreFileGetScan.getStoreFile(StoreFileGetScan.java:80)
                at org.apache.hadoop.hbase.regionserver.StoreFileGetScan.get(StoreFileGetScan.java:65)
                at org.apache.hadoop.hbase.regionserver.Store.get(Store.java:1442)
                at org.apache.hadoop.hbase.regionserver.HRegion.getClosestRowBefore(HRegion.java:1043)
                at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1755)
                at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
                at java.lang.reflect.Method.invoke(Unknown Source)
                at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
                at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
        

        So a compaction completion and then just after – about 30ms – a read comes in and gets NPE I guess because its going against a closed storefile.

        Show
        stack added a comment - I just ran into this on an 0.20.0RC2 hbase: 2009-09-24 23:14:45,578 DEBUG org.apache.hadoop.hbase.regionserver.Store: Completed compaction of info; new storefile is hdfs: //XX.XX.XX.XX:10000/hbase/coral_hbase/.META./1028785192/info/1652111193973935565; store size is 752.9k 2009-09-24 23:14:45,581 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region .META.,,1 in 0sec 2009-09-24 23:14:45,607 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: java.lang.NullPointerException at org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(BoundedRangeFileInputStream.java:97) at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:100) at org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:979) at org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:936) at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.loadBlock(HFile.java:1258) at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.seekTo(HFile.java:1141) at org.apache.hadoop.hbase.regionserver.StoreFileGetScan.getStoreFile(StoreFileGetScan.java:80) at org.apache.hadoop.hbase.regionserver.StoreFileGetScan.get(StoreFileGetScan.java:65) at org.apache.hadoop.hbase.regionserver.Store.get(Store.java:1442) at org.apache.hadoop.hbase.regionserver.HRegion.getClosestRowBefore(HRegion.java:1043) at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1755) at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915) 2009-09-24 23:14:45,607 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: java.io.IOException: Stream closed at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1705) at java.io.DataInputStream.read(Unknown Source) at org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(BoundedRangeFileInputStream.java:99) at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:100) at org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:979) at org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:936) at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.loadBlock(HFile.java:1258) at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.seekTo(HFile.java:1141) at org.apache.hadoop.hbase.regionserver.StoreFileGetScan.getStoreFile(StoreFileGetScan.java:80) at org.apache.hadoop.hbase.regionserver.StoreFileGetScan.get(StoreFileGetScan.java:65) at org.apache.hadoop.hbase.regionserver.Store.get(Store.java:1442) at org.apache.hadoop.hbase.regionserver.HRegion.getClosestRowBefore(HRegion.java:1043) at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1755) at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915) So a compaction completion and then just after – about 30ms – a read comes in and gets NPE I guess because its going against a closed storefile.
        Hide
        stack added a comment -

        Committed branch and trunk

        Show
        stack added a comment - Committed branch and trunk
        Hide
        stack added a comment -

        OK. The issue is that everything takes out a read lock in store EXCEPT get. The first stack trace was misleading in that it was in getclosest. When I looked at getclosest, it was taking out the read lock so I couldn't figure it but up in HRegion#getclosest, at its tail, it makes a call to get with the row returned by getclosest. The second stack trace is bare about what the issue is in that its just a pure get.

        The attached patch adds getting of a read lock to get. It lets go of the messing I made in the first version of the patch where I added a flag to BoundedRangeFilterInputStream. It was not fixing the root cause.

        Show
        stack added a comment - OK. The issue is that everything takes out a read lock in store EXCEPT get. The first stack trace was misleading in that it was in getclosest. When I looked at getclosest, it was taking out the read lock so I couldn't figure it but up in HRegion#getclosest, at its tail, it makes a call to get with the row returned by getclosest. The second stack trace is bare about what the issue is in that its just a pure get. The attached patch adds getting of a read lock to get. It lets go of the messing I made in the first version of the patch where I added a flag to BoundedRangeFilterInputStream. It was not fixing the root cause.
        Hide
        ryan rawson added a comment -

        +1, surprised how long it took to find this one.

        Show
        ryan rawson added a comment - +1, surprised how long it took to find this one.
        Hide
        stack added a comment -

        Mathias later ran with the above patch applied. The above patch makes the stream final in BoundedRangeFileInputStream which would seem to indicate that it was passed a null stream... that the stream had already been closed by Reader. Need to back up into Reader and try to figure why a read came in after it had been closed.

        Show
        stack added a comment - Mathias later ran with the above patch applied. The above patch makes the stream final in BoundedRangeFileInputStream which would seem to indicate that it was passed a null stream... that the stream had already been closed by Reader. Need to back up into Reader and try to figure why a read came in after it had been closed.
        Hide
        stack added a comment -

        This patch would address the immediate NPE (What I do in the constructor might not be the smartest) but something else is going on if we have one thread reading off a closed stream. Chatting w/ Mathias, he sees this NPE with get and getclosest. Both take out read locks on the store so another thread shouldn't be doing close this store file. Mathias is trying to figure more input.

        Show
        stack added a comment - This patch would address the immediate NPE (What I do in the constructor might not be the smartest) but something else is going on if we have one thread reading off a closed stream. Chatting w/ Mathias, he sees this NPE with get and getclosest. Both take out read locks on the store so another thread shouldn't be doing close this store file. Mathias is trying to figure more input.
        Hide
        Mathias Herberts added a comment -

        I hit this NPE again, this time in a call to 'get', as the log message contains a row key, I will try to hunt the region to see if there was any particular activity around that time.

        2009-09-04 02:31:52,192 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: 
        java.lang.NullPointerException
                at org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(BoundedRangeFileInputStream.java:97)
                at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:100)
                at org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:982)
                at org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:936)
                at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.loadBlock(HFile.java:1265)
                at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.seekTo(HFile.java:1148)
                at org.apache.hadoop.hbase.regionserver.StoreFileGetScan.getStoreFile(StoreFileGetScan.java:80)
                at org.apache.hadoop.hbase.regionserver.StoreFileGetScan.get(StoreFileGetScan.java:65)
                at org.apache.hadoop.hbase.regionserver.Store.get(Store.java:1442)
                at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2301)
                at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2290)
                at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1768)
                at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                at java.lang.reflect.Method.invoke(Method.java:597)
                at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
                at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
        2009-09-04 02:31:52,195 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 4 on 60020, call get([B@3e507609, row=domirama,0TE332647699C��^?���,1252024306264, maxVersions=1, timeRange=[0,9223372036854775807), families={(family=info, columns=ALL}) from 10.154.99.180:57418: error: java.i
        o.IOException: java.lang.NullPointerException
        java.io.IOException: java.lang.NullPointerException
                at org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:847)
                at org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:837)
                at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1770)
                at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                at java.lang.reflect.Method.invoke(Method.java:597)
                at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
                at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
        Caused by: java.lang.NullPointerException
                at org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(BoundedRangeFileInputStream.java:97)
                at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:100)
                at org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:982)
                at org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:936)
                at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.loadBlock(HFile.java:1265)
                at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.seekTo(HFile.java:1148)
                at org.apache.hadoop.hbase.regionserver.StoreFileGetScan.getStoreFile(StoreFileGetScan.java:80)
                at org.apache.hadoop.hbase.regionserver.StoreFileGetScan.get(StoreFileGetScan.java:65)
                at org.apache.hadoop.hbase.regionserver.Store.get(Store.java:1442)
                at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2301)
                at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2290)
                at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1768)
                ... 5 more
        
        Show
        Mathias Herberts added a comment - I hit this NPE again, this time in a call to 'get', as the log message contains a row key, I will try to hunt the region to see if there was any particular activity around that time. 2009-09-04 02:31:52,192 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: java.lang.NullPointerException at org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(BoundedRangeFileInputStream.java:97) at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:100) at org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:982) at org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:936) at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.loadBlock(HFile.java:1265) at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.seekTo(HFile.java:1148) at org.apache.hadoop.hbase.regionserver.StoreFileGetScan.getStoreFile(StoreFileGetScan.java:80) at org.apache.hadoop.hbase.regionserver.StoreFileGetScan.get(StoreFileGetScan.java:65) at org.apache.hadoop.hbase.regionserver.Store.get(Store.java:1442) at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2301) at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2290) at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1768) at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915) 2009-09-04 02:31:52,195 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 4 on 60020, call get([B@3e507609, row=domirama,0TE332647699C��^?���,1252024306264, maxVersions=1, timeRange=[0,9223372036854775807), families={(family=info, columns=ALL}) from 10.154.99.180:57418: error: java.i o.IOException: java.lang.NullPointerException java.io.IOException: java.lang.NullPointerException at org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:847) at org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:837) at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1770) at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915) Caused by: java.lang.NullPointerException at org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(BoundedRangeFileInputStream.java:97) at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:100) at org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:982) at org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:936) at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.loadBlock(HFile.java:1265) at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.seekTo(HFile.java:1148) at org.apache.hadoop.hbase.regionserver.StoreFileGetScan.getStoreFile(StoreFileGetScan.java:80) at org.apache.hadoop.hbase.regionserver.StoreFileGetScan.get(StoreFileGetScan.java:65) at org.apache.hadoop.hbase.regionserver.Store.get(Store.java:1442) at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2301) at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2290) at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1768) ... 5 more
        Hide
        stack added a comment -

        I applied the above try/catch addition from Mathias to branch and trunk.

        Show
        stack added a comment - I applied the above try/catch addition from Mathias to branch and trunk.
        Hide
        stack added a comment -

        My guess is the file was closed by one thread while another was trying to read. This is a bit of a race condition. Simple fix would be to add a volatile 'closed' flag.. returning immediately from read and available when set (or just check for a null 'in'). Better would be to figure how this race condition arose (Something peculiar to getClosestRowBefore).

        Show
        stack added a comment - My guess is the file was closed by one thread while another was trying to read. This is a bit of a race condition. Simple fix would be to add a volatile 'closed' flag.. returning immediately from read and available when set (or just check for a null 'in'). Better would be to figure how this race condition arose (Something peculiar to getClosestRowBefore).
        Hide
        Mathias Herberts added a comment -

        While digging in the code for this issue I came across a problem in HFile$Reader.decompress.

        In case an exception is thrown, the decompressor is not returned to the pool.

        The attached patch fixes this by enclosing the content in a try ... finally

        Show
        Mathias Herberts added a comment - While digging in the code for this issue I came across a problem in HFile$Reader.decompress. In case an exception is thrown, the decompressor is not returned to the pool. The attached patch fixes this by enclosing the content in a try ... finally
        Hide
        Mathias Herberts added a comment -

        This error probably caused a row not to be inserted during a bulk upload (with WriteBuffer size set to 0, AutoFlush set to false and setWriteToWAL set to false in the Put instance).

        Show
        Mathias Herberts added a comment - This error probably caused a row not to be inserted during a bulk upload (with WriteBuffer size set to 0, AutoFlush set to false and setWriteToWAL set to false in the Put instance).
        Hide
        Mathias Herberts added a comment -
        2009-09-01 18:05:15,644 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: 
        java.lang.NullPointerException
                at org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(BoundedRangeFileInputStream.java:97)
                at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:100)
                at org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:979)
                at org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:936)
                at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.loadBlock(HFile.java:1258)
                at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.seekTo(HFile.java:1141)
                at org.apache.hadoop.hbase.regionserver.StoreFileGetScan.getStoreFile(StoreFileGetScan.java:80)
                at org.apache.hadoop.hbase.regionserver.StoreFileGetScan.get(StoreFileGetScan.java:65)
                at org.apache.hadoop.hbase.regionserver.Store.get(Store.java:1442)
                at org.apache.hadoop.hbase.regionserver.HRegion.getClosestRowBefore(HRegion.java:1043)
                at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1755)
                at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                at java.lang.reflect.Method.invoke(Method.java:597)
                at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
                at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
        2009-09-01 18:05:15,665 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 8 on 60020, call getClosestRowBefore([B@68f2f88, [B@22798d15, [B@5a1d6f61) from 10.154.99.180:37118: error: java.io.IOException: java.lang.NullPointerException
        java.io.IOException: java.lang.NullPointerException
                at org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:847)
                at org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:837)
                at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1758)
                at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                at java.lang.reflect.Method.invoke(Method.java:597)
                at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
                at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
        Caused by: java.lang.NullPointerException
                at org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(BoundedRangeFileInputStream.java:97)
                at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:100)
                at org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:979)
                at org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:936)
                at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.loadBlock(HFile.java:1258)
                at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.seekTo(HFile.java:1141)
                at org.apache.hadoop.hbase.regionserver.StoreFileGetScan.getStoreFile(StoreFileGetScan.java:80)
                at org.apache.hadoop.hbase.regionserver.StoreFileGetScan.get(StoreFileGetScan.java:65)
                at org.apache.hadoop.hbase.regionserver.Store.get(Store.java:1442)
                at org.apache.hadoop.hbase.regionserver.HRegion.getClosestRowBefore(HRegion.java:1043)
                at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1755)
                ... 5 more
        
        Show
        Mathias Herberts added a comment - 2009-09-01 18:05:15,644 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: java.lang.NullPointerException at org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(BoundedRangeFileInputStream.java:97) at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:100) at org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:979) at org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:936) at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.loadBlock(HFile.java:1258) at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.seekTo(HFile.java:1141) at org.apache.hadoop.hbase.regionserver.StoreFileGetScan.getStoreFile(StoreFileGetScan.java:80) at org.apache.hadoop.hbase.regionserver.StoreFileGetScan.get(StoreFileGetScan.java:65) at org.apache.hadoop.hbase.regionserver.Store.get(Store.java:1442) at org.apache.hadoop.hbase.regionserver.HRegion.getClosestRowBefore(HRegion.java:1043) at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1755) at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915) 2009-09-01 18:05:15,665 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 8 on 60020, call getClosestRowBefore([B@68f2f88, [B@22798d15, [B@5a1d6f61) from 10.154.99.180:37118: error: java.io.IOException: java.lang.NullPointerException java.io.IOException: java.lang.NullPointerException at org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:847) at org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:837) at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1758) at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915) Caused by: java.lang.NullPointerException at org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(BoundedRangeFileInputStream.java:97) at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:100) at org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:979) at org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:936) at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.loadBlock(HFile.java:1258) at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.seekTo(HFile.java:1141) at org.apache.hadoop.hbase.regionserver.StoreFileGetScan.getStoreFile(StoreFileGetScan.java:80) at org.apache.hadoop.hbase.regionserver.StoreFileGetScan.get(StoreFileGetScan.java:65) at org.apache.hadoop.hbase.regionserver.Store.get(Store.java:1442) at org.apache.hadoop.hbase.regionserver.HRegion.getClosestRowBefore(HRegion.java:1043) at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1755) ... 5 more

          People

          • Assignee:
            stack
            Reporter:
            Mathias Herberts
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development