[HBASE-1250] hdfs.DFSClient: DFS Read: java.io.IOException: Could not obtain block: After restarting Datanode in HDFS

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 0.19.0
• Fix Version/s: 0.90.0
• Component/s: regionserver

Environment

Cluster is runing on one Laptop.
HDFS (1 Namenode + 1 Datanode)
HBase (1 master + 1 HRS)

• Assignee: Evgeny Ryabitskiy
• Reporter: Evgeny Ryabitskiy
• Created: 10/Mar/09 01:31
• Updated: 20/Nov/15 13:02
• Resolved: 21/Aug/09 20:47

Description

I am stoping Datanode.
And get a lot of exception. Then I am starting Datanode back.
Master is working again but Region Server returns exceptions:

INFO ipc.HBaseServer: IPC Server handler 9 on 60020, call openScanner([B@e3404f, [[B@1930611, [B@1735f84, 9223372036854775807, null) from 127.0.0.1:55743: error: java.io.IOException: HStoreScanner failed construction
java.io.IOException: HStoreScanner failed construction
	at org.apache.hadoop.hbase.regionserver.StoreFileScanner.<init>(StoreFileScanner.java:73)
	at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:87)
	at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1769)
	at org.apache.hadoop.hbase.regionserver.HRegion$HScanner.<init>(HRegion.java:1971)
	at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1161)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1720)
	at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
	at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912)
Caused by: java.io.IOException: Could not obtain block: blk_-5468922917120193795_1013 file=/hbase/-ROOT-/70236052/info/2088312954622080873
	at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1708)
	at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1536)
	at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1663)
	at java.io.DataInputStream.read(DataInputStream.java:132)
	at org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(BoundedRangeFileInputStream.java:99)
	at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:100)
	at org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:910)
	at org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:863)
	at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.seekTo(HFile.java:1151)
	at org.apache.hadoop.hbase.regionserver.StoreFileScanner.getNext(StoreFileScanner.java:301)
	at org.apache.hadoop.hbase.regionserver.StoreFileScanner.openScanner(StoreFileScanner.java:102)
	at org.apache.hadoop.hbase.regionserver.StoreFileScanner.<init>(StoreFileScanner.java:70)
	... 10 more

this one is returned in a loop and there is second one:

WARN hdfs.DFSClient: DFS Read: java.io.IOException: Could not obtain block: blk_-5468922917120193795_1013 file=/hbase/-ROOT-/70236052/info/2088312954622080873
	at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1708)
	at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1536)
	at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1663)
	at java.io.DataInputStream.read(DataInputStream.java:132)
	at org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(BoundedRangeFileInputStream.java:99)
	at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:100)
	at org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:910)
	at org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:863)
	at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.seekTo(HFile.java:1151)
	at org.apache.hadoop.hbase.regionserver.StoreFileScanner.getNext(StoreFileScanner.java:301)
	at org.apache.hadoop.hbase.regionserver.StoreFileScanner.openScanner(StoreFileScanner.java:102)
	at org.apache.hadoop.hbase.regionserver.StoreFileScanner.<init>(StoreFileScanner.java:70)
	at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:87)
	at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1769)
	at org.apache.hadoop.hbase.regionserver.HRegion$HScanner.<init>(HRegion.java:1971)
	at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1161)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1720)
	at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
	at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912)

Issue Links

• is related to: HBASE-1078 "java.io.IOException: Could not obtain block": allthough file is there and accessible through the dfs client