[HBASE-24469] Hedged read might hang infinitely if read data from all DN failed

• Type: Bug
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Assignee: Unassigned
• Reporter: Javier Akira Luca de Tena
• Created: 29/May/20 08:35
• Updated: 29/May/20 08:48

Description

Found out that after an ungraceful Datanode shutdown, the number of HBase active handlers started to grow, making RegionServer stuck and not able to serve any RPC.

Took the thread dump and found out multiple read handlers were in some kind of dead lock state and also write handlers stuck.

This also caused to not be able to flush the memstore because it was waiting for this lock: https://github.com/apache/hbase/blob/136414dd72a80f379b80cd6f74b5b6ebd78f33ec/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java#L1225

Without being able to flush it, I could not gracefully stop the RegionServer, since we can't move out the flushing region.

 

Found out that the real issue was in Hadoop's DFSInputStream. When no hedged reads succeed, the internal hedgedService.take() call hangs forever since it's internally using a BlockingQueue: https://github.com/apache/hadoop/blob/rel/release-2.8.5/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java#L1435

 

Reproduced in HBase 1.4.13, but I think it affects to other versions: 

1. Datanode dies
2. A read handler holding a read lock of an HStore is blocked by hedged read that does not succeed.
3. Other read handlers try to acquire the lock and stuck.
4. Memstore flusher tries to acquire write lock in HStore and also blocked because of the other read locks.
5. Others like CompactedHFilesDischarger also blocks because memstore holds the lock.
6. Tried to use graceful_stop.sh, but region_mover.rb fails because can't move out the region being flushed.
7. Forcefully killed the RegionServer because no other option (I am not sure if there is possibility of data loss, since HStore#updateStorefiles is not finished at this point).

 

This is the Hadoop side issue: https://issues.apache.org/jira/browse/HDFS-11303 and it's fixed for 2.9.0.

This is not directly related with HBase code, but just wanted community to be aware that with current used Hadoop version (2.8.5), this issue could happen.

 

I would like to suggest to upgrade the used Hadoop version to 2.9.0.

Issue Links

• is fixed by: HDFS-11303 Hedged read might hang infinitely if read data from all DN failed