[HDFS-15348] [SBN Read] IllegalStateException happened when doing failover

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Assignee: Unassigned
• Reporter: ZanderXu
• Created: 10/May/20 13:32
• Updated: 12/May/20 16:13
• Resolved: 12/May/20 16:13

Description

Standby shutdown when doing failover, and throw IllegalStateException.

getJournaledEdits only return dfs.ha.tail-edits.qjm.rpc.max-txns edits, resulting in failure to replay all edits in catchupDuringFailover.

 

And check streams.isEmpty() will be throw this exception in FSEditLog#openForWrite

The exception like:

 

2020-05-10 09:20:02,235 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode IPC Server handler 763 on 8022: Error encountered requiring NN sh
utdown. Shutting down immediately.
java.lang.IllegalStateException: Cannot start writing at txid 173922195318 when there is a stream available for read: org.apache.hadoop.hdfs.se
rver.namenode.RedundantEditLogInputStream@47b73995
        at org.apache.hadoop.hdfs.server.namenode.FSEditLog.openForWrite(FSEditLog.java:320)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startActiveServices(FSNamesystem.java:1352)
        at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.startActiveServices(NameNode.java:1890)
        at org.apache.hadoop.hdfs.server.namenode.ha.ActiveState.enterState(ActiveState.java:61)
        at org.apache.hadoop.hdfs.server.namenode.ha.HAState.setStateInternal(HAState.java:64)
        at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.setState(StandbyState.java:49)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.transitionToActive(NameNode.java:1763)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.transitionToActive(NameNodeRpcServer.java:1605)

 

Issue Links

• is duplicated by: HDFS-15323 StandbyNode fails transition to active due to insufficient transaction tailing