[HDFS-12575] Improve test coverage for EC related edit logs ops

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.0.0-alpha4
• Component/s: erasure-coding
• Labels: hdfs-ec-3.0-nice-to-have
• Target Version/s: 3.5.0
• Assignee: Sammi Chen
• Reporter: Lei (Eddy) Xu
• Created: 02/Oct/17 18:26
• Updated: 04/Jan/24 07:58

Description

HDFS-12569 found that we have little test coverage for edit logs ops of erasure coding.

And we've seen the following bug bring down SNN in our test environments:

6:42:18.177 AM	ERROR	FSEditLogLoader	
Encountered exception on operation AddBlockOp [path=/tmp/foo/bar, penultimateBlock=NULL, lastBlock=blk_1073743386_69322, RpcClientId=, RpcCallId=-2]
java.lang.IllegalArgumentException: reportedBlock is not striped
	at com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
	at 
....
6:42:18.190 AM	FATAL	EditLogTailer	
Unknown error encountered while tailing edits. Shutting down standby NN.
java.io.IOException: java.lang.IllegalArgumentException: reportedBlock is not striped
	at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:251)
	at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:150)
	at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:882)
	at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:863)
	at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.doTailEdits(EditLogTailer.java:293)
	at 

We should add coverage for these important edit logs, i.e., set/unset policy, enable/remove policies and etc are correctly persisted in edit logs, and test the scenarios like:

• Restart NN
• Replay edits after checkpoint
• Apply edits on SNN.
• and etc.

Issue Links

• relates to: HDFS-12569 Unset EC policy logs empty payload in edit log