Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-12575

Improve test coverage for EC related edit logs ops

    XMLWordPrintableJSON

Details

    Description

      HDFS-12569 found that we have little test coverage for edit logs ops of erasure coding.

      And we've seen the following bug bring down SNN in our test environments:

      6:42:18.177 AM	ERROR	FSEditLogLoader	
      Encountered exception on operation AddBlockOp [path=/tmp/foo/bar, penultimateBlock=NULL, lastBlock=blk_1073743386_69322, RpcClientId=, RpcCallId=-2]
      java.lang.IllegalArgumentException: reportedBlock is not striped
      	at com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
      	at 
      ....
      6:42:18.190 AM	FATAL	EditLogTailer	
      Unknown error encountered while tailing edits. Shutting down standby NN.
      java.io.IOException: java.lang.IllegalArgumentException: reportedBlock is not striped
      	at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:251)
      	at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:150)
      	at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:882)
      	at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:863)
      	at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.doTailEdits(EditLogTailer.java:293)
      	at 
      

      We should add coverage for these important edit logs, i.e., set/unset policy, enable/remove policies and etc are correctly persisted in edit logs, and test the scenarios like:

      • Restart NN
      • Replay edits after checkpoint
      • Apply edits on SNN.
      • and etc.

      Attachments

        Issue Links

          Activity

            People

              Sammi Sammi Chen
              eddyxu Lei (Eddy) Xu
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated: