[HDFS-9962] Erasure Coding: need a way to test multiple EC policies

• Type: Test
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Labels: hdfs-ec-3.0-nice-to-have
• Assignee: Takanobu Asanuma
• Reporter: Rui Li
• Created: 15/Mar/16 06:12
• Updated: 16/Oct/17 03:02

Description

Now that we support multiple EC policies, we need a way test it to catch potential issues.

UPDATE:
Since many existing EC unit tests need some changes to use multiple EC policies, it would be good to create a jira for every few tests. Based on some discussion in HDFS-7866 and HDFS-9962, this task follows the below idea.

• Default policy (RS-6-3) always gets tested.
• Create a random ec policy test if the test is long-running (e.g. using minicluster many times).
• If the test is not long, all policies test with parametrizing would be better.

If you are interested in this work, please join it and create a jira under this jira.

Issue Links

• incorporates: HDFS-11823 Extend TestDFSStripedIutputStream/TestDFSStripedOutputStream with a random EC policy
• is related to: HDFS-7866 Erasure coding: NameNode manages multiple erasure coding policies
• is related to: HDFS-11964 Decoding inputs should be correctly prepared in pread

Sub-Tasks

• Extend TestDFSStripedIutputStream/TestDFSStripedOutputStream with a random EC policy: Resolved
• Extend TestErasureCodingPolicies/TestErasureCodingPolicyWithSnapshot with a random EC policy: Resolved
• Extend TestFileStatusWithECPolicy with a random EC policy: Resolved
• Extend TestSafeModeWithStripedFile with a random EC policy: Resolved
• Extend TestQuotaWithStripedBlocks with a random EC policy: Resolved
• Use Parameterized tests in TestBlockInfoStriped and TestLowRedundancyBlockQueues to test all EC policies: Resolved
• Extend TestDistributedFileSystemWithECFile with a random EC policy: Resolved
• Extend TestReconstructStripedFile with a random EC policy: Resolved