Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11505

Do not enable any erasure coding policies by default

    Details

    • Target Version/s:
    • Hadoop Flags:
      Incompatible change
    • Release Note:
      By default, none of the built-in erasure coding policies are enabled. Users have to explicitly enable the erasure coding policy via the hdfs configuration 'dfs.namenode.ec.policies.enabled' before setting the policy on any directories.

      Description

      As discussed on HDFS-11314, administrators need to choose the correct set of EC policies based on cluster size and desired fault-tolerance properties.

      This means we should not enable any EC policies by default, since any default value could be incorrect.

      1. HDFS-11505.01.patch
        8 kB
        Manoj Govindassamy
      2. HDFS-11505.02.patch
        9 kB
        Manoj Govindassamy
      3. HDFS-11505.03.patch
        10 kB
        Manoj Govindassamy
      4. HDFS-11505.04.patch
        42 kB
        Manoj Govindassamy

        Issue Links

          Activity

          Hide
          drankye Kai Zheng added a comment -

          So assume some policies are enabled by the configuration, will users have to specify some one when create a file/folder or assign/set policy to it later? I'm asking this because, I'm wondering, for most applications/users they might not want to bother to query/check the enabled policies first to pick up some to specify. If users don't have to or the policy parameter is optional, I guess we could use the first one in the enabled list. Thoughts?

          Show
          drankye Kai Zheng added a comment - So assume some policies are enabled by the configuration, will users have to specify some one when create a file/folder or assign/set policy to it later? I'm asking this because, I'm wondering, for most applications/users they might not want to bother to query/check the enabled policies first to pick up some to specify. If users don't have to or the policy parameter is optional, I guess we could use the first one in the enabled list. Thoughts?
          Hide
          andrew.wang Andrew Wang added a comment -

          Hi Kai,

          Your understanding is correct. However, I think the vast majority of EC writes will happen by inheriting the policy from a parent directory at create time, so usage will be transparent. It's likely that only command-line users will be calling the setPolicy API, so they'll (hopefully) get nice error messages about what's an invalid or disabled policy and how to enable policies. As a reference point, I don't think we've seen a similar need for the storage policy APIs. Apps like HBase or Hive have often depended on admins setting up directory-level policies (e.g. storage policies, encryption).

          However, if you think having a default policy is important, we could add a client-side config similar to dfs.replication to specify a default policy. Alternatively, this could also be set in NN configuration and queried via getServerDefaults. As per above though, I'd hope that dir-level policies are the best fit for users who want a simple way of using EC.

          Show
          andrew.wang Andrew Wang added a comment - Hi Kai, Your understanding is correct. However, I think the vast majority of EC writes will happen by inheriting the policy from a parent directory at create time, so usage will be transparent. It's likely that only command-line users will be calling the setPolicy API, so they'll (hopefully) get nice error messages about what's an invalid or disabled policy and how to enable policies. As a reference point, I don't think we've seen a similar need for the storage policy APIs. Apps like HBase or Hive have often depended on admins setting up directory-level policies (e.g. storage policies, encryption). However, if you think having a default policy is important, we could add a client-side config similar to dfs.replication to specify a default policy. Alternatively, this could also be set in NN configuration and queried via getServerDefaults. As per above though, I'd hope that dir-level policies are the best fit for users who want a simple way of using EC.
          Hide
          drankye Kai Zheng added a comment -

          Thanks Andrew for your thought and it makes sense. The client-side config similar to dfs.replication to specify a default policy is a great idea and I guess we can do it later given some real user requirement that incurs much need of setting policy per file.

          Show
          drankye Kai Zheng added a comment - Thanks Andrew for your thought and it makes sense. The client-side config similar to dfs.replication to specify a default policy is a great idea and I guess we can do it later given some real user requirement that incurs much need of setting policy per file.
          Hide
          tasanuma0829 Takanobu Asanuma added a comment -

          Hi all, thanks for the discussion.

          If we don't enable any EC policies by default, I feel a little anxious that some admins might set low-redundant EC policies to high-level directories for large clusters. IIUC, RS(6,3) or RS(10,4) is a recommended policy for large clusters. How about adding the notes to the EC document?

          Show
          tasanuma0829 Takanobu Asanuma added a comment - Hi all, thanks for the discussion. If we don't enable any EC policies by default, I feel a little anxious that some admins might set low-redundant EC policies to high-level directories for large clusters. IIUC, RS(6,3) or RS(10,4) is a recommended policy for large clusters. How about adding the notes to the EC document?
          Hide
          andrew.wang Andrew Wang added a comment -

          Hi Takanobu Asanuma, I made some improvements to the documentation in the HDFS-11314 patch (which isn't committed yet), this may satisfy some of your concerns.

          I think it's still be worthwhile to take a fresh look at the documentation later on in the dev cycle when things are feature complete; filed HDFS-11510 for this.

          Show
          andrew.wang Andrew Wang added a comment - Hi Takanobu Asanuma , I made some improvements to the documentation in the HDFS-11314 patch (which isn't committed yet), this may satisfy some of your concerns. I think it's still be worthwhile to take a fresh look at the documentation later on in the dev cycle when things are feature complete; filed HDFS-11510 for this.
          Hide
          manojg Manoj Govindassamy added a comment -

          Andrew Wang,
          Attached a patch which goes on top of HDFS-11314 to remove the default EC Policy. Also updated doc and tests accordingly. Please take a look and let me know if you have more things in mind that needs to be taken care for this. Will update the status of this bug to PATCH AVAILABLE once HDFS-11314 is committed as the patch is dependent on it.

          Show
          manojg Manoj Govindassamy added a comment - Andrew Wang , Attached a patch which goes on top of HDFS-11314 to remove the default EC Policy. Also updated doc and tests accordingly. Please take a look and let me know if you have more things in mind that needs to be taken care for this. Will update the status of this bug to PATCH AVAILABLE once HDFS-11314 is committed as the patch is dependent on it.
          Hide
          andrew.wang Andrew Wang added a comment -

          Looks good overall, thanks for the patch Manoj! Bit surprised that there weren't more tests that needed to be updated, but maybe I caught them all already in HDFS-11314.

          I see one little doc typo: "bulit-in". There's also still a reference to the "default EC policy" in the docs.

          I think we can wait to fix these and mark as PA until after HDFS-11314 goes in to reduce churn, hopefully soon.

          Show
          andrew.wang Andrew Wang added a comment - Looks good overall, thanks for the patch Manoj! Bit surprised that there weren't more tests that needed to be updated, but maybe I caught them all already in HDFS-11314 . I see one little doc typo: "bulit-in". There's also still a reference to the "default EC policy" in the docs. I think we can wait to fix these and mark as PA until after HDFS-11314 goes in to reduce churn, hopefully soon.
          Hide
          manojg Manoj Govindassamy added a comment - - edited

          There were a lot of test failures as ErasureCodingPolicyManager was expecting a valid policy to passed in during construction. After the following change, I avoided the crash during the construction and most of the tests started to pass through. Sure, will wait for HDFS-11314 and post a revised patch with doc fixes.

          +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ErasureCodingPolicyManager.java
          @@ -98,12 +98,15 @@
                   DFSConfigKeys.DFS_NAMENODE_EC_POLICIES_ENABLED_DEFAULT);
               this.enabledPoliciesByName = new TreeMap<>();
               for (String policyName : policyNames) {
          +      if (policyName.trim().isEmpty()) {
          +        continue;
          +      }
          
          Show
          manojg Manoj Govindassamy added a comment - - edited There were a lot of test failures as ErasureCodingPolicyManager was expecting a valid policy to passed in during construction. After the following change, I avoided the crash during the construction and most of the tests started to pass through. Sure, will wait for HDFS-11314 and post a revised patch with doc fixes. +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ErasureCodingPolicyManager.java @@ -98,12 +98,15 @@ DFSConfigKeys.DFS_NAMENODE_EC_POLICIES_ENABLED_DEFAULT); this .enabledPoliciesByName = new TreeMap<>(); for ( String policyName : policyNames) { + if (policyName.trim().isEmpty()) { + continue ; + }
          Hide
          manojg Manoj Govindassamy added a comment -

          Attached v02 patch to fix the typo and default policy reference in doc.
          Andrew Wang, Takanobu Asanuma, please take a look at the patch.

          Show
          manojg Manoj Govindassamy added a comment - Attached v02 patch to fix the typo and default policy reference in doc. Andrew Wang , Takanobu Asanuma , please take a look at the patch.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 11m 11s trunk passed
          +1 compile 0m 42s trunk passed
          +1 checkstyle 0m 39s trunk passed
          +1 mvnsite 0m 44s trunk passed
          +1 mvneclipse 0m 11s trunk passed
          +1 findbugs 1m 40s trunk passed
          +1 javadoc 0m 35s trunk passed
          +1 mvninstall 0m 46s the patch passed
          +1 compile 0m 43s the patch passed
          +1 javac 0m 43s the patch passed
          +1 checkstyle 0m 33s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 403 unchanged - 1 fixed = 403 total (was 404)
          +1 mvnsite 0m 49s the patch passed
          +1 mvneclipse 0m 9s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 1m 46s the patch passed
          +1 javadoc 0m 35s the patch passed
          -1 unit 67m 3s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 19s The patch does not generate ASF License warnings.
          89m 57s



          Reason Tests
          Failed junit tests hadoop.hdfs.TestErasureCodingPolicyWithSnapshot
            hadoop.hdfs.server.namenode.TestAddStripedBlockInFBR
            hadoop.hdfs.TestDecommissionWithStriped
            hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped
            hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness
            hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics
            hadoop.hdfs.server.namenode.TestFsck
            hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations
            hadoop.hdfs.server.namenode.TestReconstructStripedBlocks
            hadoop.hdfs.server.blockmanagement.TestSequentialBlockGroupId
            hadoop.hdfs.server.namenode.ha.TestEditLogTailer
            hadoop.hdfs.server.namenode.TestFSEditLogLoader
            hadoop.hdfs.TestLeaseRecoveryStriped
            hadoop.hdfs.TestReadStripedFileWithMissingBlocks
            hadoop.hdfs.TestReconstructStripedFile
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
            hadoop.hdfs.server.datanode.TestDataNodeUUID
            hadoop.hdfs.server.namenode.TestNameNodeMXBean
            hadoop.hdfs.TestFileStatusWithECPolicy
            hadoop.hdfs.server.mover.TestMover
            hadoop.hdfs.server.namenode.TestStripedINodeFile
            hadoop.hdfs.server.namenode.TestQuotaWithStripedBlocks
            hadoop.hdfs.server.balancer.TestBalancer
            hadoop.hdfs.server.datanode.checker.TestThrottledAsyncChecker
            hadoop.hdfs.TestReadStripedFileWithDecoding
            hadoop.hdfs.web.TestWebHdfsTimeouts
            hadoop.hdfs.TestSafeModeWithStripedFile
            hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks
            hadoop.hdfs.server.datanode.TestDataNodeMXBean
            hadoop.hdfs.TestWriteReadStripedFile
            hadoop.hdfs.server.namenode.TestAddStripedBlocks
            hadoop.hdfs.TestErasureCodeBenchmarkThroughput
            hadoop.hdfs.TestFileChecksum
            hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerWithStripedBlocks



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-11505
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12857070/HDFS-11505.02.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 77224c247932 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 822a74f
          Default Java 1.8.0_121
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/18668/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18668/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18668/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 11m 11s trunk passed +1 compile 0m 42s trunk passed +1 checkstyle 0m 39s trunk passed +1 mvnsite 0m 44s trunk passed +1 mvneclipse 0m 11s trunk passed +1 findbugs 1m 40s trunk passed +1 javadoc 0m 35s trunk passed +1 mvninstall 0m 46s the patch passed +1 compile 0m 43s the patch passed +1 javac 0m 43s the patch passed +1 checkstyle 0m 33s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 403 unchanged - 1 fixed = 403 total (was 404) +1 mvnsite 0m 49s the patch passed +1 mvneclipse 0m 9s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 1m 46s the patch passed +1 javadoc 0m 35s the patch passed -1 unit 67m 3s hadoop-hdfs in the patch failed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 89m 57s Reason Tests Failed junit tests hadoop.hdfs.TestErasureCodingPolicyWithSnapshot   hadoop.hdfs.server.namenode.TestAddStripedBlockInFBR   hadoop.hdfs.TestDecommissionWithStriped   hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped   hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness   hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics   hadoop.hdfs.server.namenode.TestFsck   hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations   hadoop.hdfs.server.namenode.TestReconstructStripedBlocks   hadoop.hdfs.server.blockmanagement.TestSequentialBlockGroupId   hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.server.namenode.TestFSEditLogLoader   hadoop.hdfs.TestLeaseRecoveryStriped   hadoop.hdfs.TestReadStripedFileWithMissingBlocks   hadoop.hdfs.TestReconstructStripedFile   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.server.datanode.TestDataNodeUUID   hadoop.hdfs.server.namenode.TestNameNodeMXBean   hadoop.hdfs.TestFileStatusWithECPolicy   hadoop.hdfs.server.mover.TestMover   hadoop.hdfs.server.namenode.TestStripedINodeFile   hadoop.hdfs.server.namenode.TestQuotaWithStripedBlocks   hadoop.hdfs.server.balancer.TestBalancer   hadoop.hdfs.server.datanode.checker.TestThrottledAsyncChecker   hadoop.hdfs.TestReadStripedFileWithDecoding   hadoop.hdfs.web.TestWebHdfsTimeouts   hadoop.hdfs.TestSafeModeWithStripedFile   hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks   hadoop.hdfs.server.datanode.TestDataNodeMXBean   hadoop.hdfs.TestWriteReadStripedFile   hadoop.hdfs.server.namenode.TestAddStripedBlocks   hadoop.hdfs.TestErasureCodeBenchmarkThroughput   hadoop.hdfs.TestFileChecksum   hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerWithStripedBlocks Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11505 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12857070/HDFS-11505.02.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 77224c247932 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 822a74f Default Java 1.8.0_121 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/18668/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18668/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18668/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          manojg Manoj Govindassamy added a comment -

          Above test failures are from the patch. My last patch version passed many of these failed tests. May be missing something in the latest patch. Anyways, will fix these and upload a new patch.

          Show
          manojg Manoj Govindassamy added a comment - Above test failures are from the patch. My last patch version passed many of these failed tests. May be missing something in the latest patch. Anyways, will fix these and upload a new patch.
          Hide
          manojg Manoj Govindassamy added a comment - - edited

          Attaching v03 patch to address the test failures via enabling default EC policy if none are enabled during the construction of MiniCluster. This passes through all the failed tests.

          Show
          manojg Manoj Govindassamy added a comment - - edited Attaching v03 patch to address the test failures via enabling default EC policy if none are enabled during the construction of MiniCluster. This passes through all the failed tests.
          Hide
          andrew.wang Andrew Wang added a comment -

          Hi Manoj, I'd rather we enable the correct policy on a test-by-test basis since enabling this in MiniDFSCluster is a bit opaque (you'd expect it to have the default config).

          Show
          andrew.wang Andrew Wang added a comment - Hi Manoj, I'd rather we enable the correct policy on a test-by-test basis since enabling this in MiniDFSCluster is a bit opaque (you'd expect it to have the default config).
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 12m 23s trunk passed
          +1 compile 0m 46s trunk passed
          +1 checkstyle 0m 42s trunk passed
          +1 mvnsite 0m 50s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 43s trunk passed
          +1 javadoc 0m 40s trunk passed
          +1 mvninstall 0m 46s the patch passed
          +1 compile 0m 44s the patch passed
          +1 javac 0m 44s the patch passed
          +1 checkstyle 0m 40s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 606 unchanged - 1 fixed = 606 total (was 607)
          +1 mvnsite 0m 50s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 1m 47s the patch passed
          +1 javadoc 0m 37s the patch passed
          +1 unit 63m 44s hadoop-hdfs in the patch passed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          88m 28s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-11505
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12857436/HDFS-11505.03.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 80a68f3ff88f 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 229c7c9
          Default Java 1.8.0_121
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18686/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18686/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 12m 23s trunk passed +1 compile 0m 46s trunk passed +1 checkstyle 0m 42s trunk passed +1 mvnsite 0m 50s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 43s trunk passed +1 javadoc 0m 40s trunk passed +1 mvninstall 0m 46s the patch passed +1 compile 0m 44s the patch passed +1 javac 0m 44s the patch passed +1 checkstyle 0m 40s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 606 unchanged - 1 fixed = 606 total (was 607) +1 mvnsite 0m 50s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 1m 47s the patch passed +1 javadoc 0m 37s the patch passed +1 unit 63m 44s hadoop-hdfs in the patch passed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 88m 28s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11505 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12857436/HDFS-11505.03.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 80a68f3ff88f 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 229c7c9 Default Java 1.8.0_121 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18686/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18686/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          tasanuma0829 Takanobu Asanuma added a comment -

          Thanks for the patch, Manoj Govindassamy, and thanks for reviewing, Andrew Wang! I agree with Andrew's review. One minor comment:

          It seems the tests in the patch don't cover this change:

          ErasureCodingPolicyManager.java
          if (policyName.trim().isEmpty()) {
            continue;
          }
          

          We should add the regression test to TestEnabledECPolicies#testValid.

          expectValidPolicy(", " + ecPolicyName, 1);
          
          Show
          tasanuma0829 Takanobu Asanuma added a comment - Thanks for the patch, Manoj Govindassamy , and thanks for reviewing, Andrew Wang ! I agree with Andrew's review. One minor comment: It seems the tests in the patch don't cover this change: ErasureCodingPolicyManager.java if (policyName.trim().isEmpty()) { continue ; } We should add the regression test to TestEnabledECPolicies#testValid . expectValidPolicy( ", " + ecPolicyName, 1);
          Hide
          manojg Manoj Govindassamy added a comment -

          Thanks Andrew Wang and Takanobu Asanuma for the review. Attaching v04 patch to enable EC policy in the needed tests only. Also, extended TestEnabledECPolicies#testValid based on Takanobu's comment. Please tae a look at the patch.

          Show
          manojg Manoj Govindassamy added a comment - Thanks Andrew Wang and Takanobu Asanuma for the review. Attaching v04 patch to enable EC policy in the needed tests only. Also, extended TestEnabledECPolicies#testValid based on Takanobu's comment. Please tae a look at the patch.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 21s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 28 new or modified test files.
          +1 mvninstall 11m 29s trunk passed
          +1 compile 0m 39s trunk passed
          +1 checkstyle 0m 36s trunk passed
          +1 mvnsite 0m 52s trunk passed
          +1 mvneclipse 0m 9s trunk passed
          +1 findbugs 2m 38s trunk passed
          +1 javadoc 1m 25s trunk passed
          +1 mvninstall 1m 36s the patch passed
          +1 compile 0m 39s the patch passed
          +1 javac 0m 39s the patch passed
          -0 checkstyle 0m 39s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 770 unchanged - 1 fixed = 771 total (was 771)
          +1 mvnsite 0m 46s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          +1 findbugs 1m 41s the patch passed
          +1 javadoc 0m 37s the patch passed
          -1 unit 63m 37s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 15s The patch does not generate ASF License warnings.
          90m 2s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
            hadoop.hdfs.web.TestWebHdfsTimeouts
            hadoop.hdfs.web.TestWebHdfsFileSystemContract
            hadoop.hdfs.TestHDFSFileSystemContract



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-11505
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12858572/HDFS-11505.04.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux ef9dee34b376 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 55796a0
          Default Java 1.8.0_121
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18699/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/18699/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18699/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18699/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 28 new or modified test files. +1 mvninstall 11m 29s trunk passed +1 compile 0m 39s trunk passed +1 checkstyle 0m 36s trunk passed +1 mvnsite 0m 52s trunk passed +1 mvneclipse 0m 9s trunk passed +1 findbugs 2m 38s trunk passed +1 javadoc 1m 25s trunk passed +1 mvninstall 1m 36s the patch passed +1 compile 0m 39s the patch passed +1 javac 0m 39s the patch passed -0 checkstyle 0m 39s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 770 unchanged - 1 fixed = 771 total (was 771) +1 mvnsite 0m 46s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 findbugs 1m 41s the patch passed +1 javadoc 0m 37s the patch passed -1 unit 63m 37s hadoop-hdfs in the patch failed. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 90m 2s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.web.TestWebHdfsTimeouts   hadoop.hdfs.web.TestWebHdfsFileSystemContract   hadoop.hdfs.TestHDFSFileSystemContract Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11505 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12858572/HDFS-11505.04.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux ef9dee34b376 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 55796a0 Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18699/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/18699/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18699/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18699/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          manojg Manoj Govindassamy added a comment -

          Above unit test errors are not related to the latest patch.

          Show
          manojg Manoj Govindassamy added a comment - Above unit test errors are not related to the latest patch.
          Hide
          tasanuma0829 Takanobu Asanuma added a comment -

          +1 (non-binding). Thanks!

          Show
          tasanuma0829 Takanobu Asanuma added a comment - +1 (non-binding). Thanks!
          Hide
          andrew.wang Andrew Wang added a comment -

          Committed to trunk, thanks Manoj for the patch and Takanobu for the review!

          Manoj, do you mind adding a release note for this change? I'm going to be conservative and mark this as incompatible so it is flagged in the release notes.

          Show
          andrew.wang Andrew Wang added a comment - Committed to trunk, thanks Manoj for the patch and Takanobu for the review! Manoj, do you mind adding a release note for this change? I'm going to be conservative and mark this as incompatible so it is flagged in the release notes.
          Hide
          manojg Manoj Govindassamy added a comment -

          Thanks for the review and commit help Andrew Wang, Takanobu Asanuma.

          Show
          manojg Manoj Govindassamy added a comment - Thanks for the review and commit help Andrew Wang , Takanobu Asanuma .
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11401 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11401/)
          HDFS-11505. Do not enable any erasure coding policies by default. (wang: rev 7515e75103c06ce7139b305dd04d4fb2e94b12ad)

          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReadStripedFileWithMissingBlocks.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReconstructStripedFile.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestWriteReadStripedFile.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockTokenWithDFSStriped.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileChecksum.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/TestOfflineImageViewerWithStripedBlocks.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSEditLogLoader.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeMXBean.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestErasureCodeBenchmarkThroughput.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddStripedBlockInFBR.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStripedINodeFile.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLeaseRecoveryStriped.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestMover.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEnabledECPolicies.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReadStripedFileWithDecoding.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSafeModeWithStripedFile.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddStripedBlocks.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileStatusWithECPolicy.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestQuotaWithStripedBlocks.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSErasureCoding.md
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommissionWithStriped.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeErasureCodingMetrics.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ErasureCodingPolicyManager.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestErasureCodingPolicyWithSnapshot.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestSequentialBlockGroupId.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddOverReplicatedStripedBlocks.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReconstructStripedBlocksWithRackAwareness.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestReconstructStripedBlocks.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11401 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11401/ ) HDFS-11505 . Do not enable any erasure coding policies by default. (wang: rev 7515e75103c06ce7139b305dd04d4fb2e94b12ad) (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReadStripedFileWithMissingBlocks.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReconstructStripedFile.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestWriteReadStripedFile.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockTokenWithDFSStriped.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileChecksum.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/TestOfflineImageViewerWithStripedBlocks.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSEditLogLoader.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeMXBean.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestErasureCodeBenchmarkThroughput.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddStripedBlockInFBR.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStripedINodeFile.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLeaseRecoveryStriped.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestMover.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEnabledECPolicies.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReadStripedFileWithDecoding.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSafeModeWithStripedFile.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddStripedBlocks.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileStatusWithECPolicy.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestQuotaWithStripedBlocks.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSErasureCoding.md (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommissionWithStriped.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeErasureCodingMetrics.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ErasureCodingPolicyManager.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestErasureCodingPolicyWithSnapshot.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestSequentialBlockGroupId.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddOverReplicatedStripedBlocks.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReconstructStripedBlocksWithRackAwareness.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestReconstructStripedBlocks.java

            People

            • Assignee:
              manojg Manoj Govindassamy
              Reporter:
              andrew.wang Andrew Wang
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development