Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6774

Add support for HDFS erasure code policy to TestDFSIO

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0-alpha2
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      HDFS erasure code policy allows user to store directory and file to predefined erasure code policies. Currently only 3x replication is supported in TestDFSIO implementation. This is going to add an new option to enable tests of files with erasure code policy enabled.

      1. MAPREDUCE-6774-v1.patch
        9 kB
        SammiChen
      2. MAPREDUCE-6774-v2.patch
        9 kB
        SammiChen
      3. MAPREDUCE-6774-v3.patch
        9 kB
        SammiChen
      4. MAPREDUCE-6774-v4.patch
        9 kB
        SammiChen
      5. MAPREDUCE-6774-v5.patch
        10 kB
        SammiChen
      6. MAPREDUCE-6774-v6.patch
        10 kB
        SammiChen

        Activity

        Hide
        Sammi SammiChen added a comment -

        Thanks Kai for review the patch and help commit to trunk!

        Show
        Sammi SammiChen added a comment - Thanks Kai for review the patch and help commit to trunk!
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10452 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10452/)
        MAPREDUCE-6774. Add support for HDFS erasure code policy to TestDFSIO. (kai.zheng: rev 501a77856d6b6edfb261547117e719da7a9cd221)

        • (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/TestDFSIO.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10452 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10452/ ) MAPREDUCE-6774 . Add support for HDFS erasure code policy to TestDFSIO. (kai.zheng: rev 501a77856d6b6edfb261547117e719da7a9cd221) (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/TestDFSIO.java
        Hide
        drankye Kai Zheng added a comment -

        Committed to trunk. Thanks Sammi for the contribution!

        Show
        drankye Kai Zheng added a comment - Committed to trunk. Thanks Sammi for the contribution!
        Hide
        drankye Kai Zheng added a comment -

        Thanks Sammi for the update! The latest patch LGTM and +1.

        Show
        drankye Kai Zheng added a comment - Thanks Sammi for the update! The latest patch LGTM and +1.
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 26s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 8m 44s trunk passed
        +1 compile 0m 25s trunk passed
        +1 checkstyle 0m 18s trunk passed
        +1 mvnsite 0m 32s trunk passed
        +1 mvneclipse 0m 16s trunk passed
        +1 findbugs 0m 25s trunk passed
        +1 javadoc 0m 15s trunk passed
        +1 mvninstall 0m 22s the patch passed
        +1 compile 0m 21s the patch passed
        +1 javac 0m 21s the patch passed
        +1 checkstyle 0m 15s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: The patch generated 0 new + 45 unchanged - 5 fixed = 45 total (was 50)
        +1 mvnsite 0m 28s the patch passed
        +1 mvneclipse 0m 13s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 0m 29s the patch passed
        +1 javadoc 0m 11s the patch passed
        +1 unit 119m 42s hadoop-mapreduce-client-jobclient in the patch passed.
        +1 asflicense 0m 27s The patch does not generate ASF License warnings.
        134m 30s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12828433/MAPREDUCE-6774-v6.patch
        JIRA Issue MAPREDUCE-6774
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux aaaf6aff956f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / ea0c2b8
        Default Java 1.8.0_101
        findbugs v3.0.0
        Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6718/testReport/
        modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6718/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 26s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 44s trunk passed +1 compile 0m 25s trunk passed +1 checkstyle 0m 18s trunk passed +1 mvnsite 0m 32s trunk passed +1 mvneclipse 0m 16s trunk passed +1 findbugs 0m 25s trunk passed +1 javadoc 0m 15s trunk passed +1 mvninstall 0m 22s the patch passed +1 compile 0m 21s the patch passed +1 javac 0m 21s the patch passed +1 checkstyle 0m 15s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: The patch generated 0 new + 45 unchanged - 5 fixed = 45 total (was 50) +1 mvnsite 0m 28s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 29s the patch passed +1 javadoc 0m 11s the patch passed +1 unit 119m 42s hadoop-mapreduce-client-jobclient in the patch passed. +1 asflicense 0m 27s The patch does not generate ASF License warnings. 134m 30s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12828433/MAPREDUCE-6774-v6.patch JIRA Issue MAPREDUCE-6774 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux aaaf6aff956f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ea0c2b8 Default Java 1.8.0_101 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6718/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6718/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        Sammi SammiChen added a comment -

        Fix 2 style issues

        Show
        Sammi SammiChen added a comment - Fix 2 style issues
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 10s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 7m 41s trunk passed
        +1 compile 0m 23s trunk passed
        +1 checkstyle 0m 18s trunk passed
        +1 mvnsite 0m 29s trunk passed
        +1 mvneclipse 0m 16s trunk passed
        +1 findbugs 0m 24s trunk passed
        +1 javadoc 0m 13s trunk passed
        +1 mvninstall 0m 21s the patch passed
        +1 compile 0m 19s the patch passed
        +1 javac 0m 19s the patch passed
        -1 checkstyle 0m 14s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: The patch generated 2 new + 45 unchanged - 5 fixed = 47 total (was 50)
        +1 mvnsite 0m 26s the patch passed
        +1 mvneclipse 0m 13s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 0m 27s the patch passed
        +1 javadoc 0m 10s the patch passed
        +1 unit 126m 30s hadoop-mapreduce-client-jobclient in the patch passed.
        +1 asflicense 0m 27s The patch does not generate ASF License warnings.
        139m 35s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12828406/MAPREDUCE-6774-v5.patch
        JIRA Issue MAPREDUCE-6774
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 746e781eb2ed 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / ea0c2b8
        Default Java 1.8.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6717/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
        Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6717/testReport/
        modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6717/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 41s trunk passed +1 compile 0m 23s trunk passed +1 checkstyle 0m 18s trunk passed +1 mvnsite 0m 29s trunk passed +1 mvneclipse 0m 16s trunk passed +1 findbugs 0m 24s trunk passed +1 javadoc 0m 13s trunk passed +1 mvninstall 0m 21s the patch passed +1 compile 0m 19s the patch passed +1 javac 0m 19s the patch passed -1 checkstyle 0m 14s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: The patch generated 2 new + 45 unchanged - 5 fixed = 47 total (was 50) +1 mvnsite 0m 26s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 27s the patch passed +1 javadoc 0m 10s the patch passed +1 unit 126m 30s hadoop-mapreduce-client-jobclient in the patch passed. +1 asflicense 0m 27s The patch does not generate ASF License warnings. 139m 35s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12828406/MAPREDUCE-6774-v5.patch JIRA Issue MAPREDUCE-6774 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 746e781eb2ed 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ea0c2b8 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6717/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6717/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6717/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        Sammi SammiChen added a comment -

        Improved patch based on Kai's suggestion

        Show
        Sammi SammiChen added a comment - Improved patch based on Kai's suggestion
        Hide
        drankye Kai Zheng added a comment -

        Thanks Sammi for the update! It looks good and only some minors now.

        1. Could you define constants for: test.io.erasure.code.policy, test.io.block.storage.policy?
        2. Would like to see minor refinements for checkErasureCodePolicy. For example, having some line breaks, avoiding the else.

        +  private boolean checkErasureCodePolicy(String erasureCodePolicyName,
        +      FileSystem fs, TestType testType) throws IOException {
        +    Collection<ErasureCodingPolicy> list =
        +        ((DistributedFileSystem) fs).getAllErasureCodingPolicies();
        +    boolean isValid = false;
        +    int i = 0;
        +    for (ErasureCodingPolicy ec : list) {
        +      if (erasureCodePolicyName.equals(ec.getName())) {
        +        isValid = true;
        +        break;
        +      }
        +    }
        +    if (!isValid) {
        +      System.out.println("Invalid erasure code policy: " +
        +          erasureCodePolicyName);
        +      System.out.println("Current supported erasure code policy list: ");
        +      for (ErasureCodingPolicy ec : list) {
        +        System.out.println(ec.getName());
        +      }
        +      return false;
        +    } else {
        +      if (testType == TestType.TEST_TYPE_APPEND ||
        +          testType == TestType.TEST_TYPE_TRUNCATE) {
        +        System.out.println("So far append or truncate operation" +
        +            " with erasureCodePolicy enabled is not supported");
        +        return false;
        +      }
        +    }
        +    config.set("test.io.erasure.code.policy", erasureCodePolicyName);
        +    LOG.info("erasureCodePolicy = " + erasureCodePolicyName);
        +    return true;
        +  }
        
        Show
        drankye Kai Zheng added a comment - Thanks Sammi for the update! It looks good and only some minors now. 1. Could you define constants for: test.io.erasure.code.policy , test.io.block.storage.policy ? 2. Would like to see minor refinements for checkErasureCodePolicy . For example, having some line breaks, avoiding the else . + private boolean checkErasureCodePolicy( String erasureCodePolicyName, + FileSystem fs, TestType testType) throws IOException { + Collection<ErasureCodingPolicy> list = + ((DistributedFileSystem) fs).getAllErasureCodingPolicies(); + boolean isValid = false ; + int i = 0; + for (ErasureCodingPolicy ec : list) { + if (erasureCodePolicyName.equals(ec.getName())) { + isValid = true ; + break ; + } + } + if (!isValid) { + System .out.println( "Invalid erasure code policy: " + + erasureCodePolicyName); + System .out.println( "Current supported erasure code policy list: " ); + for (ErasureCodingPolicy ec : list) { + System .out.println(ec.getName()); + } + return false ; + } else { + if (testType == TestType.TEST_TYPE_APPEND || + testType == TestType.TEST_TYPE_TRUNCATE) { + System .out.println( "So far append or truncate operation" + + " with erasureCodePolicy enabled is not supported" ); + return false ; + } + } + config.set( "test.io.erasure.code.policy" , erasureCodePolicyName); + LOG.info( "erasureCodePolicy = " + erasureCodePolicyName); + return true ; + }
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 18s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 6m 57s trunk passed
        +1 compile 0m 23s trunk passed
        +1 checkstyle 0m 17s trunk passed
        +1 mvnsite 0m 30s trunk passed
        +1 mvneclipse 0m 16s trunk passed
        +1 findbugs 0m 23s trunk passed
        +1 javadoc 0m 13s trunk passed
        +1 mvninstall 0m 22s the patch passed
        +1 compile 0m 20s the patch passed
        +1 javac 0m 20s the patch passed
        +1 checkstyle 0m 14s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: The patch generated 0 new + 45 unchanged - 5 fixed = 45 total (was 50)
        +1 mvnsite 0m 27s the patch passed
        +1 mvneclipse 0m 13s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 0m 28s the patch passed
        +1 javadoc 0m 10s the patch passed
        +1 unit 117m 35s hadoop-mapreduce-client-jobclient in the patch passed.
        +1 asflicense 0m 26s The patch does not generate ASF License warnings.
        130m 9s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12828201/MAPREDUCE-6774-v4.patch
        JIRA Issue MAPREDUCE-6774
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux d30c2069690d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 729de3e
        Default Java 1.8.0_101
        findbugs v3.0.0
        Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6716/testReport/
        modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6716/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 57s trunk passed +1 compile 0m 23s trunk passed +1 checkstyle 0m 17s trunk passed +1 mvnsite 0m 30s trunk passed +1 mvneclipse 0m 16s trunk passed +1 findbugs 0m 23s trunk passed +1 javadoc 0m 13s trunk passed +1 mvninstall 0m 22s the patch passed +1 compile 0m 20s the patch passed +1 javac 0m 20s the patch passed +1 checkstyle 0m 14s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: The patch generated 0 new + 45 unchanged - 5 fixed = 45 total (was 50) +1 mvnsite 0m 27s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 28s the patch passed +1 javadoc 0m 10s the patch passed +1 unit 117m 35s hadoop-mapreduce-client-jobclient in the patch passed. +1 asflicense 0m 26s The patch does not generate ASF License warnings. 130m 9s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12828201/MAPREDUCE-6774-v4.patch JIRA Issue MAPREDUCE-6774 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux d30c2069690d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 729de3e Default Java 1.8.0_101 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6716/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6716/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        Sammi SammiChen added a comment -

        Thanks Kai, for take time review the patch and give very good suggestion!
        The name 'checkStoragePolicy' sounds better than 'isStoragePolicyValid'. So does the name 'checkErasureCodePolicy'. I uploaded Patch v4 which contains the improvement. One the other hand, I would suggest keep the return value boolean of each check function. Because the main function depends on the return value, not the exception, to handle the invalid parameter case.

        Show
        Sammi SammiChen added a comment - Thanks Kai, for take time review the patch and give very good suggestion! The name 'checkStoragePolicy' sounds better than 'isStoragePolicyValid'. So does the name 'checkErasureCodePolicy'. I uploaded Patch v4 which contains the improvement. One the other hand, I would suggest keep the return value boolean of each check function. Because the main function depends on the return value, not the exception, to handle the invalid parameter case.
        Hide
        drankye Kai Zheng added a comment -

        So given above, the following can then be:

        +    if (storagePolicy != null) {
        +      if (!isStoragePolicyValid(storagePolicy, fs)) {
        +        return -1;
        +      }
        

        =>

        +    if (storagePolicy != null) {
        +      checkStoragePolicy(storagePolicy, fs);
        +    }
        
        Show
        drankye Kai Zheng added a comment - So given above, the following can then be: + if (storagePolicy != null ) { + if (!isStoragePolicyValid(storagePolicy, fs)) { + return -1; + } => + if (storagePolicy != null ) { + checkStoragePolicy(storagePolicy, fs); + }
        Hide
        drankye Kai Zheng added a comment -

        Thanks SammiChen for the work!

        It looks good overall. Suggestions:

        boolean isStoragePolicyValid(String storagePolicy, FileSystem fs)
        boolean isErasureCodePolicyValid(String erasureCodePolicyName, FileSystem fs, TestType testType) throws IOException
        

        would be good to:

        void checkStoragePolicy(String storagePolicy, FileSystem fs) throws Exception
        void checkErasureCodePolicy(String erasureCodePolicyName, FileSystem fs, TestType testType) throws Exception
        
        Show
        drankye Kai Zheng added a comment - Thanks SammiChen for the work! It looks good overall. Suggestions: boolean isStoragePolicyValid( String storagePolicy, FileSystem fs) boolean isErasureCodePolicyValid( String erasureCodePolicyName, FileSystem fs, TestType testType) throws IOException would be good to: void checkStoragePolicy( String storagePolicy, FileSystem fs) throws Exception void checkErasureCodePolicy( String erasureCodePolicyName, FileSystem fs, TestType testType) throws Exception
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 6m 45s trunk passed
        +1 compile 0m 24s trunk passed
        +1 checkstyle 0m 16s trunk passed
        +1 mvnsite 0m 30s trunk passed
        +1 mvneclipse 0m 15s trunk passed
        +1 findbugs 0m 25s trunk passed
        +1 javadoc 0m 13s trunk passed
        +1 mvninstall 0m 22s the patch passed
        +1 compile 0m 21s the patch passed
        +1 javac 0m 21s the patch passed
        +1 checkstyle 0m 13s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: The patch generated 0 new + 45 unchanged - 5 fixed = 45 total (was 50)
        +1 mvnsite 0m 26s the patch passed
        +1 mvneclipse 0m 13s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 0m 28s the patch passed
        +1 javadoc 0m 10s the patch passed
        -1 unit 135m 32s hadoop-mapreduce-client-jobclient in the patch failed.
        +1 asflicense 0m 23s The patch does not generate ASF License warnings.
        147m 51s



        Reason Tests
        Timed out junit tests org.apache.hadoop.mapreduce.lib.jobcontrol.TestMapReduceJobControl



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12827546/MAPREDUCE-6774-v3.patch
        JIRA Issue MAPREDUCE-6774
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux cf3bca10b7c8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 63f5948
        Default Java 1.8.0_101
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6709/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
        unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6709/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
        Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6709/testReport/
        modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6709/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 45s trunk passed +1 compile 0m 24s trunk passed +1 checkstyle 0m 16s trunk passed +1 mvnsite 0m 30s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 0m 25s trunk passed +1 javadoc 0m 13s trunk passed +1 mvninstall 0m 22s the patch passed +1 compile 0m 21s the patch passed +1 javac 0m 21s the patch passed +1 checkstyle 0m 13s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: The patch generated 0 new + 45 unchanged - 5 fixed = 45 total (was 50) +1 mvnsite 0m 26s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 28s the patch passed +1 javadoc 0m 10s the patch passed -1 unit 135m 32s hadoop-mapreduce-client-jobclient in the patch failed. +1 asflicense 0m 23s The patch does not generate ASF License warnings. 147m 51s Reason Tests Timed out junit tests org.apache.hadoop.mapreduce.lib.jobcontrol.TestMapReduceJobControl Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12827546/MAPREDUCE-6774-v3.patch JIRA Issue MAPREDUCE-6774 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux cf3bca10b7c8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 63f5948 Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6709/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6709/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6709/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6709/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        Sammi SammiChen added a comment -

        fix style issues

        Show
        Sammi SammiChen added a comment - fix style issues
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 11s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 7m 22s trunk passed
        +1 compile 0m 23s trunk passed
        +1 checkstyle 0m 17s trunk passed
        +1 mvnsite 0m 30s trunk passed
        +1 mvneclipse 0m 15s trunk passed
        +1 findbugs 0m 24s trunk passed
        +1 javadoc 0m 12s trunk passed
        +1 mvninstall 0m 21s the patch passed
        +1 compile 0m 21s the patch passed
        +1 javac 0m 21s the patch passed
        -1 checkstyle 0m 14s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: The patch generated 5 new + 45 unchanged - 5 fixed = 50 total (was 50)
        +1 mvnsite 0m 25s the patch passed
        +1 mvneclipse 0m 12s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 0m 30s the patch passed
        +1 javadoc 0m 10s the patch passed
        +1 unit 123m 7s hadoop-mapreduce-client-jobclient in the patch passed.
        +1 asflicense 0m 29s The patch does not generate ASF License warnings.
        136m 0s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12827513/MAPREDUCE-6774-v2.patch
        JIRA Issue MAPREDUCE-6774
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 3b6c36917bf8 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 63f5948
        Default Java 1.8.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6708/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
        Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6708/testReport/
        modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6708/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 22s trunk passed +1 compile 0m 23s trunk passed +1 checkstyle 0m 17s trunk passed +1 mvnsite 0m 30s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 0m 24s trunk passed +1 javadoc 0m 12s trunk passed +1 mvninstall 0m 21s the patch passed +1 compile 0m 21s the patch passed +1 javac 0m 21s the patch passed -1 checkstyle 0m 14s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: The patch generated 5 new + 45 unchanged - 5 fixed = 50 total (was 50) +1 mvnsite 0m 25s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 30s the patch passed +1 javadoc 0m 10s the patch passed +1 unit 123m 7s hadoop-mapreduce-client-jobclient in the patch passed. +1 asflicense 0m 29s The patch does not generate ASF License warnings. 136m 0s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12827513/MAPREDUCE-6774-v2.patch JIRA Issue MAPREDUCE-6774 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 3b6c36917bf8 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 63f5948 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6708/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6708/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6708/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        Sammi SammiChen added a comment -

        1. improve the patch, only io_data directory is mattered when apply EC policy
        2. fix checkstyle reported issue

        Show
        Sammi SammiChen added a comment - 1. improve the patch, only io_data directory is mattered when apply EC policy 2. fix checkstyle reported issue
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 8m 47s trunk passed
        +1 compile 0m 27s trunk passed
        +1 checkstyle 0m 18s trunk passed
        +1 mvnsite 0m 33s trunk passed
        +1 mvneclipse 0m 18s trunk passed
        +1 findbugs 0m 28s trunk passed
        +1 javadoc 0m 14s trunk passed
        +1 mvninstall 0m 27s the patch passed
        +1 compile 0m 26s the patch passed
        +1 javac 0m 26s the patch passed
        -1 checkstyle 0m 17s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: The patch generated 5 new + 50 unchanged - 0 fixed = 55 total (was 50)
        +1 mvnsite 0m 32s the patch passed
        +1 mvneclipse 0m 15s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 0m 38s the patch passed
        +1 javadoc 0m 11s the patch passed
        -1 unit 143m 25s hadoop-mapreduce-client-jobclient in the patch failed.
        +1 asflicense 0m 28s The patch does not generate ASF License warnings.
        158m 46s



        Reason Tests
        Failed junit tests hadoop.mapred.TestMROpportunisticMaps
          hadoop.mapred.TestReduceFetch
          hadoop.mapred.TestMerge
          hadoop.mapreduce.TestMapReduceLazyOutput
          hadoop.mapred.TestMRIntermediateDataEncryption
          hadoop.mapred.TestLazyOutput
          hadoop.mapreduce.TestLargeSort
          hadoop.mapred.TestReduceFetchFromPartialMem
          hadoop.mapreduce.v2.TestMRJobsWithProfiler
          hadoop.mapreduce.lib.output.TestJobOutputCommitter
          hadoop.mapreduce.security.ssl.TestEncryptedShuffle
          hadoop.mapreduce.v2.TestMROldApiJobs
          hadoop.mapred.TestJobCleanup
          hadoop.mapreduce.v2.TestSpeculativeExecution
          hadoop.mapred.TestClusterMRNotification
          hadoop.mapreduce.security.TestUmbilicalProtocolWithJobToken
          hadoop.mapreduce.v2.TestMRAMWithNonNormalizedCapabilities
          hadoop.mapreduce.v2.TestMRJobs
          hadoop.mapred.TestJobName
          hadoop.mapreduce.TestMRJobClient
          hadoop.mapred.TestClusterMapReduceTestCase
          hadoop.mapred.TestAuditLogger
          hadoop.mapreduce.security.TestMRCredentials
          hadoop.mapred.TestMRTimelineEventHandling
          hadoop.mapreduce.v2.TestMiniMRProxyUser
          hadoop.mapreduce.v2.TestMRJobsWithHistoryService
          hadoop.mapred.TestMiniMRClientCluster
          hadoop.mapred.TestMiniMRChildTask
          hadoop.mapreduce.TestChild
          hadoop.mapreduce.security.TestBinaryTokenFile
          hadoop.mapred.TestJobCounters
        Timed out junit tests org.apache.hadoop.mapred.TestMiniMRClasspath
          org.apache.hadoop.mapred.TestJobSysDirWithDFS
          org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12827186/MAPREDUCE-6774-v1.patch
        JIRA Issue MAPREDUCE-6774
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 28e213582d6e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 62a9667
        Default Java 1.8.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6706/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
        unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6706/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
        unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6706/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
        Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6706/testReport/
        modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6706/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 47s trunk passed +1 compile 0m 27s trunk passed +1 checkstyle 0m 18s trunk passed +1 mvnsite 0m 33s trunk passed +1 mvneclipse 0m 18s trunk passed +1 findbugs 0m 28s trunk passed +1 javadoc 0m 14s trunk passed +1 mvninstall 0m 27s the patch passed +1 compile 0m 26s the patch passed +1 javac 0m 26s the patch passed -1 checkstyle 0m 17s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: The patch generated 5 new + 50 unchanged - 0 fixed = 55 total (was 50) +1 mvnsite 0m 32s the patch passed +1 mvneclipse 0m 15s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 38s the patch passed +1 javadoc 0m 11s the patch passed -1 unit 143m 25s hadoop-mapreduce-client-jobclient in the patch failed. +1 asflicense 0m 28s The patch does not generate ASF License warnings. 158m 46s Reason Tests Failed junit tests hadoop.mapred.TestMROpportunisticMaps   hadoop.mapred.TestReduceFetch   hadoop.mapred.TestMerge   hadoop.mapreduce.TestMapReduceLazyOutput   hadoop.mapred.TestMRIntermediateDataEncryption   hadoop.mapred.TestLazyOutput   hadoop.mapreduce.TestLargeSort   hadoop.mapred.TestReduceFetchFromPartialMem   hadoop.mapreduce.v2.TestMRJobsWithProfiler   hadoop.mapreduce.lib.output.TestJobOutputCommitter   hadoop.mapreduce.security.ssl.TestEncryptedShuffle   hadoop.mapreduce.v2.TestMROldApiJobs   hadoop.mapred.TestJobCleanup   hadoop.mapreduce.v2.TestSpeculativeExecution   hadoop.mapred.TestClusterMRNotification   hadoop.mapreduce.security.TestUmbilicalProtocolWithJobToken   hadoop.mapreduce.v2.TestMRAMWithNonNormalizedCapabilities   hadoop.mapreduce.v2.TestMRJobs   hadoop.mapred.TestJobName   hadoop.mapreduce.TestMRJobClient   hadoop.mapred.TestClusterMapReduceTestCase   hadoop.mapred.TestAuditLogger   hadoop.mapreduce.security.TestMRCredentials   hadoop.mapred.TestMRTimelineEventHandling   hadoop.mapreduce.v2.TestMiniMRProxyUser   hadoop.mapreduce.v2.TestMRJobsWithHistoryService   hadoop.mapred.TestMiniMRClientCluster   hadoop.mapred.TestMiniMRChildTask   hadoop.mapreduce.TestChild   hadoop.mapreduce.security.TestBinaryTokenFile   hadoop.mapred.TestJobCounters Timed out junit tests org.apache.hadoop.mapred.TestMiniMRClasspath   org.apache.hadoop.mapred.TestJobSysDirWithDFS   org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12827186/MAPREDUCE-6774-v1.patch JIRA Issue MAPREDUCE-6774 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 28e213582d6e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 62a9667 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6706/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6706/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6706/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6706/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6706/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.

          People

          • Assignee:
            Sammi SammiChen
            Reporter:
            Sammi SammiChen
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development