Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8031 Follow-on work for erasure coding phase I (striping layout)
  3. HDFS-9621

getListing wrongly associates Erasure Coding policy to pre-existing replicated files under an EC directory

    Details

    • Hadoop Flags:
      Reviewed

      Description

      This is reported by Sushmitha Sreenivasan:

      If we set Erasure Coding policy to a directory which contains some files with replicated blocks, later when listing files under the directory these files will be reported as EC files.

      1. HDFS-9621.002.branch-2.patch
        15 kB
        Jing Zhao
      2. HDFS-9621.002.patch
        22 kB
        Jing Zhao
      3. HDFS-9621.001.patch
        21 kB
        Jing Zhao
      4. HDFS-9621.000.patch
        6 kB
        Jing Zhao

        Issue Links

          Activity

          Hide
          zhz Zhe Zhang added a comment -

          Thanks Jing for the work and Kihwal for backporting to branch-2.8. I just backported to branch-2.7 because it's an improvement and makes backporting HDFS-10616 subtasks easier.

          Show
          zhz Zhe Zhang added a comment - Thanks Jing for the work and Kihwal for backporting to branch-2.8. I just backported to branch-2.7 because it's an improvement and makes backporting HDFS-10616 subtasks easier.
          Hide
          kihwal Kihwal Lee added a comment -

          Picked to branch-2.8. Although it does not have the same bug, but it is a good improvement and makes merging easier.

          Show
          kihwal Kihwal Lee added a comment - Picked to branch-2.8. Although it does not have the same bug, but it is a good improvement and makes merging easier.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9092 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9092/)
          move HDFS-9621 from trunk to 2.9.0 in CHANGES.txt. (jing9: rev 17158647f8b7cda96bfbf82a78d543befac1f01c)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9092 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9092/ ) move HDFS-9621 from trunk to 2.9.0 in CHANGES.txt. (jing9: rev 17158647f8b7cda96bfbf82a78d543befac1f01c) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          jingzhao Jing Zhao added a comment -

          Thanks Nicholas for reviewing the branch-2 patch! I've committed it.

          Show
          jingzhao Jing Zhao added a comment - Thanks Nicholas for reviewing the branch-2 patch! I've committed it.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          +1 the branch-2 patch looks good.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - +1 the branch-2 patch looks good.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9085 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9085/)
          HDFS-9621. getListing wrongly associates Erasure Coding policy to (jing9: rev 9f4bf3bdf9e74800643477cfb18361e01cf6859c)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestErasureCodingPolicies.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirErasureCodingOp.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodesInPath.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9085 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9085/ ) HDFS-9621 . getListing wrongly associates Erasure Coding policy to (jing9: rev 9f4bf3bdf9e74800643477cfb18361e01cf6859c) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestErasureCodingPolicies.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirErasureCodingOp.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodesInPath.java
          Hide
          jingzhao Jing Zhao added a comment -

          The createFileStatus change should also be included in branch-2. Upload a patch for branch-2.

          Show
          jingzhao Jing Zhao added a comment - The createFileStatus change should also be included in branch-2. Upload a patch for branch-2.
          Hide
          jingzhao Jing Zhao added a comment -

          I've committed this to trunk. Thanks Nicholas, Zhe, and Kai for the review!

          Show
          jingzhao Jing Zhao added a comment - I've committed this to trunk. Thanks Nicholas, Zhe, and Kai for the review!
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          +1 patch looks good

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - +1 patch looks good
          Hide
          zhz Zhe Zhang added a comment -

          Thanks Jing for the fix. The patch LGTM. A nit on the Javadoc:

          -   * @param src The string representation of the path to the file
          +   * @param iip The path to the file, the file is included
          
          Show
          zhz Zhe Zhang added a comment - Thanks Jing for the fix. The patch LGTM. A nit on the Javadoc: - * @param src The string representation of the path to the file + * @param iip The path to the file, the file is included
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 7m 55s trunk passed
          +1 compile 0m 56s trunk passed with JDK v1.8.0_66
          +1 compile 0m 44s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 17s trunk passed
          +1 mvnsite 0m 53s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          +1 findbugs 1m 56s trunk passed
          +1 javadoc 1m 5s trunk passed with JDK v1.8.0_66
          +1 javadoc 1m 48s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 50s the patch passed
          +1 compile 0m 40s the patch passed with JDK v1.8.0_66
          +1 javac 0m 40s the patch passed
          +1 compile 0m 42s the patch passed with JDK v1.7.0_91
          +1 javac 0m 42s the patch passed
          -1 checkstyle 0m 18s Patch generated 3 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 189, now 184).
          +1 mvnsite 0m 54s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 0s the patch passed
          +1 javadoc 1m 2s the patch passed with JDK v1.8.0_66
          +1 javadoc 1m 43s the patch passed with JDK v1.7.0_91
          -1 unit 69m 2s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          +1 unit 66m 38s hadoop-hdfs in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 21s Patch does not generate ASF License warnings.
          163m 3s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.datanode.TestBlockScanner
            hadoop.hdfs.server.datanode.TestBlockReplacement



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12781077/HDFS-9621.001.patch
          JIRA Issue HDFS-9621
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux b0efb59cd365 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 52b7757
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14058/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14058/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14058/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14058/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14058/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 7m 55s trunk passed +1 compile 0m 56s trunk passed with JDK v1.8.0_66 +1 compile 0m 44s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 17s trunk passed +1 mvnsite 0m 53s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 1m 56s trunk passed +1 javadoc 1m 5s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 48s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 50s the patch passed +1 compile 0m 40s the patch passed with JDK v1.8.0_66 +1 javac 0m 40s the patch passed +1 compile 0m 42s the patch passed with JDK v1.7.0_91 +1 javac 0m 42s the patch passed -1 checkstyle 0m 18s Patch generated 3 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 189, now 184). +1 mvnsite 0m 54s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 0s the patch passed +1 javadoc 1m 2s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 43s the patch passed with JDK v1.7.0_91 -1 unit 69m 2s hadoop-hdfs in the patch failed with JDK v1.8.0_66. +1 unit 66m 38s hadoop-hdfs in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 21s Patch does not generate ASF License warnings. 163m 3s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.server.datanode.TestBlockReplacement Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12781077/HDFS-9621.001.patch JIRA Issue HDFS-9621 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux b0efb59cd365 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 52b7757 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14058/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14058/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14058/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14058/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14058/console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 8m 22s trunk passed
          +1 compile 0m 46s trunk passed with JDK v1.8.0_66
          +1 compile 0m 43s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 20s trunk passed
          +1 mvnsite 0m 55s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          +1 findbugs 2m 7s trunk passed
          +1 javadoc 1m 13s trunk passed with JDK v1.8.0_66
          +1 javadoc 1m 53s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 51s the patch passed
          +1 compile 0m 42s the patch passed with JDK v1.8.0_66
          +1 javac 0m 42s the patch passed
          +1 compile 0m 44s the patch passed with JDK v1.7.0_91
          +1 javac 0m 44s the patch passed
          -1 checkstyle 0m 20s Patch generated 3 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 189, now 184).
          +1 mvnsite 0m 55s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 13s the patch passed
          +1 javadoc 1m 5s the patch passed with JDK v1.8.0_66
          +1 javadoc 1m 48s the patch passed with JDK v1.7.0_91
          -1 unit 52m 54s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          +1 unit 49m 16s hadoop-hdfs in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 20s Patch does not generate ASF License warnings.
          130m 47s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestAppendSnapshotTruncate
            hadoop.hdfs.server.namenode.TestNNThroughputBenchmark
            hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12781080/HDFS-9621.002.patch
          JIRA Issue HDFS-9621
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux fc3fc363842e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 89022f8
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14059/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14059/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14059/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14059/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14059/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 8m 22s trunk passed +1 compile 0m 46s trunk passed with JDK v1.8.0_66 +1 compile 0m 43s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 20s trunk passed +1 mvnsite 0m 55s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 2m 7s trunk passed +1 javadoc 1m 13s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 53s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 51s the patch passed +1 compile 0m 42s the patch passed with JDK v1.8.0_66 +1 javac 0m 42s the patch passed +1 compile 0m 44s the patch passed with JDK v1.7.0_91 +1 javac 0m 44s the patch passed -1 checkstyle 0m 20s Patch generated 3 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 189, now 184). +1 mvnsite 0m 55s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 13s the patch passed +1 javadoc 1m 5s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 48s the patch passed with JDK v1.7.0_91 -1 unit 52m 54s hadoop-hdfs in the patch failed with JDK v1.8.0_66. +1 unit 49m 16s hadoop-hdfs in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 20s Patch does not generate ASF License warnings. 130m 47s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestAppendSnapshotTruncate   hadoop.hdfs.server.namenode.TestNNThroughputBenchmark   hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12781080/HDFS-9621.002.patch JIRA Issue HDFS-9621 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux fc3fc363842e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 89022f8 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14059/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14059/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14059/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14059/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14059/console This message was automatically generated.
          Hide
          jingzhao Jing Zhao added a comment -

          With the change looks like the fromINode call is no longer required for checking if the INode is encrypted. Update the patch.

          Show
          jingzhao Jing Zhao added a comment - With the change looks like the fromINode call is no longer required for checking if the INode is encrypted. Update the patch.
          Hide
          jingzhao Jing Zhao added a comment -

          The root cause of the issue is the semantic of the current createFileStatus is confusing: for getListing the iip parameter does not include the target node, while for getFileInfo the iip parameter includes the target node.

          The 001 patch refactors the createFileStatus methods and let iip always include the target node.

          Show
          jingzhao Jing Zhao added a comment - The root cause of the issue is the semantic of the current createFileStatus is confusing: for getListing the iip parameter does not include the target node, while for getFileInfo the iip parameter includes the target node. The 001 patch refactors the createFileStatus methods and let iip always include the target node.
          Hide
          jingzhao Jing Zhao added a comment -

          Thanks for the review, Kai! Will update the patch to address your comments.

          Looks like we need to cover more scenarios in the fix. The current fix only checks the ancestral path of a file for the erasure coding policy, but cannot handle

          1. the file is moved from an EC dir into a non-EC dir
          2. the node is a directory and has EC policy associated

          I will fix these and include more unit tests.

          Show
          jingzhao Jing Zhao added a comment - Thanks for the review, Kai! Will update the patch to address your comments. Looks like we need to cover more scenarios in the fix. The current fix only checks the ancestral path of a file for the erasure coding policy, but cannot handle the file is moved from an EC dir into a non-EC dir the node is a directory and has EC policy associated I will fix these and include more unit tests.
          Hide
          drankye Kai Zheng added a comment -

          Thanks for the work Jing. The patch looks great. Some minor comments:
          1. For this change:

          +      ecPolicy = fileNode.isStriped() ? ecPolicy : null;
          

          How about:

          ecPolicy = null;
          if (fileNode.isStriped()) {
            ecPolicy = FSDirErasureCodingOp.getErasureCodingPolicy(fsd.getFSNamesystem(), iip);
          }
          

          2. For this codes:

          +    DirectoryListing listing = fs.getClient().listPaths(dir.toString(),
          +        new byte[0], false);
          +    HdfsFileStatus[] files = listing.getPartialListing();
          +    assertNotNull(files[0].getErasureCodingPolicy()); // ecSubDir
          +    assertNull(files[1].getErasureCodingPolicy()); // replicatedFile
          

          Might be not very reliable relying on the listed entry order considering listPaths or getPartialListing may change in implementation.

          Show
          drankye Kai Zheng added a comment - Thanks for the work Jing. The patch looks great. Some minor comments: 1. For this change: + ecPolicy = fileNode.isStriped() ? ecPolicy : null ; How about: ecPolicy = null ; if (fileNode.isStriped()) { ecPolicy = FSDirErasureCodingOp.getErasureCodingPolicy(fsd.getFSNamesystem(), iip); } 2. For this codes: + DirectoryListing listing = fs.getClient().listPaths(dir.toString(), + new byte [0], false ); + HdfsFileStatus[] files = listing.getPartialListing(); + assertNotNull(files[0].getErasureCodingPolicy()); // ecSubDir + assertNull(files[1].getErasureCodingPolicy()); // replicatedFile Might be not very reliable relying on the listed entry order considering listPaths or getPartialListing may change in implementation.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 7m 35s trunk passed
          +1 compile 0m 46s trunk passed with JDK v1.8.0_66
          +1 compile 0m 44s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 19s trunk passed
          +1 mvnsite 0m 56s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          +1 findbugs 1m 59s trunk passed
          +1 javadoc 1m 14s trunk passed with JDK v1.8.0_66
          +1 javadoc 1m 55s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 49s the patch passed
          +1 compile 0m 43s the patch passed with JDK v1.8.0_66
          +1 javac 0m 43s the patch passed
          +1 compile 0m 42s the patch passed with JDK v1.7.0_91
          +1 javac 0m 42s the patch passed
          +1 checkstyle 0m 18s the patch passed
          +1 mvnsite 0m 55s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 10s the patch passed
          +1 javadoc 1m 9s the patch passed with JDK v1.8.0_66
          +1 javadoc 1m 51s the patch passed with JDK v1.7.0_91
          -1 unit 57m 11s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          +1 unit 49m 50s hadoop-hdfs in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 20s Patch does not generate ASF License warnings.
          134m 42s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestDistributedFileSystem
            hadoop.hdfs.server.namenode.TestNNThroughputBenchmark



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12780865/HDFS-9621.000.patch
          JIRA Issue HDFS-9621
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 51ade031330c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / b6c9d3f
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14049/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14049/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14049/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14049/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 7m 35s trunk passed +1 compile 0m 46s trunk passed with JDK v1.8.0_66 +1 compile 0m 44s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 19s trunk passed +1 mvnsite 0m 56s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 1m 59s trunk passed +1 javadoc 1m 14s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 55s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 49s the patch passed +1 compile 0m 43s the patch passed with JDK v1.8.0_66 +1 javac 0m 43s the patch passed +1 compile 0m 42s the patch passed with JDK v1.7.0_91 +1 javac 0m 42s the patch passed +1 checkstyle 0m 18s the patch passed +1 mvnsite 0m 55s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 10s the patch passed +1 javadoc 1m 9s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 51s the patch passed with JDK v1.7.0_91 -1 unit 57m 11s hadoop-hdfs in the patch failed with JDK v1.8.0_66. +1 unit 49m 50s hadoop-hdfs in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 20s Patch does not generate ASF License warnings. 134m 42s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestDistributedFileSystem   hadoop.hdfs.server.namenode.TestNNThroughputBenchmark Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12780865/HDFS-9621.000.patch JIRA Issue HDFS-9621 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 51ade031330c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / b6c9d3f Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/14049/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14049/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14049/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14049/console This message was automatically generated.
          Hide
          jingzhao Jing Zhao added a comment -

          Upload a patch to fix.

          Show
          jingzhao Jing Zhao added a comment - Upload a patch to fix.

            People

            • Assignee:
              jingzhao Jing Zhao
              Reporter:
              ssreenivasan Sushmitha Sreenivasan
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development