Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11084

Add a regression test for sticky bit support of OIV ReverseXML processor

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.8.0
    • Fix Version/s: 2.9.0, 3.0.0-alpha4, 2.8.2
    • Component/s: tools
    • Labels:
      None

      Description

      HDFS-10505 added a new feature OIV ReverseXML processor to generate a fsimage from a xml file. However, if the files/directories in it has sticky bit, ReverseXML processor can not recognize it due to HADOOP-13508.

      It seems HADOOP-13508 is an incompatible change in Hadoop 3. Would it be reasonable to add an overloaded FsPermission constructor that uses RawParser so that it reads sticky bits correctly? Or is it reasonable to backport HADOOP-13508 to branch-2?

      1. HDFS-11084.02.patch
        2 kB
        Akira Ajisaka
      2. HDFS-11804.branch-2.002.patch
        2 kB
        Wei-Chiu Chuang
      3. HDFS-11804.branch-2.patch
        1 kB
        Wei-Chiu Chuang

        Issue Links

          Activity

          Hide
          chris.douglas Chris Douglas added a comment -

          Sorry Wei-Chiu Chuang, I'd missed your comment on HADOOP-13508 and this issue.

          Can you be more specific? You seem to have traced a bug, but your report contains too few details to reconstruct your analysis. Did HADOOP-13508 cause a regression? Or are you suggesting backporting it to solve a similar issue in branch-2?

          Show
          chris.douglas Chris Douglas added a comment - Sorry Wei-Chiu Chuang , I'd missed your comment on HADOOP-13508 and this issue. Can you be more specific? You seem to have traced a bug, but your report contains too few details to reconstruct your analysis. Did HADOOP-13508 cause a regression? Or are you suggesting backporting it to solve a similar issue in branch-2?
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          For the context, OIV is a tool that dumps fsimage into a human-readable file, such as XML.
          OIV ReverseXML is a tool that imports an XML file into a fsimage.

          So ideally, you can do fsimage --> XML --> fsimage, and the sourcee fsimage is exactly the same as the destination fsimage. However, in branch-2 if the fsimage contains directories with sticky bits, it fails with the following exception:

          Invalid processor specified : invalid
          OfflineImageReconstructor failed: 1777
          java.lang.IllegalArgumentException: 1777
          	at org.apache.hadoop.fs.permission.PermissionParser.<init>(PermissionParser.java:60)
          	at org.apache.hadoop.fs.permission.UmaskParser.<init>(UmaskParser.java:42)
          	at org.apache.hadoop.fs.permission.FsPermission.<init>(FsPermission.java:106)
          	at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.permissionXmlToU64(OfflineImageReconstructor.java:1486)
          	at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processDirectoryXml(OfflineImageReconstructor.java:676)
          	at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processINodeXml(OfflineImageReconstructor.java:555)
          	at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.access$1300(OfflineImageReconstructor.java:95)
          	at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$INodeSectionProcessor.process(OfflineImageReconstructor.java:526)
          	at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1658)
          	at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1713)
          	at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:191)
          

          Once I backport HADOOP-13508, this exception goes away. However, given that HADOOP-13508 is an incompatible change, I doubt people want to backport it just to fix this bug. So I propose FsPermission implements a similar helper method (which implements the semantics in trunk) and let OfflineImageReconstructor call that helper method.

          Attach a patch to reproduce the bug.

          Show
          jojochuang Wei-Chiu Chuang added a comment - For the context, OIV is a tool that dumps fsimage into a human-readable file, such as XML. OIV ReverseXML is a tool that imports an XML file into a fsimage. So ideally, you can do fsimage --> XML --> fsimage, and the sourcee fsimage is exactly the same as the destination fsimage. However, in branch-2 if the fsimage contains directories with sticky bits, it fails with the following exception: Invalid processor specified : invalid OfflineImageReconstructor failed: 1777 java.lang.IllegalArgumentException: 1777 at org.apache.hadoop.fs.permission.PermissionParser.<init>(PermissionParser.java:60) at org.apache.hadoop.fs.permission.UmaskParser.<init>(UmaskParser.java:42) at org.apache.hadoop.fs.permission.FsPermission.<init>(FsPermission.java:106) at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.permissionXmlToU64(OfflineImageReconstructor.java:1486) at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processDirectoryXml(OfflineImageReconstructor.java:676) at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processINodeXml(OfflineImageReconstructor.java:555) at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.access$1300(OfflineImageReconstructor.java:95) at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$INodeSectionProcessor.process(OfflineImageReconstructor.java:526) at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1658) at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1713) at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:191) Once I backport HADOOP-13508 , this exception goes away. However, given that HADOOP-13508 is an incompatible change, I doubt people want to backport it just to fix this bug. So I propose FsPermission implements a similar helper method (which implements the semantics in trunk) and let OfflineImageReconstructor call that helper method. Attach a patch to reproduce the bug.
          Hide
          chris.douglas Chris Douglas added a comment -

          Ah, got it. I thought HADOOP-13508 had caused a regression.

          AFAIK there's no behavior that depends on FsPermission ignoring the sticky bit. The semantics in branch-2 are incorrect, HADOOP-13508 isn't marked as incompatible... backporting seems OK to me.

          Show
          chris.douglas Chris Douglas added a comment - Ah, got it. I thought HADOOP-13508 had caused a regression. AFAIK there's no behavior that depends on FsPermission ignoring the sticky bit. The semantics in branch-2 are incorrect, HADOOP-13508 isn't marked as incompatible... backporting seems OK to me.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Cool. Thanks Chris Douglas. I'll proceed with the backport.

          Show
          jojochuang Wei-Chiu Chuang added a comment - Cool. Thanks Chris Douglas . I'll proceed with the backport.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Now that HADOOP-13508 is in branch-2, I am going to submit this patch for precommit check. We can use this as a regression test.

          Show
          jojochuang Wei-Chiu Chuang added a comment - Now that HADOOP-13508 is in branch-2, I am going to submit this patch for precommit check. We can use this as a regression test.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 13m 45s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 31s branch-2 passed
          +1 compile 0m 39s branch-2 passed with JDK v1.8.0_111
          +1 compile 0m 43s branch-2 passed with JDK v1.7.0_121
          +1 checkstyle 0m 26s branch-2 passed
          +1 mvnsite 0m 52s branch-2 passed
          +1 mvneclipse 0m 15s branch-2 passed
          +1 findbugs 1m 55s branch-2 passed
          +1 javadoc 0m 54s branch-2 passed with JDK v1.8.0_111
          +1 javadoc 1m 34s branch-2 passed with JDK v1.7.0_121
          +1 mvninstall 0m 43s the patch passed
          +1 compile 0m 36s the patch passed with JDK v1.8.0_111
          +1 javac 0m 36s the patch passed
          +1 compile 0m 39s the patch passed with JDK v1.7.0_121
          +1 javac 0m 39s the patch passed
          -0 checkstyle 0m 24s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 28 unchanged - 0 fixed = 29 total (was 28)
          +1 mvnsite 0m 48s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 6s the patch passed
          +1 javadoc 0m 52s the patch passed with JDK v1.8.0_111
          +1 javadoc 1m 33s the patch passed with JDK v1.7.0_121
          -1 unit 50m 26s hadoop-hdfs in the patch failed with JDK v1.7.0_121.
          -1 asflicense 0m 20s The patch generated 1 ASF License warnings.
          144m 47s



          Reason Tests
          JDK v1.8.0_111 Failed junit tests hadoop.hdfs.server.datanode.checker.TestThrottledAsyncChecker
            hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer
          JDK v1.8.0_111 Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2
            org.apache.hadoop.hdfs.TestQuota
          JDK v1.7.0_121 Failed junit tests hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:b59b8b7
          JIRA Issue HDFS-11084
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12843049/HDFS-11804.branch-2.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 79aa268feb73 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / 84ec079
          Default Java 1.7.0_121
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_111 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17872/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17872/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_121.txt
          JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17872/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/17872/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17872/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 13m 45s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 31s branch-2 passed +1 compile 0m 39s branch-2 passed with JDK v1.8.0_111 +1 compile 0m 43s branch-2 passed with JDK v1.7.0_121 +1 checkstyle 0m 26s branch-2 passed +1 mvnsite 0m 52s branch-2 passed +1 mvneclipse 0m 15s branch-2 passed +1 findbugs 1m 55s branch-2 passed +1 javadoc 0m 54s branch-2 passed with JDK v1.8.0_111 +1 javadoc 1m 34s branch-2 passed with JDK v1.7.0_121 +1 mvninstall 0m 43s the patch passed +1 compile 0m 36s the patch passed with JDK v1.8.0_111 +1 javac 0m 36s the patch passed +1 compile 0m 39s the patch passed with JDK v1.7.0_121 +1 javac 0m 39s the patch passed -0 checkstyle 0m 24s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 28 unchanged - 0 fixed = 29 total (was 28) +1 mvnsite 0m 48s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 6s the patch passed +1 javadoc 0m 52s the patch passed with JDK v1.8.0_111 +1 javadoc 1m 33s the patch passed with JDK v1.7.0_121 -1 unit 50m 26s hadoop-hdfs in the patch failed with JDK v1.7.0_121. -1 asflicense 0m 20s The patch generated 1 ASF License warnings. 144m 47s Reason Tests JDK v1.8.0_111 Failed junit tests hadoop.hdfs.server.datanode.checker.TestThrottledAsyncChecker   hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer JDK v1.8.0_111 Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2   org.apache.hadoop.hdfs.TestQuota JDK v1.7.0_121 Failed junit tests hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Issue HDFS-11084 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12843049/HDFS-11804.branch-2.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 79aa268feb73 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 84ec079 Default Java 1.7.0_121 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_111 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17872/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17872/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_121.txt JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17872/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/17872/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17872/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          ajisakaa Akira Ajisaka added a comment - - edited

          Some comments from me:

          • Would you rename the new directory from /tmp to /stickybit or something to specify that the directory has sticky bit?
          • Would you add a entry for the directory to writtenFiles to pass the regression test?

          I'm thinking HADOOP-13508 should be backported to branch-2.8 as well. What do you think, Wei-Chiu Chuang?

          Show
          ajisakaa Akira Ajisaka added a comment - - edited Some comments from me: Would you rename the new directory from /tmp to /stickybit or something to specify that the directory has sticky bit? Would you add a entry for the directory to writtenFiles to pass the regression test? I'm thinking HADOOP-13508 should be backported to branch-2.8 as well. What do you think, Wei-Chiu Chuang ?
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Thanks Akira Ajisaka.
          Good idea use a directory name other than /tmp. In practice /tmp is often associated with sticky bit though.

          Attached a second patch.

          I think the patch can go into trunk as well. HADOOP-13508 can go into branch-2.8 for release in 2.8.1.

          Show
          jojochuang Wei-Chiu Chuang added a comment - Thanks Akira Ajisaka . Good idea use a directory name other than /tmp. In practice /tmp is often associated with sticky bit though. Attached a second patch. I think the patch can go into trunk as well. HADOOP-13508 can go into branch-2.8 for release in 2.8.1.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 13m 37s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 49s branch-2 passed
          +1 compile 0m 40s branch-2 passed with JDK v1.8.0_121
          +1 compile 0m 43s branch-2 passed with JDK v1.7.0_121
          +1 checkstyle 0m 29s branch-2 passed
          +1 mvnsite 0m 51s branch-2 passed
          +1 mvneclipse 0m 15s branch-2 passed
          +1 findbugs 2m 0s branch-2 passed
          +1 javadoc 0m 54s branch-2 passed with JDK v1.8.0_121
          +1 javadoc 1m 42s branch-2 passed with JDK v1.7.0_121
          +1 mvninstall 0m 45s the patch passed
          +1 compile 0m 46s the patch passed with JDK v1.8.0_121
          +1 javac 0m 46s the patch passed
          +1 compile 0m 42s the patch passed with JDK v1.7.0_121
          +1 javac 0m 42s the patch passed
          +1 checkstyle 0m 25s the patch passed
          +1 mvnsite 0m 49s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 9s the patch passed
          +1 javadoc 0m 59s the patch passed with JDK v1.8.0_121
          +1 javadoc 1m 37s the patch passed with JDK v1.7.0_121
          -1 unit 55m 16s hadoop-hdfs in the patch failed with JDK v1.7.0_121.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          165m 23s



          Reason Tests
          JDK v1.8.0_121 Failed junit tests hadoop.hdfs.server.datanode.metrics.TestDataNodeOutlierDetectionViaMetrics
            hadoop.hdfs.server.datanode.TestDataNodeUUID
          JDK v1.8.0_121 Timed out junit tests org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
            org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean
            org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
          JDK v1.7.0_121 Failed junit tests hadoop.hdfs.server.datanode.metrics.TestDataNodeOutlierDetectionViaMetrics
            hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:b59b8b7
          JIRA Issue HDFS-11084
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12852159/HDFS-11804.branch-2.002.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux ada25d41cee3 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / 33c62d2
          Default Java 1.7.0_121
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_121 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/18351/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_121.txt
          JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18351/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18351/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 13m 37s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 49s branch-2 passed +1 compile 0m 40s branch-2 passed with JDK v1.8.0_121 +1 compile 0m 43s branch-2 passed with JDK v1.7.0_121 +1 checkstyle 0m 29s branch-2 passed +1 mvnsite 0m 51s branch-2 passed +1 mvneclipse 0m 15s branch-2 passed +1 findbugs 2m 0s branch-2 passed +1 javadoc 0m 54s branch-2 passed with JDK v1.8.0_121 +1 javadoc 1m 42s branch-2 passed with JDK v1.7.0_121 +1 mvninstall 0m 45s the patch passed +1 compile 0m 46s the patch passed with JDK v1.8.0_121 +1 javac 0m 46s the patch passed +1 compile 0m 42s the patch passed with JDK v1.7.0_121 +1 javac 0m 42s the patch passed +1 checkstyle 0m 25s the patch passed +1 mvnsite 0m 49s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 9s the patch passed +1 javadoc 0m 59s the patch passed with JDK v1.8.0_121 +1 javadoc 1m 37s the patch passed with JDK v1.7.0_121 -1 unit 55m 16s hadoop-hdfs in the patch failed with JDK v1.7.0_121. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 165m 23s Reason Tests JDK v1.8.0_121 Failed junit tests hadoop.hdfs.server.datanode.metrics.TestDataNodeOutlierDetectionViaMetrics   hadoop.hdfs.server.datanode.TestDataNodeUUID JDK v1.8.0_121 Timed out junit tests org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean   org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure JDK v1.7.0_121 Failed junit tests hadoop.hdfs.server.datanode.metrics.TestDataNodeOutlierDetectionViaMetrics   hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Issue HDFS-11084 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12852159/HDFS-11804.branch-2.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux ada25d41cee3 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 33c62d2 Default Java 1.7.0_121 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_121 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/18351/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_121.txt JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18351/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18351/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          The failed tests are unrelated and can't be reproduced.

          Show
          jojochuang Wei-Chiu Chuang added a comment - The failed tests are unrelated and can't be reproduced.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Akira Ajisaka I pushed HADOOP-13508 into branch-2.8. Thanks.

          Show
          jojochuang Wei-Chiu Chuang added a comment - Akira Ajisaka I pushed HADOOP-13508 into branch-2.8. Thanks.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Attaching the same patch to run precommit job on trunk.

          Show
          ajisakaa Akira Ajisaka added a comment - Attaching the same patch to run precommit job on trunk.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          +1 pending Jenkins on trunk. Thanks Wei-Chiu Chuang for updating the patch.

          Show
          ajisakaa Akira Ajisaka added a comment - +1 pending Jenkins on trunk. Thanks Wei-Chiu Chuang for updating the patch.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Hi Wei-Chiu Chuang, would you rename the summary to "Add a regression test for OIV ReverseXML processor" or something since the patch is to add the test case?

          Show
          ajisakaa Akira Ajisaka added a comment - Hi Wei-Chiu Chuang , would you rename the summary to "Add a regression test for OIV ReverseXML processor" or something since the patch is to add the test case?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 21s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 13m 2s trunk passed
          +1 compile 0m 54s trunk passed
          +1 checkstyle 0m 28s trunk passed
          +1 mvnsite 1m 3s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 2m 1s trunk passed
          +1 javadoc 0m 43s trunk passed
          +1 mvninstall 0m 48s the patch passed
          +1 compile 0m 44s the patch passed
          +1 javac 0m 44s the patch passed
          +1 checkstyle 0m 22s the patch passed
          +1 mvnsite 0m 51s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 51s the patch passed
          +1 javadoc 0m 37s the patch passed
          -1 unit 110m 42s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 25s The patch does not generate ASF License warnings.
          136m 35s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
            hadoop.hdfs.tools.TestDFSAdminWithHA
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-11084
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12852491/HDFS-11084.02.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 154b22ee0c96 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 71c23c9
          Default Java 1.8.0_121
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/18364/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18364/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18364/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 13m 2s trunk passed +1 compile 0m 54s trunk passed +1 checkstyle 0m 28s trunk passed +1 mvnsite 1m 3s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 2m 1s trunk passed +1 javadoc 0m 43s trunk passed +1 mvninstall 0m 48s the patch passed +1 compile 0m 44s the patch passed +1 javac 0m 44s the patch passed +1 checkstyle 0m 22s the patch passed +1 mvnsite 0m 51s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 51s the patch passed +1 javadoc 0m 37s the patch passed -1 unit 110m 42s hadoop-hdfs in the patch failed. +1 asflicense 0m 25s The patch does not generate ASF License warnings. 136m 35s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.tools.TestDFSAdminWithHA   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11084 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12852491/HDFS-11084.02.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 154b22ee0c96 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 71c23c9 Default Java 1.8.0_121 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/18364/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18364/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18364/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          The patch only updates tests for OIV, so failed tests are unrelated.

          Show
          jojochuang Wei-Chiu Chuang added a comment - The patch only updates tests for OIV, so failed tests are unrelated.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          +1

          Show
          ajisakaa Akira Ajisaka added a comment - +1
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Committed the patch to trunk, branch-2 and branch-2.8.
          Many thanks to Akira Ajisaka for the review and Chris Douglas for comments regarding the correct behavior!

          Show
          jojochuang Wei-Chiu Chuang added a comment - Committed the patch to trunk, branch-2 and branch-2.8. Many thanks to Akira Ajisaka for the review and Chris Douglas for comments regarding the correct behavior!
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11246 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11246/)
          HDFS-11084. Add a regression test for sticky bit support of OIV (weichiu: rev 0cf5993712a01993bd701bd9664e6af284378b55)

          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/TestOfflineImageViewer.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11246 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11246/ ) HDFS-11084 . Add a regression test for sticky bit support of OIV (weichiu: rev 0cf5993712a01993bd701bd9664e6af284378b55) (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/TestOfflineImageViewer.java
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          2.8.1 became a security release. Moving fix-version to 2.8.2 after the fact.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - 2.8.1 became a security release. Moving fix-version to 2.8.2 after the fact.

            People

            • Assignee:
              jojochuang Wei-Chiu Chuang
              Reporter:
              jojochuang Wei-Chiu Chuang
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development