Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-10506

OIV's ReverseXML processor cannot reconstruct some snapshot details

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.8.0
    • Fix Version/s: 2.9.0, 3.0.0-alpha4, 2.8.2
    • Component/s: tools
    • Labels:
      None
    • Target Version/s:

      Description

      OIV's ReverseXML processor cannot reconstruct some snapshot details. Specifically, <fileDiff> should contain a <snapshotCopy> and <blocks> field, but does not. <dirDiff> should contain a <snapshotCopy> field. OIV also needs to be changed to emit these fields into the XML (they are currently missing).

      1. HDFS-10506.01.patch
        11 kB
        Akira Ajisaka
      2. HDFS-10506.02.patch
        11 kB
        Akira Ajisaka
      3. HDFS-10506.03.patch
        11 kB
        Akira Ajisaka
      4. HDFS-10506.04.patch
        12 kB
        Akira Ajisaka
      5. HDFS-10506-addendum.patch
        3 kB
        Akira Ajisaka
      6. HDFS-10506-branch-2.01.patch
        12 kB
        Akira Ajisaka

        Issue Links

          Activity

          Hide
          cmccabe Colin P. McCabe added a comment -

          From hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/OfflineImageReconstructor.java:

              private void processDirDiffEntry() throws IOException {
                LOG.debug("Processing dirDiffEntry");
          ...
                  // TODO: add missing snapshotCopy field to XML
          
              private void processFileDiffEntry() throws IOException {
                LOG.debug("Processing fileDiffEntry");
          ...
                  // TODO: missing snapshotCopy
                  // TODO: missing blocks
                  fileDiff.verifyNoRemainingKeys("fileDiff");
                  bld.build().writeDelimitedTo(out);
                }
                expectTagEnd(SNAPSHOT_DIFF_SECTION_FILE_DIFF_ENTRY);
              }
          
          Show
          cmccabe Colin P. McCabe added a comment - From hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/OfflineImageReconstructor.java : private void processDirDiffEntry() throws IOException { LOG.debug( "Processing dirDiffEntry" ); ... // TODO: add missing snapshotCopy field to XML private void processFileDiffEntry() throws IOException { LOG.debug( "Processing fileDiffEntry" ); ... // TODO: missing snapshotCopy // TODO: missing blocks fileDiff.verifyNoRemainingKeys( "fileDiff" ); bld.build().writeDelimitedTo(out); } expectTagEnd(SNAPSHOT_DIFF_SECTION_FILE_DIFF_ENTRY); }
          Hide
          ajisakaa Akira Ajisaka added a comment -

          01 patch

          • The output of OIV XML processor now contains the missing fields.
          • ReverseXML processor can reconstruct the fields.
          • Added the fields to the fsimage in TestOfflineImageViewer.
          Show
          ajisakaa Akira Ajisaka added a comment - 01 patch The output of OIV XML processor now contains the missing fields. ReverseXML processor can reconstruct the fields. Added the fields to the fsimage in TestOfflineImageViewer.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 26s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 18m 47s trunk passed
          +1 compile 1m 9s trunk passed
          +1 checkstyle 0m 42s trunk passed
          +1 mvnsite 1m 19s trunk passed
          +1 mvneclipse 0m 17s trunk passed
          +1 findbugs 2m 5s trunk passed
          +1 javadoc 0m 48s trunk passed
          +1 mvninstall 1m 1s the patch passed
          +1 compile 0m 53s the patch passed
          +1 javac 0m 53s the patch passed
          +1 checkstyle 0m 24s the patch passed
          +1 mvnsite 0m 48s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 47s the patch passed
          +1 javadoc 0m 37s the patch passed
          -1 unit 85m 24s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          118m 13s



          Reason Tests
          Failed junit tests hadoop.hdfs.TestMaintenanceState



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-10506
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12847590/HDFS-10506.01.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux dfbf97f811ee 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 2f8e9b7
          Default Java 1.8.0_111
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/18180/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18180/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18180/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 26s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 18m 47s trunk passed +1 compile 1m 9s trunk passed +1 checkstyle 0m 42s trunk passed +1 mvnsite 1m 19s trunk passed +1 mvneclipse 0m 17s trunk passed +1 findbugs 2m 5s trunk passed +1 javadoc 0m 48s trunk passed +1 mvninstall 1m 1s the patch passed +1 compile 0m 53s the patch passed +1 javac 0m 53s the patch passed +1 checkstyle 0m 24s the patch passed +1 mvnsite 0m 48s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 47s the patch passed +1 javadoc 0m 37s the patch passed -1 unit 85m 24s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 118m 13s Reason Tests Failed junit tests hadoop.hdfs.TestMaintenanceState Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-10506 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12847590/HDFS-10506.01.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux dfbf97f811ee 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 2f8e9b7 Default Java 1.8.0_111 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/18180/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18180/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18180/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          02 patch

          • Added comments in the test.
          Show
          ajisakaa Akira Ajisaka added a comment - 02 patch Added comments in the test.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 22s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 15m 19s trunk passed
          +1 compile 0m 47s trunk passed
          +1 checkstyle 0m 28s trunk passed
          +1 mvnsite 0m 55s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 1m 48s trunk passed
          +1 javadoc 0m 40s trunk passed
          +1 mvninstall 0m 50s the patch passed
          +1 compile 0m 47s the patch passed
          +1 javac 0m 47s the patch passed
          +1 checkstyle 0m 26s the patch passed
          +1 mvnsite 0m 53s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 52s the patch passed
          +1 javadoc 0m 38s the patch passed
          -1 unit 111m 26s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 23s The patch does not generate ASF License warnings.
          139m 19s



          Reason Tests
          Failed junit tests hadoop.hdfs.TestRollingUpgrade



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-10506
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12848023/HDFS-10506.02.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 6206c40296d6 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 9130af3
          Default Java 1.8.0_111
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/18194/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18194/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18194/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 22s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 15m 19s trunk passed +1 compile 0m 47s trunk passed +1 checkstyle 0m 28s trunk passed +1 mvnsite 0m 55s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 48s trunk passed +1 javadoc 0m 40s trunk passed +1 mvninstall 0m 50s the patch passed +1 compile 0m 47s the patch passed +1 javac 0m 47s the patch passed +1 checkstyle 0m 26s the patch passed +1 mvnsite 0m 53s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 52s the patch passed +1 javadoc 0m 38s the patch passed -1 unit 111m 26s hadoop-hdfs in the patch failed. +1 asflicense 0m 23s The patch does not generate ASF License warnings. 139m 19s Reason Tests Failed junit tests hadoop.hdfs.TestRollingUpgrade Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-10506 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12848023/HDFS-10506.02.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 6206c40296d6 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 9130af3 Default Java 1.8.0_111 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/18194/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18194/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18194/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Hi Colin P. McCabe and Wei-Chiu Chuang, would you review this?

          Show
          ajisakaa Akira Ajisaka added a comment - Hi Colin P. McCabe and Wei-Chiu Chuang , would you review this?
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Rebased.

          Show
          ajisakaa Akira Ajisaka added a comment - Rebased.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 27s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 17m 20s trunk passed
          +1 compile 1m 5s trunk passed
          +1 checkstyle 0m 47s trunk passed
          +1 mvnsite 1m 8s trunk passed
          +1 mvneclipse 0m 17s trunk passed
          +1 findbugs 2m 9s trunk passed
          +1 javadoc 0m 49s trunk passed
          +1 mvninstall 1m 2s the patch passed
          +1 compile 0m 56s the patch passed
          +1 javac 0m 56s the patch passed
          +1 checkstyle 0m 39s the patch passed
          +1 mvnsite 1m 5s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 17s the patch passed
          +1 javadoc 0m 45s the patch passed
          +1 unit 96m 55s hadoop-hdfs in the patch passed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          129m 50s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-10506
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12852953/HDFS-10506.03.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux d037e9ef6a16 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 0741dd3
          Default Java 1.8.0_121
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18385/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18385/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 27s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 17m 20s trunk passed +1 compile 1m 5s trunk passed +1 checkstyle 0m 47s trunk passed +1 mvnsite 1m 8s trunk passed +1 mvneclipse 0m 17s trunk passed +1 findbugs 2m 9s trunk passed +1 javadoc 0m 49s trunk passed +1 mvninstall 1m 2s the patch passed +1 compile 0m 56s the patch passed +1 javac 0m 56s the patch passed +1 checkstyle 0m 39s the patch passed +1 mvnsite 1m 5s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 17s the patch passed +1 javadoc 0m 45s the patch passed +1 unit 96m 55s hadoop-hdfs in the patch passed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 129m 50s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-10506 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12852953/HDFS-10506.03.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux d037e9ef6a16 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0741dd3 Default Java 1.8.0_121 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18385/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18385/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          surendrasingh Surendra Singh Lilhore added a comment -

          Reviewed and tested the patch.. Looks good to me.

          Show
          surendrasingh Surendra Singh Lilhore added a comment - Reviewed and tested the patch.. Looks good to me.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          I'll review it today.

          Show
          jojochuang Wei-Chiu Chuang added a comment - I'll review it today.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Akira Ajisaka thanks for the patch.
          One nit: could you rename PBImageXmlWriter.INODE_SECTION_GEMSTAMP to INODE_SECTION_GE"N"STAMP?

          Other than that +1.

          Show
          jojochuang Wei-Chiu Chuang added a comment - Akira Ajisaka thanks for the patch. One nit: could you rename PBImageXmlWriter.INODE_SECTION_GEMSTAMP to INODE_SECTION_GE"N"STAMP? Other than that +1.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thank you for the review, Wei-Chiu Chuang. Renamed the variable.

          Show
          ajisakaa Akira Ajisaka added a comment - Thank you for the review, Wei-Chiu Chuang . Renamed the variable.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 12m 35s trunk passed
          +1 compile 0m 47s trunk passed
          +1 checkstyle 0m 34s trunk passed
          +1 mvnsite 0m 51s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 41s trunk passed
          +1 javadoc 0m 39s trunk passed
          +1 mvninstall 0m 45s the patch passed
          +1 compile 0m 42s the patch passed
          +1 javac 0m 42s the patch passed
          +1 checkstyle 0m 32s the patch passed
          +1 mvnsite 0m 50s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 7s the patch passed
          +1 javadoc 0m 43s the patch passed
          -1 unit 109m 48s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 25s The patch does not generate ASF License warnings.
          135m 9s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
            hadoop.hdfs.TestFileCorruption
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-10506
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12854513/HDFS-10506.04.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 1537d7aaa6d5 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / b32ffa2
          Default Java 1.8.0_121
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/18438/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18438/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18438/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 12m 35s trunk passed +1 compile 0m 47s trunk passed +1 checkstyle 0m 34s trunk passed +1 mvnsite 0m 51s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 41s trunk passed +1 javadoc 0m 39s trunk passed +1 mvninstall 0m 45s the patch passed +1 compile 0m 42s the patch passed +1 javac 0m 42s the patch passed +1 checkstyle 0m 32s the patch passed +1 mvnsite 0m 50s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 7s the patch passed +1 javadoc 0m 43s the patch passed -1 unit 109m 48s hadoop-hdfs in the patch failed. +1 asflicense 0m 25s The patch does not generate ASF License warnings. 135m 9s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.TestFileCorruption   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-10506 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12854513/HDFS-10506.04.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 1537d7aaa6d5 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / b32ffa2 Default Java 1.8.0_121 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/18438/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18438/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18438/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          +1. Failed tests are unrelated.

          Show
          jojochuang Wei-Chiu Chuang added a comment - +1. Failed tests are unrelated.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Pushed the commit to trunk.
          Akira Ajisaka there's a conflict cherry picking the commit to branch-2. Would you please submit a branch-2 patch after resolving conflicts?

          Show
          jojochuang Wei-Chiu Chuang added a comment - Pushed the commit to trunk. Akira Ajisaka there's a conflict cherry picking the commit to branch-2. Would you please submit a branch-2 patch after resolving conflicts?
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11309 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11309/)
          HDFS-10506. OIV's ReverseXML processor cannot reconstruct some snapshot (weichiu: rev 05391c1845639d4f01da8e5df966e2dc2682f2ca)

          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/PBImageXmlWriter.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/TestOfflineImageViewer.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/OfflineImageReconstructor.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11309 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11309/ ) HDFS-10506 . OIV's ReverseXML processor cannot reconstruct some snapshot (weichiu: rev 05391c1845639d4f01da8e5df966e2dc2682f2ca) (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/PBImageXmlWriter.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/TestOfflineImageViewer.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/OfflineImageReconstructor.java
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Rebased for branch-2. Thanks Surendra Singh Lilhore and Wei-Chiu Chuang for review.

          Show
          ajisakaa Akira Ajisaka added a comment - Rebased for branch-2. Thanks Surendra Singh Lilhore and Wei-Chiu Chuang for review.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 44s branch-2 passed
          +1 compile 0m 38s branch-2 passed with JDK v1.8.0_121
          +1 compile 0m 44s branch-2 passed with JDK v1.7.0_121
          +1 checkstyle 0m 28s branch-2 passed
          +1 mvnsite 0m 52s branch-2 passed
          +1 mvneclipse 0m 16s branch-2 passed
          +1 findbugs 1m 56s branch-2 passed
          +1 javadoc 0m 54s branch-2 passed with JDK v1.8.0_121
          +1 javadoc 1m 35s branch-2 passed with JDK v1.7.0_121
          +1 mvninstall 0m 43s the patch passed
          +1 compile 0m 37s the patch passed with JDK v1.8.0_121
          +1 javac 0m 37s the patch passed
          +1 compile 0m 40s the patch passed with JDK v1.7.0_121
          +1 javac 0m 40s the patch passed
          +1 checkstyle 0m 25s the patch passed
          +1 mvnsite 0m 49s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 7s the patch passed
          +1 javadoc 0m 51s the patch passed with JDK v1.8.0_121
          +1 javadoc 1m 31s the patch passed with JDK v1.7.0_121
          -1 unit 67m 50s hadoop-hdfs in the patch failed with JDK v1.7.0_121.
          +1 asflicense 0m 19s The patch does not generate ASF License warnings.
          166m 31s



          Reason Tests
          JDK v1.8.0_121 Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2
            org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
          JDK v1.7.0_121 Failed junit tests hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery
            hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:b59b8b7
          JIRA Issue HDFS-10506
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12854819/HDFS-10506-branch-2.01.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 448d31577a7b 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / c24325b
          Default Java 1.7.0_121
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_121 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/18451/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_121.txt
          JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18451/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18451/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 44s branch-2 passed +1 compile 0m 38s branch-2 passed with JDK v1.8.0_121 +1 compile 0m 44s branch-2 passed with JDK v1.7.0_121 +1 checkstyle 0m 28s branch-2 passed +1 mvnsite 0m 52s branch-2 passed +1 mvneclipse 0m 16s branch-2 passed +1 findbugs 1m 56s branch-2 passed +1 javadoc 0m 54s branch-2 passed with JDK v1.8.0_121 +1 javadoc 1m 35s branch-2 passed with JDK v1.7.0_121 +1 mvninstall 0m 43s the patch passed +1 compile 0m 37s the patch passed with JDK v1.8.0_121 +1 javac 0m 37s the patch passed +1 compile 0m 40s the patch passed with JDK v1.7.0_121 +1 javac 0m 40s the patch passed +1 checkstyle 0m 25s the patch passed +1 mvnsite 0m 49s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 7s the patch passed +1 javadoc 0m 51s the patch passed with JDK v1.8.0_121 +1 javadoc 1m 31s the patch passed with JDK v1.7.0_121 -1 unit 67m 50s hadoop-hdfs in the patch failed with JDK v1.7.0_121. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 166m 31s Reason Tests JDK v1.8.0_121 Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2   org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting JDK v1.7.0_121 Failed junit tests hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery   hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Issue HDFS-10506 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12854819/HDFS-10506-branch-2.01.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 448d31577a7b 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / c24325b Default Java 1.7.0_121 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_121 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/18451/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_121.txt JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18451/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18451/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Hi Wei-Chiu Chuang, would you review the patch for branch-2?

          Show
          ajisakaa Akira Ajisaka added a comment - Hi Wei-Chiu Chuang , would you review the patch for branch-2?
          Hide
          ajisakaa Akira Ajisaka added a comment -
          Show
          ajisakaa Akira Ajisaka added a comment - Ping Wei-Chiu Chuang
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Thanks Akira Ajisaka for the backport. My bad. I'll take a look now.

          Show
          jojochuang Wei-Chiu Chuang added a comment - Thanks Akira Ajisaka for the backport. My bad. I'll take a look now.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Done. For the record, I just realized I didn't push v04 patch to trunk. Instead, I probably pushed in v03 patch. The difference between v03 and v04 is just cosmetics (one extra line of comment and changing variable name).

          Show
          jojochuang Wei-Chiu Chuang added a comment - Done. For the record, I just realized I didn't push v04 patch to trunk. Instead, I probably pushed in v03 patch. The difference between v03 and v04 is just cosmetics (one extra line of comment and changing variable name).
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Hi Wei-Chiu Chuang. Thank you for the information. Attaching a patch for trunk to rename the variable. Would you review the patch and commit it?

          Show
          ajisakaa Akira Ajisaka added a comment - Hi Wei-Chiu Chuang . Thank you for the information. Attaching a patch for trunk to rename the variable. Would you review the patch and commit it?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 12m 43s trunk passed
          +1 compile 0m 49s trunk passed
          +1 checkstyle 0m 37s trunk passed
          +1 mvnsite 1m 2s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 56s trunk passed
          +1 javadoc 0m 41s trunk passed
          +1 mvninstall 0m 53s the patch passed
          +1 compile 0m 52s the patch passed
          +1 javac 0m 52s the patch passed
          +1 checkstyle 0m 35s the patch passed
          +1 mvnsite 0m 57s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 1s The patch has no whitespace issues.
          +1 findbugs 1m 49s the patch passed
          +1 javadoc 0m 38s the patch passed
          -1 unit 68m 35s hadoop-hdfs in the patch failed.
          -1 asflicense 0m 21s The patch generated 3 ASF License warnings.
          94m 30s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
            hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-10506
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12860585/HDFS-10506-addendum.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux aea6a34b9acf 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / e7b7c96
          Default Java 1.8.0_121
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/18839/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18839/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/18839/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18839/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 12m 43s trunk passed +1 compile 0m 49s trunk passed +1 checkstyle 0m 37s trunk passed +1 mvnsite 1m 2s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 56s trunk passed +1 javadoc 0m 41s trunk passed +1 mvninstall 0m 53s the patch passed +1 compile 0m 52s the patch passed +1 javac 0m 52s the patch passed +1 checkstyle 0m 35s the patch passed +1 mvnsite 0m 57s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 1s The patch has no whitespace issues. +1 findbugs 1m 49s the patch passed +1 javadoc 0m 38s the patch passed -1 unit 68m 35s hadoop-hdfs in the patch failed. -1 asflicense 0m 21s The patch generated 3 ASF License warnings. 94m 30s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-10506 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12860585/HDFS-10506-addendum.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux aea6a34b9acf 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e7b7c96 Default Java 1.8.0_121 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/18839/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18839/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/18839/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18839/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          +1. Test failures are unrelated. The asf warning is fixed by MAPREDUCE-6868

          Show
          jojochuang Wei-Chiu Chuang added a comment - +1. Test failures are unrelated. The asf warning is fixed by MAPREDUCE-6868
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #11471 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11471/)
          HDFS-10506. Addendum patch to include missing changes. Contributed by (weichiu: rev 46d37a65cf09c2714b4c0c4ec0399031d60027a5)

          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/PBImageXmlWriter.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/OfflineImageReconstructor.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #11471 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11471/ ) HDFS-10506 . Addendum patch to include missing changes. Contributed by (weichiu: rev 46d37a65cf09c2714b4c0c4ec0399031d60027a5) (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/PBImageXmlWriter.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/OfflineImageReconstructor.java
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Pushed the addendum patch to trunk. Thanks again for Akira Ajisaka's contribution!

          Show
          jojochuang Wei-Chiu Chuang added a comment - Pushed the addendum patch to trunk. Thanks again for Akira Ajisaka 's contribution!
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Hi Andrew Wang and Wei-Chiu Chuang, I'd like to backport this issue to 2.8.1. Possible incompatible issue is that XML files generated by 2.8.1 XML processor cannot be parsed by 2.8.0 ReverseXML processor, however, I'm thinking the situation is not likely happen. What do you think?

          Show
          ajisakaa Akira Ajisaka added a comment - Hi Andrew Wang and Wei-Chiu Chuang , I'd like to backport this issue to 2.8.1. Possible incompatible issue is that XML files generated by 2.8.1 XML processor cannot be parsed by 2.8.0 ReverseXML processor, however, I'm thinking the situation is not likely happen. What do you think?
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          I am okay with backporting this fix to 2.8.1.
          Otherwise, it could generate incorrect fsimage, which is bad bad bad.

          Thanks!

          Show
          jojochuang Wei-Chiu Chuang added a comment - I am okay with backporting this fix to 2.8.1. Otherwise, it could generate incorrect fsimage, which is bad bad bad. Thanks!
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thanks! Backported to branch-2.8 and branch-2.8.1.

          Show
          ajisakaa Akira Ajisaka added a comment - Thanks! Backported to branch-2.8 and branch-2.8.1.
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          2.8.1 became a security release. Moving fix-version to 2.8.2 after the fact.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - 2.8.1 became a security release. Moving fix-version to 2.8.2 after the fact.

            People

            • Assignee:
              ajisakaa Akira Ajisaka
              Reporter:
              cmccabe Colin P. McCabe
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development