Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9721

Allow Delimited PB OIV tool to run upon fsimage that contains INodeReference

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      HDFS-6673 added the feature of Delimited format OIV tool on protocol buffer based fsimage.
      However, if the fsimage contains INodeReference, the tool fails because:

      Preconditions.checkState(e.getRefChildrenCount() == 0);

      This jira is to propose allow the tool to finish, so that user can get full metadata.

      1. HDFS-9721.01.patch
        10 kB
        Xiao Chen
      2. HDFS-9721.02.patch
        12 kB
        Xiao Chen
      3. HDFS-9721.03.patch
        13 kB
        Xiao Chen
      4. HDFS-9721.04.patch
        15 kB
        Xiao Chen
      5. HDFS-9721.05.patch
        18 kB
        Xiao Chen

        Issue Links

          Activity

          Hide
          xiaochen Xiao Chen added a comment -

          I studied the legacy_oiv Delimited tool, XML format OIV tool, and have decided to just allow the Delimited OIV tool to finish loading normal metadata, without worrying about snapshots.
          The legacy_oiv tool seems to print out both the normal namespace and the all found paths for snapshots all together, plus the snapshot name itself... IMO this is more confusing than not printing at all. If snapshot info is needed, one can easily get it from XML OIV tool.

          Attached patch 1.

          • Load the INodeReference section
          • Store and lookup needed mappings along the INodeReference line
          • Ignore snapshots
          • Fixed a minor bug that writer's close didn't flush the stream.

          Below is an example of the legacy OIV, and the Delimited OIV in patch 1:
          Metadata was simply constructed with:

          bin/hdfs dfs -mkdir /src
          bin/hdfs dfs -touchz /src/file
          bin/hdfs dfsadmin -allowSnapshot /src
          bin/hdfs dfs -createSnapshot /src snap
          
          Legacy OIV output:
          drwxr-xr-x  -     xiao supergroup          0 2016-01-28 10:37 /snap
          drwxr-xr-x  -     xiao supergroup          0 2016-01-28 10:36 /
          drwxr-xr-x  -     xiao supergroup          0 2016-01-28 10:37 /src
          -rw-r--r--  1     xiao supergroup          0 2016-01-28 10:36 /src/file
          
          Patch 1 oiv -p Delimited output:
          /	0	2016-01-28 10:36	1969-12-31 16:00	0	0	0	9223372036854775807	-1	rwxr-xr-x	xiao	supergroup
          /src	0	2016-01-28 10:37	1969-12-31 16:00	0	0	0	-1	-1	rwxr-xr-x	xiao	supergroup
          /src/file	1	2016-01-28 10:36	2016-01-28 10:36	134217728	0	0	0	0	rw-r--r--	xiao	supergroup
          

          A (a little bit) more complex case that involves INodeReference is included in the unit test.

          Show
          xiaochen Xiao Chen added a comment - I studied the legacy_oiv Delimited tool, XML format OIV tool, and have decided to just allow the Delimited OIV tool to finish loading normal metadata, without worrying about snapshots. The legacy_oiv tool seems to print out both the normal namespace and the all found paths for snapshots all together, plus the snapshot name itself... IMO this is more confusing than not printing at all. If snapshot info is needed, one can easily get it from XML OIV tool. Attached patch 1. Load the INodeReference section Store and lookup needed mappings along the INodeReference line Ignore snapshots Fixed a minor bug that writer's close didn't flush the stream. Below is an example of the legacy OIV, and the Delimited OIV in patch 1: Metadata was simply constructed with: bin/hdfs dfs -mkdir /src bin/hdfs dfs -touchz /src/file bin/hdfs dfsadmin -allowSnapshot /src bin/hdfs dfs -createSnapshot /src snap Legacy OIV output: drwxr-xr-x - xiao supergroup 0 2016-01-28 10:37 /snap drwxr-xr-x - xiao supergroup 0 2016-01-28 10:36 / drwxr-xr-x - xiao supergroup 0 2016-01-28 10:37 /src -rw-r--r-- 1 xiao supergroup 0 2016-01-28 10:36 /src/file Patch 1 oiv -p Delimited output: / 0 2016-01-28 10:36 1969-12-31 16:00 0 0 0 9223372036854775807 -1 rwxr-xr-x xiao supergroup /src 0 2016-01-28 10:37 1969-12-31 16:00 0 0 0 -1 -1 rwxr-xr-x xiao supergroup /src/file 1 2016-01-28 10:36 2016-01-28 10:36 134217728 0 0 0 0 rw-r--r-- xiao supergroup A (a little bit) more complex case that involves INodeReference is included in the unit test.
          Hide
          xiaochen Xiao Chen added a comment -

          I think at least we need to separate the snapshots and regular paths in the listing. If there's requirement for that, I'd prefer to address it in a separate JIRA. Since the XML OIV tool clearly lists snapshots along with INodes / INodeReference, I doubt we really need the Delimit tool to do the duplicate work.

          Show
          xiaochen Xiao Chen added a comment - I think at least we need to separate the snapshots and regular paths in the listing. If there's requirement for that, I'd prefer to address it in a separate JIRA. Since the XML OIV tool clearly lists snapshots along with INodes / INodeReference, I doubt we really need the Delimit tool to do the duplicate work.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 56s trunk passed
          +1 compile 1m 11s trunk passed with JDK v1.8.0_66
          +1 compile 0m 52s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 24s trunk passed
          +1 mvnsite 1m 4s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          +1 findbugs 1m 55s trunk passed
          +1 javadoc 1m 2s trunk passed with JDK v1.8.0_66
          +1 javadoc 1m 43s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 42s the patch passed
          +1 compile 0m 35s the patch passed with JDK v1.8.0_66
          +1 javac 0m 35s the patch passed
          +1 compile 0m 38s the patch passed with JDK v1.7.0_91
          +1 javac 0m 38s the patch passed
          +1 checkstyle 0m 18s the patch passed
          +1 mvnsite 0m 48s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          -1 findbugs 2m 13s hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
          +1 javadoc 1m 2s the patch passed with JDK v1.8.0_66
          +1 javadoc 1m 56s the patch passed with JDK v1.7.0_91
          -1 unit 57m 30s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          -1 unit 50m 8s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
          +1 asflicense 0m 21s Patch does not generate ASF License warnings.
          135m 26s



          Reason Tests
          FindBugs module:hadoop-hdfs-project/hadoop-hdfs
            Switch statement found in org.apache.hadoop.hdfs.tools.offlineImageViewer.PBImageTextWriter.visit(RandomAccessFile) where default case is missing At PBImageTextWriter.java:where default case is missing At PBImageTextWriter.java:[lines 467-476]
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.shortcircuit.TestShortCircuitCache
            hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer
          JDK v1.7.0_91 Failed junit tests hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12785080/HDFS-9721.01.patch
          JIRA Issue HDFS-9721
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 58b41aa3f89a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / ee005e0
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/14286/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14286/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14286/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14286/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14286/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14286/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Max memory used 77MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14286/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 56s trunk passed +1 compile 1m 11s trunk passed with JDK v1.8.0_66 +1 compile 0m 52s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 24s trunk passed +1 mvnsite 1m 4s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 1m 55s trunk passed +1 javadoc 1m 2s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 43s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 42s the patch passed +1 compile 0m 35s the patch passed with JDK v1.8.0_66 +1 javac 0m 35s the patch passed +1 compile 0m 38s the patch passed with JDK v1.7.0_91 +1 javac 0m 38s the patch passed +1 checkstyle 0m 18s the patch passed +1 mvnsite 0m 48s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. -1 findbugs 2m 13s hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) +1 javadoc 1m 2s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 56s the patch passed with JDK v1.7.0_91 -1 unit 57m 30s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 50m 8s hadoop-hdfs in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 21s Patch does not generate ASF License warnings. 135m 26s Reason Tests FindBugs module:hadoop-hdfs-project/hadoop-hdfs   Switch statement found in org.apache.hadoop.hdfs.tools.offlineImageViewer.PBImageTextWriter.visit(RandomAccessFile) where default case is missing At PBImageTextWriter.java:where default case is missing At PBImageTextWriter.java: [lines 467-476] JDK v1.8.0_66 Failed junit tests hadoop.hdfs.shortcircuit.TestShortCircuitCache   hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer JDK v1.7.0_91 Failed junit tests hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12785080/HDFS-9721.01.patch JIRA Issue HDFS-9721 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 58b41aa3f89a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ee005e0 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/14286/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/14286/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14286/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14286/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14286/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14286/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 77MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14286/console This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          Patch 2 to fix findbugs and test case.

          Show
          xiaochen Xiao Chen added a comment - Patch 2 to fix findbugs and test case.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 48s trunk passed
          +1 compile 0m 50s trunk passed with JDK v1.8.0_66
          +1 compile 0m 44s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 21s trunk passed
          +1 mvnsite 0m 52s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 58s trunk passed
          +1 javadoc 1m 9s trunk passed with JDK v1.8.0_66
          +1 javadoc 1m 55s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 44s the patch passed
          +1 compile 0m 40s the patch passed with JDK v1.8.0_66
          +1 javac 0m 40s the patch passed
          +1 compile 0m 40s the patch passed with JDK v1.7.0_91
          +1 javac 0m 40s the patch passed
          +1 checkstyle 0m 20s the patch passed
          +1 mvnsite 0m 50s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 9s the patch passed
          +1 javadoc 1m 5s the patch passed with JDK v1.8.0_66
          +1 javadoc 1m 48s the patch passed with JDK v1.7.0_91
          -1 unit 52m 43s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          -1 unit 48m 52s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
          +1 asflicense 0m 22s Patch does not generate ASF License warnings.
          127m 31s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes
            hadoop.hdfs.TestRecoverStripedFile
            hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency
            hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer
          JDK v1.7.0_91 Failed junit tests hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12785114/HDFS-9721.02.patch
          JIRA Issue HDFS-9721
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux d0333aae66fc 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / df99ea8
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14287/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14287/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14287/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14287/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14287/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Max memory used 77MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14287/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 48s trunk passed +1 compile 0m 50s trunk passed with JDK v1.8.0_66 +1 compile 0m 44s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 21s trunk passed +1 mvnsite 0m 52s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 58s trunk passed +1 javadoc 1m 9s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 55s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 44s the patch passed +1 compile 0m 40s the patch passed with JDK v1.8.0_66 +1 javac 0m 40s the patch passed +1 compile 0m 40s the patch passed with JDK v1.7.0_91 +1 javac 0m 40s the patch passed +1 checkstyle 0m 20s the patch passed +1 mvnsite 0m 50s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 9s the patch passed +1 javadoc 1m 5s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 48s the patch passed with JDK v1.7.0_91 -1 unit 52m 43s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 48m 52s hadoop-hdfs in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 22s Patch does not generate ASF License warnings. 127m 31s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes   hadoop.hdfs.TestRecoverStripedFile   hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency   hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer JDK v1.7.0_91 Failed junit tests hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12785114/HDFS-9721.02.patch JIRA Issue HDFS-9721 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux d0333aae66fc 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / df99ea8 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/14287/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14287/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14287/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14287/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14287/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 77MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14287/console This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          Missed 1 test case.

          Patch 3 fixed that, locally passed now. Also added some comments, no real code change.

          Show
          xiaochen Xiao Chen added a comment - Missed 1 test case. Patch 3 fixed that, locally passed now. Also added some comments, no real code change.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 1s trunk passed
          +1 compile 0m 41s trunk passed with JDK v1.8.0_66
          +1 compile 0m 41s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 23s trunk passed
          +1 mvnsite 0m 52s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 2m 1s trunk passed
          +1 javadoc 1m 5s trunk passed with JDK v1.8.0_66
          +1 javadoc 1m 48s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 47s the patch passed
          +1 compile 0m 37s the patch passed with JDK v1.8.0_66
          +1 javac 0m 37s the patch passed
          +1 compile 0m 39s the patch passed with JDK v1.7.0_91
          +1 javac 0m 39s the patch passed
          +1 checkstyle 0m 19s the patch passed
          +1 mvnsite 0m 49s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 8s the patch passed
          +1 javadoc 1m 2s the patch passed with JDK v1.8.0_66
          +1 javadoc 1m 41s the patch passed with JDK v1.7.0_91
          -1 unit 51m 50s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          +1 unit 52m 11s hadoop-hdfs in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 33s Patch does not generate ASF License warnings.
          130m 1s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.datanode.TestBlockScanner



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12785221/HDFS-9721.03.patch
          JIRA Issue HDFS-9721
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 8a3a6ddb5792 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / c9a09d6
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14292/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14292/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14292/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14292/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 1s trunk passed +1 compile 0m 41s trunk passed with JDK v1.8.0_66 +1 compile 0m 41s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 23s trunk passed +1 mvnsite 0m 52s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 2m 1s trunk passed +1 javadoc 1m 5s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 48s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 47s the patch passed +1 compile 0m 37s the patch passed with JDK v1.8.0_66 +1 javac 0m 37s the patch passed +1 compile 0m 39s the patch passed with JDK v1.7.0_91 +1 javac 0m 39s the patch passed +1 checkstyle 0m 19s the patch passed +1 mvnsite 0m 49s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 8s the patch passed +1 javadoc 1m 2s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 41s the patch passed with JDK v1.7.0_91 -1 unit 51m 50s hadoop-hdfs in the patch failed with JDK v1.8.0_66. +1 unit 52m 11s hadoop-hdfs in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 33s Patch does not generate ASF License warnings. 130m 1s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.datanode.TestBlockScanner Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12785221/HDFS-9721.03.patch JIRA Issue HDFS-9721 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 8a3a6ddb5792 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c9a09d6 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/14292/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14292/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14292/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14292/console This message was automatically generated.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          Hi, Xiao Chen

          The patch LGTM in general. Will +1 after addressing the following comments:

           } catch (IOException ioe) {
                  ++ignored;
                  if (LOG.isDebugEnabled()) {
                    LOG.debug("Exception caught, ignoring node:{}.", p.getId());
                  }
          

          Would you log the IOE in log as well? Also I feel that it should put an IOE to higher log level, e.g., INFO?

          getSnapshotName() seems just ignoring a ref Id. Would you change the function name accordingly? Additionally, can you add some comments to the following code?

          if (parent == null) {
             return getSnapshotName(inode);
          }
          

          Thanks!

          Show
          eddyxu Lei (Eddy) Xu added a comment - Hi, Xiao Chen The patch LGTM in general. Will +1 after addressing the following comments: } catch (IOException ioe) { ++ignored; if (LOG.isDebugEnabled()) { LOG.debug( "Exception caught, ignoring node:{}." , p.getId()); } Would you log the IOE in log as well? Also I feel that it should put an IOE to higher log level, e.g., INFO? getSnapshotName() seems just ignoring a ref Id. Would you change the function name accordingly? Additionally, can you add some comments to the following code? if (parent == null ) { return getSnapshotName(inode); } Thanks!
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks very much for reviewing Lei (Eddy) Xu! I'll address the last 2 comments shortly.

          I'm not sure about the log level though. Logging it at INFO or above will flood the console, and may confuse the user. For example, in a big cluster, currently the tool prints the following:

          16/01/29 11:07:01 INFO offlineImageViewer.PBImageTextWriter: Found 44801245 INodes in the INode section
          16/01/29 11:22:34 INFO offlineImageViewer.PBImageTextWriter: Ignored 2235 nodes.
          16/01/29 11:22:34 INFO offlineImageViewer.PBImageTextWriter: Outputted 44801245 INodes.
          

          If we log every exception, there will be 2235 log entries, hence flooding out the summary info.
          I understand your concern. How about I add a sentence to the Ignored 2235 nodes. to say please turn on debug log for details?, and change this log level to WARN?

          Show
          xiaochen Xiao Chen added a comment - Thanks very much for reviewing Lei (Eddy) Xu ! I'll address the last 2 comments shortly. I'm not sure about the log level though. Logging it at INFO or above will flood the console, and may confuse the user. For example, in a big cluster, currently the tool prints the following: 16/01/29 11:07:01 INFO offlineImageViewer.PBImageTextWriter: Found 44801245 INodes in the INode section 16/01/29 11:22:34 INFO offlineImageViewer.PBImageTextWriter: Ignored 2235 nodes. 16/01/29 11:22:34 INFO offlineImageViewer.PBImageTextWriter: Outputted 44801245 INodes. If we log every exception, there will be 2235 log entries, hence flooding out the summary info. I understand your concern. How about I add a sentence to the Ignored 2235 nodes. to say please turn on debug log for details? , and change this log level to WARN?
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          Xiao Chen would you consider that getSnapshotName() dose not throw an IOE? If that's an expected case.

          Show
          eddyxu Lei (Eddy) Xu added a comment - Xiao Chen would you consider that getSnapshotName() dose not throw an IOE ? If that's an expected case.
          Hide
          xiaochen Xiao Chen added a comment -

          Hm, since it's called in a recursive method, I don't think returning can distinguish it. How does a specific type of Exception (e.g. IgnoreSnapshotException) sound to you Lei (Eddy) Xu? We can ignore that and log IOE then.

          Show
          xiaochen Xiao Chen added a comment - Hm, since it's called in a recursive method, I don't think returning can distinguish it. How does a specific type of Exception (e.g. IgnoreSnapshotException ) sound to you Lei (Eddy) Xu ? We can ignore that and log IOE then.
          Hide
          xiaochen Xiao Chen added a comment -

          Patch 4 addressed the comments above - better logging and warn if IOException is thrown. Added a new type IgnoreSnapshotException to handle the snapshots in the recursive call. Lei (Eddy) Xu please take a look and let me know what you think. Thank you!

          Show
          xiaochen Xiao Chen added a comment - Patch 4 addressed the comments above - better logging and warn if IOException is thrown. Added a new type IgnoreSnapshotException to handle the snapshots in the recursive call. Lei (Eddy) Xu please take a look and let me know what you think. Thank you!
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 49s trunk passed
          +1 compile 1m 8s trunk passed with JDK v1.8.0_66
          +1 compile 0m 56s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 24s trunk passed
          +1 mvnsite 1m 8s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          +1 findbugs 2m 19s trunk passed
          +1 javadoc 1m 28s trunk passed with JDK v1.8.0_66
          +1 javadoc 2m 14s trunk passed with JDK v1.7.0_91
          +1 mvninstall 1m 0s the patch passed
          +1 compile 1m 5s the patch passed with JDK v1.8.0_66
          +1 javac 1m 5s the patch passed
          +1 compile 0m 52s the patch passed with JDK v1.7.0_91
          +1 javac 0m 52s the patch passed
          +1 checkstyle 0m 22s the patch passed
          +1 mvnsite 1m 4s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 38s the patch passed
          +1 javadoc 1m 29s the patch passed with JDK v1.8.0_66
          +1 javadoc 2m 13s the patch passed with JDK v1.7.0_91
          -1 unit 78m 19s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          -1 unit 67m 8s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
          +1 asflicense 0m 21s Patch does not generate ASF License warnings.
          178m 24s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.balancer.TestBalancer
            hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation
            hadoop.hdfs.security.TestDelegationTokenForProxyUser
          JDK v1.7.0_91 Failed junit tests hadoop.hdfs.shortcircuit.TestShortCircuitCache
            hadoop.hdfs.TestEncryptionZones
            hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency
            hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation
            hadoop.hdfs.server.namenode.ha.TestRequestHedgingProxyProvider



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12785378/HDFS-9721.04.patch
          JIRA Issue HDFS-9721
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux ea5c2cf9011f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 772ea7b
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14305/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14305/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14305/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14305/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14305/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Max memory used 77MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14305/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 49s trunk passed +1 compile 1m 8s trunk passed with JDK v1.8.0_66 +1 compile 0m 56s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 24s trunk passed +1 mvnsite 1m 8s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 2m 19s trunk passed +1 javadoc 1m 28s trunk passed with JDK v1.8.0_66 +1 javadoc 2m 14s trunk passed with JDK v1.7.0_91 +1 mvninstall 1m 0s the patch passed +1 compile 1m 5s the patch passed with JDK v1.8.0_66 +1 javac 1m 5s the patch passed +1 compile 0m 52s the patch passed with JDK v1.7.0_91 +1 javac 0m 52s the patch passed +1 checkstyle 0m 22s the patch passed +1 mvnsite 1m 4s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 38s the patch passed +1 javadoc 1m 29s the patch passed with JDK v1.8.0_66 +1 javadoc 2m 13s the patch passed with JDK v1.7.0_91 -1 unit 78m 19s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 67m 8s hadoop-hdfs in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 21s Patch does not generate ASF License warnings. 178m 24s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.balancer.TestBalancer   hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation   hadoop.hdfs.security.TestDelegationTokenForProxyUser JDK v1.7.0_91 Failed junit tests hadoop.hdfs.shortcircuit.TestShortCircuitCache   hadoop.hdfs.TestEncryptionZones   hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency   hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation   hadoop.hdfs.server.namenode.ha.TestRequestHedgingProxyProvider Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12785378/HDFS-9721.04.patch JIRA Issue HDFS-9721 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux ea5c2cf9011f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 772ea7b Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/14305/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14305/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14305/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14305/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14305/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 77MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14305/console This message was automatically generated.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          Hi, Xiao Chen Thanks a lot for addressing the above comments.

          I feel that private String ignoreSnapshotName(long inode) throws IOException can be a static method of MetadataMap and returns void, what do you think?

          Btw, it seems that we do not need to use pre-increment in the code, e.g., +ignored and +dirCount. Would you mind to change it to comply the coding style used in the rest of this file?

          Would +1 after fixing these.

          Show
          eddyxu Lei (Eddy) Xu added a comment - Hi, Xiao Chen Thanks a lot for addressing the above comments. I feel that private String ignoreSnapshotName(long inode) throws IOException can be a static method of MetadataMap and returns void , what do you think? Btw, it seems that we do not need to use pre-increment in the code, e.g., + ignored and +dirCount . Would you mind to change it to comply the coding style used in the rest of this file? Would +1 after fixing these.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks for the review Eddy!
          Patch 5 addresses all comments above. MetadataMap is an interface class, so I put the static method in PBImageTextWriter.
          Patch 5 also adds a header line to the delimited oiv output. It also adds the d or - at the beginning of the permission string, to be consistent with the legacy OIV and -ls -R

          Show
          xiaochen Xiao Chen added a comment - Thanks for the review Eddy! Patch 5 addresses all comments above. MetadataMap is an interface class, so I put the static method in PBImageTextWriter . Patch 5 also adds a header line to the delimited oiv output. It also adds the d or - at the beginning of the permission string, to be consistent with the legacy OIV and -ls -R
          Hide
          xiaochen Xiao Chen added a comment -

          Hm, BTW, should we mark this as incompatible change, due to the added d / - in the permissions string?

          Show
          xiaochen Xiao Chen added a comment - Hm, BTW, should we mark this as incompatible change, due to the added d / - in the permissions string?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 13s Maven dependency ordering for branch
          +1 mvninstall 12m 14s trunk passed
          +1 compile 1m 48s trunk passed with JDK v1.8.0_66
          +1 compile 1m 17s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 35s trunk passed
          +1 mvnsite 1m 33s trunk passed
          +1 mvneclipse 0m 22s trunk passed
          +1 findbugs 3m 2s trunk passed
          +1 javadoc 2m 15s trunk passed with JDK v1.8.0_66
          +1 javadoc 3m 24s trunk passed with JDK v1.7.0_91
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 1m 32s the patch passed
          +1 compile 2m 2s the patch passed with JDK v1.8.0_66
          +1 javac 2m 2s the patch passed
          +1 compile 1m 19s the patch passed with JDK v1.7.0_91
          +1 javac 1m 19s the patch passed
          +1 checkstyle 0m 37s the patch passed
          +1 mvnsite 1m 39s the patch passed
          +1 mvneclipse 0m 20s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 3m 41s the patch passed
          +1 javadoc 2m 23s the patch passed with JDK v1.8.0_66
          +1 javadoc 3m 24s the patch passed with JDK v1.7.0_91
          -1 unit 119m 23s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          -1 unit 88m 19s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
          -1 asflicense 0m 30s Patch generated 1 ASF License warnings.
          256m 39s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.namenode.TestNamenodeCapacityReport
            hadoop.hdfs.qjournal.TestSecureNNWithQJM
            hadoop.hdfs.TestFileAppend
            hadoop.hdfs.TestDFSUpgradeFromImage
            hadoop.hdfs.server.namenode.TestFileTruncate
            hadoop.hdfs.server.datanode.TestBlockScanner
            hadoop.hdfs.server.namenode.ha.TestEditLogTailer
            hadoop.hdfs.server.datanode.TestDirectoryScanner
            hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock
            hadoop.hdfs.TestLocalDFS
            hadoop.fs.TestSymlinkHdfsFileContext
            hadoop.hdfs.security.TestDelegationTokenForProxyUser
          JDK v1.8.0_66 Timed out junit tests org.apache.hadoop.hdfs.server.blockmanagement.TestBlockManager
          JDK v1.7.0_91 Failed junit tests hadoop.hdfs.TestDFSUpgradeFromImage
            hadoop.hdfs.server.datanode.TestBlockScanner
            hadoop.hdfs.server.datanode.TestDirectoryScanner
            hadoop.hdfs.security.TestDelegationTokenForProxyUser



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12785600/HDFS-9721.05.patch
          JIRA Issue HDFS-9721
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 7e0c8881c05d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / e30ce01
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14326/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14326/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14326/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14326/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14326/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/14326/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14326/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 13s Maven dependency ordering for branch +1 mvninstall 12m 14s trunk passed +1 compile 1m 48s trunk passed with JDK v1.8.0_66 +1 compile 1m 17s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 35s trunk passed +1 mvnsite 1m 33s trunk passed +1 mvneclipse 0m 22s trunk passed +1 findbugs 3m 2s trunk passed +1 javadoc 2m 15s trunk passed with JDK v1.8.0_66 +1 javadoc 3m 24s trunk passed with JDK v1.7.0_91 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 32s the patch passed +1 compile 2m 2s the patch passed with JDK v1.8.0_66 +1 javac 2m 2s the patch passed +1 compile 1m 19s the patch passed with JDK v1.7.0_91 +1 javac 1m 19s the patch passed +1 checkstyle 0m 37s the patch passed +1 mvnsite 1m 39s the patch passed +1 mvneclipse 0m 20s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 3m 41s the patch passed +1 javadoc 2m 23s the patch passed with JDK v1.8.0_66 +1 javadoc 3m 24s the patch passed with JDK v1.7.0_91 -1 unit 119m 23s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 88m 19s hadoop-hdfs in the patch failed with JDK v1.7.0_91. -1 asflicense 0m 30s Patch generated 1 ASF License warnings. 256m 39s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.namenode.TestNamenodeCapacityReport   hadoop.hdfs.qjournal.TestSecureNNWithQJM   hadoop.hdfs.TestFileAppend   hadoop.hdfs.TestDFSUpgradeFromImage   hadoop.hdfs.server.namenode.TestFileTruncate   hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.server.datanode.TestDirectoryScanner   hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock   hadoop.hdfs.TestLocalDFS   hadoop.fs.TestSymlinkHdfsFileContext   hadoop.hdfs.security.TestDelegationTokenForProxyUser JDK v1.8.0_66 Timed out junit tests org.apache.hadoop.hdfs.server.blockmanagement.TestBlockManager JDK v1.7.0_91 Failed junit tests hadoop.hdfs.TestDFSUpgradeFromImage   hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.server.datanode.TestDirectoryScanner   hadoop.hdfs.security.TestDelegationTokenForProxyUser Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12785600/HDFS-9721.05.patch JIRA Issue HDFS-9721 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 7e0c8881c05d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e30ce01 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/14326/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14326/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14326/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14326/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14326/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/14326/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14326/console This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          Failed tests seem unrelated.

          Show
          xiaochen Xiao Chen added a comment - Failed tests seem unrelated.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          +1. Thanks for the great work, Xiao Chen

          Committed to trunk, branch-2 and branch-2.8

          Show
          eddyxu Lei (Eddy) Xu added a comment - +1. Thanks for the great work, Xiao Chen Committed to trunk , branch-2 and branch-2.8
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9225 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9225/)
          HDFS-9721. Allow Delimited PB OIV tool to run upon fsimage that contains (lei: rev 9d494f0c0eaa05417f3a3e88487d878d1731da36)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/PBImageTextWriter.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/IgnoreSnapshotException.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/PBImageDelimitedTextWriter.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/FSImageLoader.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/TestOfflineImageViewer.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9225 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9225/ ) HDFS-9721 . Allow Delimited PB OIV tool to run upon fsimage that contains (lei: rev 9d494f0c0eaa05417f3a3e88487d878d1731da36) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/PBImageTextWriter.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/IgnoreSnapshotException.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/PBImageDelimitedTextWriter.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/FSImageLoader.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/TestOfflineImageViewer.java
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks for the helpful reviews and commit Lei (Eddy) Xu.

          Show
          xiaochen Xiao Chen added a comment - Thanks for the helpful reviews and commit Lei (Eddy) Xu .

            People

            • Assignee:
              xiaochen Xiao Chen
              Reporter:
              xiaochen Xiao Chen
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development