Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 2.7.4, 3.0.0-alpha1
    • Component/s: test
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      The patch replaces -namenode option with -fs for specifying the remote name node against which the benchmark is running. Before this patch, if '-namenode' was not given, the benchmark would run in standalone mode, ignoring the 'fs.defaultFS' in config file even if it's remote. With this patch, the benchmark, as other tools, will rely on the 'fs.defaultFS' config, which is overridable by -fs command option, to run standalone mode or remote mode.
      Show
      The patch replaces -namenode option with -fs for specifying the remote name node against which the benchmark is running. Before this patch, if '-namenode' was not given, the benchmark would run in standalone mode, ignoring the 'fs.defaultFS' in config file even if it's remote. With this patch, the benchmark, as other tools, will rely on the 'fs.defaultFS' config, which is overridable by -fs command option, to run standalone mode or remote mode.

      Description

      HDFS-7847 introduced a new option -namenode, which is intended to point the benchmark to a remote NameNode. It should use a standard generic option -fs instead, which is routinely used to specify NameNode URI in shell commands.

      1. HDFS-9053.000.patch
        6 kB
        Mingliang Liu
      2. HDFS-9053.001.patch
        6 kB
        Mingliang Liu
      3. HDFS-9053.002.patch
        11 kB
        Mingliang Liu
      4. HDFS-9053.003.patch
        10 kB
        Mingliang Liu
      5. HDFS-9053.004.patch
        11 kB
        Mingliang Liu
      6. HDFS-9053.005.patch
        11 kB
        Mingliang Liu
      7. HDFS-9503-branch-2.7.patch
        12 kB
        Konstantin Shvachko

        Issue Links

          Activity

          Hide
          liuml07 Mingliang Liu added a comment -

          Feel free to re-assign it to someone who's actively working on this.

          Show
          liuml07 Mingliang Liu added a comment - Feel free to re-assign it to someone who's actively working on this.
          Hide
          liuml07 Mingliang Liu added a comment -

          The v0 patch:

          • replace the -namenode with -fs
          • Add a unit test against remote name node by building a mini cluster
          • Save datanode mapping uuid -> dnIdx to a map for calculating datanode index in addBlocks.
          Show
          liuml07 Mingliang Liu added a comment - The v0 patch: replace the -namenode with -fs Add a unit test against remote name node by building a mini cluster Save datanode mapping uuid -> dnIdx to a map for calculating datanode index in addBlocks .
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 8m 30s trunk passed
          +1 compile 0m 59s trunk passed with JDK v1.8.0_66
          +1 compile 0m 48s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 18s trunk passed
          +1 mvnsite 0m 58s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          +1 findbugs 2m 11s trunk passed
          +1 javadoc 1m 28s trunk passed with JDK v1.8.0_66
          +1 javadoc 1m 53s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 51s the patch passed
          +1 compile 0m 43s the patch passed with JDK v1.8.0_66
          -1 javac 7m 7s hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66 with JDK v1.8.0_66 generated 2 new issues (was 32, now 32).
          +1 javac 0m 43s the patch passed
          +1 compile 0m 44s the patch passed with JDK v1.7.0_91
          -1 javac 7m 51s hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91 with JDK v1.7.0_91 generated 2 new issues (was 34, now 34).
          +1 javac 0m 44s the patch passed
          +1 checkstyle 0m 16s the patch passed
          +1 mvnsite 0m 55s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 12s the patch passed
          +1 javadoc 1m 10s the patch passed with JDK v1.8.0_66
          +1 javadoc 1m 54s the patch passed with JDK v1.7.0_91
          -1 unit 56m 45s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          -1 unit 54m 44s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
          -1 asflicense 0m 20s Patch generated 58 ASF License warnings.
          141m 5s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure100
            hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
            hadoop.hdfs.TestDFSStripedInputStream
          JDK v1.7.0_91 Failed junit tests hadoop.hdfs.TestRecoverStripedFile



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12777251/HDFS-9053.000.patch
          JIRA Issue HDFS-9503
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 961fccdd16c4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / bf5295b
          findbugs v3.0.0
          javac hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-HDFS-Build/13850/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          javac hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-HDFS-Build/13850/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13850/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13850/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13850/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13850/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13850/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13850/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Max memory used 75MB
          Powered by Apache Yetus 0.1.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13850/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 8m 30s trunk passed +1 compile 0m 59s trunk passed with JDK v1.8.0_66 +1 compile 0m 48s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 18s trunk passed +1 mvnsite 0m 58s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 2m 11s trunk passed +1 javadoc 1m 28s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 53s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 51s the patch passed +1 compile 0m 43s the patch passed with JDK v1.8.0_66 -1 javac 7m 7s hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66 with JDK v1.8.0_66 generated 2 new issues (was 32, now 32). +1 javac 0m 43s the patch passed +1 compile 0m 44s the patch passed with JDK v1.7.0_91 -1 javac 7m 51s hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91 with JDK v1.7.0_91 generated 2 new issues (was 34, now 34). +1 javac 0m 44s the patch passed +1 checkstyle 0m 16s the patch passed +1 mvnsite 0m 55s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 12s the patch passed +1 javadoc 1m 10s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 54s the patch passed with JDK v1.7.0_91 -1 unit 56m 45s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 54m 44s hadoop-hdfs in the patch failed with JDK v1.7.0_91. -1 asflicense 0m 20s Patch generated 58 ASF License warnings. 141m 5s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure100   hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes   hadoop.hdfs.TestDFSStripedInputStream JDK v1.7.0_91 Failed junit tests hadoop.hdfs.TestRecoverStripedFile Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12777251/HDFS-9053.000.patch JIRA Issue HDFS-9503 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 961fccdd16c4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / bf5295b findbugs v3.0.0 javac hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-HDFS-Build/13850/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt javac hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-HDFS-Build/13850/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13850/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13850/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13850/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13850/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13850/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13850/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 75MB Powered by Apache Yetus 0.1.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13850/console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 14m 40s trunk passed
          +1 compile 2m 8s trunk passed with JDK v1.8.0_66
          +1 compile 1m 27s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 30s trunk passed
          +1 mvnsite 1m 45s trunk passed
          +1 mvneclipse 0m 23s trunk passed
          +1 findbugs 3m 35s trunk passed
          +1 javadoc 2m 37s trunk passed with JDK v1.8.0_66
          +1 javadoc 3m 38s trunk passed with JDK v1.7.0_91
          +1 mvninstall 1m 52s the patch passed
          +1 compile 2m 20s the patch passed with JDK v1.8.0_66
          -1 javac 14m 3s hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66 with JDK v1.8.0_66 generated 2 new issues (was 32, now 32).
          +1 javac 2m 20s the patch passed
          +1 compile 1m 31s the patch passed with JDK v1.7.0_91
          -1 javac 15m 34s hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91 with JDK v1.7.0_91 generated 2 new issues (was 34, now 34).
          +1 javac 1m 31s the patch passed
          +1 checkstyle 0m 30s the patch passed
          +1 mvnsite 1m 49s the patch passed
          +1 mvneclipse 0m 26s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 4m 32s the patch passed
          +1 javadoc 3m 8s the patch passed with JDK v1.8.0_66
          +1 javadoc 4m 7s the patch passed with JDK v1.7.0_91
          -1 unit 179m 11s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          -1 unit 149m 50s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
          -1 asflicense 0m 34s Patch generated 56 ASF License warnings.
          387m 22s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestEncryptedTransfer
            hadoop.hdfs.TestDFSUpgradeFromImage
            hadoop.hdfs.server.datanode.TestBlockScanner
            hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency
            hadoop.tracing.TestTracing
            hadoop.hdfs.TestDFSStripedInputStream
            hadoop.hdfs.server.namenode.TestProtectedDirectories
            hadoop.hdfs.server.namenode.ha.TestEditLogTailer
            hadoop.hdfs.TestPersistBlocks
            hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
            hadoop.hdfs.security.TestDelegationTokenForProxyUser
            hadoop.hdfs.TestLocalDFS
            hadoop.hdfs.server.namenode.TestMetadataVersionOutput
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
            hadoop.hdfs.tools.TestDFSAdminWithHA
            hadoop.hdfs.web.TestWebHdfsTokens
            hadoop.hdfs.server.namenode.TestMetaSave
            hadoop.hdfs.server.namenode.TestSecurityTokenEditLog
            hadoop.hdfs.web.TestWebHdfsFileSystemContract
            hadoop.hdfs.server.datanode.TestBlockReplacement
            hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080
            hadoop.fs.TestSymlinkHdfsFileContext
            hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA
            hadoop.hdfs.server.namenode.TestDecommissioningStatus
            hadoop.hdfs.qjournal.TestSecureNNWithQJM
            hadoop.hdfs.server.namenode.TestFileTruncate
            hadoop.hdfs.TestEncryptionZones
            hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot
            hadoop.hdfs.server.namenode.TestDeleteRace
            hadoop.hdfs.server.datanode.TestDirectoryScanner
          JDK v1.7.0_91 Failed junit tests hadoop.hdfs.TestDFSUpgradeFromImage
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure
            hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060
            hadoop.hdfs.server.namenode.TestFavoredNodesEndToEnd
            hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations
            hadoop.hdfs.security.TestDelegationTokenForProxyUser
            hadoop.hdfs.TestLocalDFS
            hadoop.hdfs.TestDistributedFileSystem
            hadoop.hdfs.server.namenode.TestHDFSConcat
            hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation
            hadoop.hdfs.server.namenode.ha.TestHAAppend
            hadoop.hdfs.server.namenode.TestDecommissioningStatus
            hadoop.hdfs.qjournal.TestSecureNNWithQJM
            hadoop.hdfs.server.namenode.TestFileTruncate
            hadoop.hdfs.server.blockmanagement.TestPendingReplication
            hadoop.hdfs.TestEncryptionZones
            hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork
            hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot
            hadoop.hdfs.server.datanode.TestDirectoryScanner



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12777251/HDFS-9053.000.patch
          JIRA Issue HDFS-9503
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux d051b8495c4f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / bf5295b
          findbugs v3.0.0
          javac hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-HDFS-Build/13852/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          javac hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-HDFS-Build/13852/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13852/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13852/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13852/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13852/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13852/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13852/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Max memory used 76MB
          Powered by Apache Yetus 0.1.0 http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13852/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 14m 40s trunk passed +1 compile 2m 8s trunk passed with JDK v1.8.0_66 +1 compile 1m 27s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 30s trunk passed +1 mvnsite 1m 45s trunk passed +1 mvneclipse 0m 23s trunk passed +1 findbugs 3m 35s trunk passed +1 javadoc 2m 37s trunk passed with JDK v1.8.0_66 +1 javadoc 3m 38s trunk passed with JDK v1.7.0_91 +1 mvninstall 1m 52s the patch passed +1 compile 2m 20s the patch passed with JDK v1.8.0_66 -1 javac 14m 3s hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66 with JDK v1.8.0_66 generated 2 new issues (was 32, now 32). +1 javac 2m 20s the patch passed +1 compile 1m 31s the patch passed with JDK v1.7.0_91 -1 javac 15m 34s hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91 with JDK v1.7.0_91 generated 2 new issues (was 34, now 34). +1 javac 1m 31s the patch passed +1 checkstyle 0m 30s the patch passed +1 mvnsite 1m 49s the patch passed +1 mvneclipse 0m 26s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 4m 32s the patch passed +1 javadoc 3m 8s the patch passed with JDK v1.8.0_66 +1 javadoc 4m 7s the patch passed with JDK v1.7.0_91 -1 unit 179m 11s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 149m 50s hadoop-hdfs in the patch failed with JDK v1.7.0_91. -1 asflicense 0m 34s Patch generated 56 ASF License warnings. 387m 22s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestEncryptedTransfer   hadoop.hdfs.TestDFSUpgradeFromImage   hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency   hadoop.tracing.TestTracing   hadoop.hdfs.TestDFSStripedInputStream   hadoop.hdfs.server.namenode.TestProtectedDirectories   hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.TestPersistBlocks   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.TestLocalDFS   hadoop.hdfs.server.namenode.TestMetadataVersionOutput   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.tools.TestDFSAdminWithHA   hadoop.hdfs.web.TestWebHdfsTokens   hadoop.hdfs.server.namenode.TestMetaSave   hadoop.hdfs.server.namenode.TestSecurityTokenEditLog   hadoop.hdfs.web.TestWebHdfsFileSystemContract   hadoop.hdfs.server.datanode.TestBlockReplacement   hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080   hadoop.fs.TestSymlinkHdfsFileContext   hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA   hadoop.hdfs.server.namenode.TestDecommissioningStatus   hadoop.hdfs.qjournal.TestSecureNNWithQJM   hadoop.hdfs.server.namenode.TestFileTruncate   hadoop.hdfs.TestEncryptionZones   hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot   hadoop.hdfs.server.namenode.TestDeleteRace   hadoop.hdfs.server.datanode.TestDirectoryScanner JDK v1.7.0_91 Failed junit tests hadoop.hdfs.TestDFSUpgradeFromImage   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure   hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060   hadoop.hdfs.server.namenode.TestFavoredNodesEndToEnd   hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.TestLocalDFS   hadoop.hdfs.TestDistributedFileSystem   hadoop.hdfs.server.namenode.TestHDFSConcat   hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation   hadoop.hdfs.server.namenode.ha.TestHAAppend   hadoop.hdfs.server.namenode.TestDecommissioningStatus   hadoop.hdfs.qjournal.TestSecureNNWithQJM   hadoop.hdfs.server.namenode.TestFileTruncate   hadoop.hdfs.server.blockmanagement.TestPendingReplication   hadoop.hdfs.TestEncryptionZones   hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork   hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot   hadoop.hdfs.server.datanode.TestDirectoryScanner Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12777251/HDFS-9053.000.patch JIRA Issue HDFS-9503 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux d051b8495c4f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / bf5295b findbugs v3.0.0 javac hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-HDFS-Build/13852/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt javac hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-HDFS-Build/13852/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13852/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13852/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13852/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13852/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13852/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13852/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 76MB Powered by Apache Yetus 0.1.0 http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13852/console This message was automatically generated.
          Hide
          liuml07 Mingliang Liu added a comment -

          The failing tests can not re-produce locally and seem unrelated. Re-submit the patch to trigger Jenkins.

          Show
          liuml07 Mingliang Liu added a comment - The failing tests can not re-produce locally and seem unrelated. Re-submit the patch to trigger Jenkins.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 9m 12s trunk passed
          +1 compile 1m 2s trunk passed with JDK v1.8.0_66
          +1 compile 0m 49s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 19s trunk passed
          +1 mvnsite 0m 59s trunk passed
          +1 mvneclipse 0m 16s trunk passed
          +1 findbugs 2m 31s trunk passed
          +1 javadoc 1m 21s trunk passed with JDK v1.8.0_66
          +1 javadoc 2m 7s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 58s the patch passed
          +1 compile 0m 58s the patch passed with JDK v1.8.0_66
          +1 javac 0m 58s the patch passed
          +1 compile 0m 43s the patch passed with JDK v1.7.0_91
          +1 javac 0m 43s the patch passed
          +1 checkstyle 0m 17s the patch passed
          +1 mvnsite 0m 59s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 26s the patch passed
          +1 javadoc 1m 15s the patch passed with JDK v1.8.0_66
          +1 javadoc 1m 54s the patch passed with JDK v1.7.0_91
          -1 unit 74m 25s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          +1 unit 76m 50s hadoop-hdfs in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 36s Patch does not generate ASF License warnings.
          183m 52s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestLeaseRecovery
            hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12783935/HDFS-9053.001.patch
          JIRA Issue HDFS-9503
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 25bd84d04bd9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 46e5ea8
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14213/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14213/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14213/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14213/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 9m 12s trunk passed +1 compile 1m 2s trunk passed with JDK v1.8.0_66 +1 compile 0m 49s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 19s trunk passed +1 mvnsite 0m 59s trunk passed +1 mvneclipse 0m 16s trunk passed +1 findbugs 2m 31s trunk passed +1 javadoc 1m 21s trunk passed with JDK v1.8.0_66 +1 javadoc 2m 7s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 58s the patch passed +1 compile 0m 58s the patch passed with JDK v1.8.0_66 +1 javac 0m 58s the patch passed +1 compile 0m 43s the patch passed with JDK v1.7.0_91 +1 javac 0m 43s the patch passed +1 checkstyle 0m 17s the patch passed +1 mvnsite 0m 59s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 26s the patch passed +1 javadoc 1m 15s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 54s the patch passed with JDK v1.7.0_91 -1 unit 74m 25s hadoop-hdfs in the patch failed with JDK v1.8.0_66. +1 unit 76m 50s hadoop-hdfs in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 36s Patch does not generate ASF License warnings. 183m 52s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestLeaseRecovery   hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12783935/HDFS-9053.001.patch JIRA Issue HDFS-9503 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 25bd84d04bd9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 46e5ea8 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/14213/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14213/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14213/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14213/console This message was automatically generated.
          Hide
          liuml07 Mingliang Liu added a comment -

          The failing tests seem unrelated. Konstantin Shvachko Would you kindly review? Thanks.

          Show
          liuml07 Mingliang Liu added a comment - The failing tests seem unrelated. Konstantin Shvachko Would you kindly review? Thanks.
          Hide
          vinayrpet Vinayakumar B added a comment -

          Since NNThroughputBenchmark is an implementation of Tool interface, which already supports parsing of -fs option as part of GenericOptions provided these are mentioned at first.

          I think, Since -fs is already provided by default, Now just need another flag ( may be instead of -namenode) to mention whether to use New Namenode instance, or just connect to remote namenode.

          Also, this would be an Incompatible change,

          Show
          vinayrpet Vinayakumar B added a comment - Since NNThroughputBenchmark is an implementation of Tool interface, which already supports parsing of -fs option as part of GenericOptions provided these are mentioned at first. I think, Since -fs is already provided by default, Now just need another flag ( may be instead of -namenode) to mention whether to use New Namenode instance, or just connect to remote namenode. Also, this would be an Incompatible change,
          Hide
          liuml07 Mingliang Liu added a comment -

          Thank you Vinayakumar B for your kind comment. I totally agree with you that we should delegate handling -fs option (and all other generic options) to the Tool and ToolRunner framework.

          If not specified, the default file system URI will be file:///. Otherwise, it should be hdfs://host:port format, parsed by GenericOptionsParser. Perhaps we don't need another flag to indicate new NN instance or remote NN?

          Show
          liuml07 Mingliang Liu added a comment - Thank you Vinayakumar B for your kind comment. I totally agree with you that we should delegate handling -fs option (and all other generic options) to the Tool and ToolRunner framework. If not specified, the default file system URI will be file:/// . Otherwise, it should be hdfs://host:port format, parsed by GenericOptionsParser . Perhaps we don't need another flag to indicate new NN instance or remote NN?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 11m 12s trunk passed
          +1 compile 1m 21s trunk passed with JDK v1.8.0_66
          +1 compile 1m 4s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 28s trunk passed
          +1 mvnsite 1m 20s trunk passed
          +1 mvneclipse 0m 18s trunk passed
          +1 findbugs 2m 37s trunk passed
          +1 javadoc 1m 44s trunk passed with JDK v1.8.0_66
          +1 javadoc 2m 42s trunk passed with JDK v1.7.0_91
          +1 mvninstall 1m 6s the patch passed
          +1 compile 1m 19s the patch passed with JDK v1.8.0_66
          +1 javac 1m 19s the patch passed
          +1 compile 1m 0s the patch passed with JDK v1.7.0_91
          +1 javac 1m 0s the patch passed
          +1 checkstyle 0m 23s the patch passed
          +1 mvnsite 1m 10s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 51s the patch passed
          +1 javadoc 1m 50s the patch passed with JDK v1.8.0_66
          +1 javadoc 2m 36s the patch passed with JDK v1.7.0_91
          -1 unit 78m 55s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          -1 unit 105m 22s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
          -1 asflicense 0m 44s Patch generated 1 ASF License warnings.
          224m 14s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestEncryptionZones
            hadoop.hdfs.server.blockmanagement.TestBlockManager
            hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency
            hadoop.hdfs.server.namenode.ha.TestEditLogTailer
            hadoop.hdfs.security.TestDelegationTokenForProxyUser
            hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA
            hadoop.hdfs.server.datanode.TestDataNodeMetrics
            hadoop.hdfs.server.namenode.TestNamenodeCapacityReport
            hadoop.hdfs.server.datanode.TestBlockScanner
          JDK v1.8.0_66 Timed out junit tests org.apache.hadoop.hdfs.TestParallelShortCircuitLegacyRead
            org.apache.hadoop.hdfs.TestAclsEndToEnd
            org.apache.hadoop.hdfs.TestDFSStorageStateRecovery
            org.apache.hadoop.hdfs.TestFileAppend2
          JDK v1.7.0_91 Failed junit tests hadoop.hdfs.server.datanode.TestDirectoryScanner
            hadoop.hdfs.TestEncryptionZones
            hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency
            hadoop.hdfs.server.namenode.TestFileTruncate
            hadoop.hdfs.TestMissingBlocksAlert
            hadoop.hdfs.server.datanode.TestBlockReplacement
            hadoop.hdfs.TestDFSClientRetries
            hadoop.hdfs.server.datanode.TestBlockScanner
          JDK v1.7.0_91 Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12784336/HDFS-9053.002.patch
          JIRA Issue HDFS-9503
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux e40935def559 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / bd909ed
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14241/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14241/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14241/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14241/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14241/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/14241/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14241/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 11m 12s trunk passed +1 compile 1m 21s trunk passed with JDK v1.8.0_66 +1 compile 1m 4s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 28s trunk passed +1 mvnsite 1m 20s trunk passed +1 mvneclipse 0m 18s trunk passed +1 findbugs 2m 37s trunk passed +1 javadoc 1m 44s trunk passed with JDK v1.8.0_66 +1 javadoc 2m 42s trunk passed with JDK v1.7.0_91 +1 mvninstall 1m 6s the patch passed +1 compile 1m 19s the patch passed with JDK v1.8.0_66 +1 javac 1m 19s the patch passed +1 compile 1m 0s the patch passed with JDK v1.7.0_91 +1 javac 1m 0s the patch passed +1 checkstyle 0m 23s the patch passed +1 mvnsite 1m 10s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 51s the patch passed +1 javadoc 1m 50s the patch passed with JDK v1.8.0_66 +1 javadoc 2m 36s the patch passed with JDK v1.7.0_91 -1 unit 78m 55s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 105m 22s hadoop-hdfs in the patch failed with JDK v1.7.0_91. -1 asflicense 0m 44s Patch generated 1 ASF License warnings. 224m 14s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestEncryptionZones   hadoop.hdfs.server.blockmanagement.TestBlockManager   hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency   hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA   hadoop.hdfs.server.datanode.TestDataNodeMetrics   hadoop.hdfs.server.namenode.TestNamenodeCapacityReport   hadoop.hdfs.server.datanode.TestBlockScanner JDK v1.8.0_66 Timed out junit tests org.apache.hadoop.hdfs.TestParallelShortCircuitLegacyRead   org.apache.hadoop.hdfs.TestAclsEndToEnd   org.apache.hadoop.hdfs.TestDFSStorageStateRecovery   org.apache.hadoop.hdfs.TestFileAppend2 JDK v1.7.0_91 Failed junit tests hadoop.hdfs.server.datanode.TestDirectoryScanner   hadoop.hdfs.TestEncryptionZones   hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency   hadoop.hdfs.server.namenode.TestFileTruncate   hadoop.hdfs.TestMissingBlocksAlert   hadoop.hdfs.server.datanode.TestBlockReplacement   hadoop.hdfs.TestDFSClientRetries   hadoop.hdfs.server.datanode.TestBlockScanner JDK v1.7.0_91 Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2 Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12784336/HDFS-9053.002.patch JIRA Issue HDFS-9503 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux e40935def559 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / bd909ed Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/14241/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14241/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14241/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14241/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14241/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/14241/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14241/console This message was automatically generated.
          Hide
          vinayrpet Vinayakumar B added a comment -

          If not specified, the default file system URI will be file:///. Otherwise, it should be hdfs://host:port format, parsed by GenericOptionsParser. Perhaps we don't need another flag to indicate new NN instance or remote NN?

          This is not quite right. This makes the restriction that, core-site.xml also should not have changed the 'fs.defaultFS' configuration.

          I think, Since -fs is already provided by default, Now just need another flag ( may be instead of -namenode) to mention whether to use New Namenode instance, or just connect to remote namenode.

          What I mean by this is, both configuring 'fs.defaultFS' or specifying '-fs' to 'hdfs://localhost:port' are same.
          Only thing we need to inform NNThroughputBenchmark is whether to use existing running Namenode (might be remote or on same node) or create its own instance. This we can do by having another option as flag ( instead of -namenode ) like, "-remotenamenode". If this present, then makes use of '-fs' or 'fs.defaultFS' option. Else creates its own Namenode.

          Since this change is incompatible,
          If anybody out there feels these changes are not required?

          Show
          vinayrpet Vinayakumar B added a comment - If not specified, the default file system URI will be file:/// . Otherwise, it should be hdfs://host:port format, parsed by GenericOptionsParser. Perhaps we don't need another flag to indicate new NN instance or remote NN? This is not quite right. This makes the restriction that, core-site.xml also should not have changed the 'fs.defaultFS' configuration. I think, Since -fs is already provided by default, Now just need another flag ( may be instead of -namenode) to mention whether to use New Namenode instance, or just connect to remote namenode. What I mean by this is, both configuring 'fs.defaultFS' or specifying '-fs' to 'hdfs://localhost:port' are same. Only thing we need to inform NNThroughputBenchmark is whether to use existing running Namenode (might be remote or on same node) or create its own instance. This we can do by having another option as flag ( instead of -namenode ) like, "-remotenamenode". If this present, then makes use of '-fs' or 'fs.defaultFS' option. Else creates its own Namenode. Since this change is incompatible, If anybody out there feels these changes are not required?
          Hide
          shv Konstantin Shvachko added a comment -

          Mingliang, I could not apply your last patch to trunk. Also it seems that you are adding some unrelated changes there, like introducing dnMap in BlockReportStats.

          Show
          shv Konstantin Shvachko added a comment - Mingliang, I could not apply your last patch to trunk. Also it seems that you are adding some unrelated changes there, like introducing dnMap in BlockReportStats .
          Hide
          liuml07 Mingliang Liu added a comment -

          Thanks for your kind remind, Konstantin Shvachko. The v3 patch is the latest one rebased from trunk branch. I deleted the unrelated changes like using a map for datanodes. I added this because I met problems when running BlockReportStats sub-test against a remote NN. I'll file a new jira if it's a potential bug.

          Meanwhile, I'm still thinking about Vinayakumar B's comment as I'm not quite sure if another option is required, in order to indicate new namenode instance creation. So his concern was not addressed in the v3 patch.

          Show
          liuml07 Mingliang Liu added a comment - Thanks for your kind remind, Konstantin Shvachko . The v3 patch is the latest one rebased from trunk branch. I deleted the unrelated changes like using a map for datanodes . I added this because I met problems when running BlockReportStats sub-test against a remote NN. I'll file a new jira if it's a potential bug. Meanwhile, I'm still thinking about Vinayakumar B 's comment as I'm not quite sure if another option is required, in order to indicate new namenode instance creation. So his concern was not addressed in the v3 patch.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 6m 35s trunk passed
          +1 compile 0m 40s trunk passed with JDK v1.8.0_66
          +1 compile 0m 40s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 21s trunk passed
          +1 mvnsite 0m 51s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 51s trunk passed
          +1 javadoc 1m 5s trunk passed with JDK v1.8.0_66
          +1 javadoc 1m 45s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 43s the patch passed
          +1 compile 0m 35s the patch passed with JDK v1.8.0_66
          +1 javac 0m 35s the patch passed
          +1 compile 0m 39s the patch passed with JDK v1.7.0_91
          +1 javac 0m 39s the patch passed
          +1 checkstyle 0m 17s the patch passed
          +1 mvnsite 0m 49s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 3s the patch passed
          +1 javadoc 1m 1s the patch passed with JDK v1.8.0_66
          +1 javadoc 1m 46s the patch passed with JDK v1.7.0_91
          -1 unit 64m 23s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          -1 unit 65m 34s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
          +1 asflicense 0m 21s Patch does not generate ASF License warnings.
          154m 44s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestRollingUpgrade
          JDK v1.7.0_91 Failed junit tests hadoop.hdfs.server.namenode.TestBackupNode
            hadoop.hdfs.shortcircuit.TestShortCircuitCache



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12785070/HDFS-9053.003.patch
          JIRA Issue HDFS-9503
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 9ae991316d3d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / ee005e0
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14285/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14285/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14285/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14285/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14285/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14285/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 6m 35s trunk passed +1 compile 0m 40s trunk passed with JDK v1.8.0_66 +1 compile 0m 40s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 21s trunk passed +1 mvnsite 0m 51s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 51s trunk passed +1 javadoc 1m 5s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 45s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 43s the patch passed +1 compile 0m 35s the patch passed with JDK v1.8.0_66 +1 javac 0m 35s the patch passed +1 compile 0m 39s the patch passed with JDK v1.7.0_91 +1 javac 0m 39s the patch passed +1 checkstyle 0m 17s the patch passed +1 mvnsite 0m 49s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 3s the patch passed +1 javadoc 1m 1s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 46s the patch passed with JDK v1.7.0_91 -1 unit 64m 23s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 65m 34s hadoop-hdfs in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 21s Patch does not generate ASF License warnings. 154m 44s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestRollingUpgrade JDK v1.7.0_91 Failed junit tests hadoop.hdfs.server.namenode.TestBackupNode   hadoop.hdfs.shortcircuit.TestShortCircuitCache Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12785070/HDFS-9053.003.patch JIRA Issue HDFS-9503 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 9ae991316d3d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ee005e0 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/14285/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14285/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14285/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14285/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14285/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14285/console This message was automatically generated.
          Hide
          shv Konstantin Shvachko added a comment -

          > If anybody out there feels these changes are not required?

          Vinayakumar B, which changes do you think are not needed? Could you please clarify.

          Show
          shv Konstantin Shvachko added a comment - > If anybody out there feels these changes are not required? Vinayakumar B , which changes do you think are not needed? Could you please clarify.
          Hide
          shv Konstantin Shvachko added a comment -
          1. Mingliang, there is still some problem with diff generation for TestNNThroughputBenchmark.java. I understand what you are trying to do, but the diff looks wrong and is not applying for me.
          2. I do not think we need another parameter to switch to a remote NN. Whatever comes from generic options or configuration should be treated as a NameNode URI. If the URI is null or file:/// we create a local NN, otherwise connect to the remote. The main use case for NNThroughputBenchmark is a standalone tool, but if people want they have many ways to connect it to a remote NN.
          3. Given the above you can still use URI to check if the NN is local or remote instead of using fs instanceof DistributedFileSystem. The URI could be a local variable in the run() method.
          Show
          shv Konstantin Shvachko added a comment - Mingliang, there is still some problem with diff generation for TestNNThroughputBenchmark.java . I understand what you are trying to do, but the diff looks wrong and is not applying for me. I do not think we need another parameter to switch to a remote NN. Whatever comes from generic options or configuration should be treated as a NameNode URI. If the URI is null or file:/// we create a local NN, otherwise connect to the remote. The main use case for NNThroughputBenchmark is a standalone tool, but if people want they have many ways to connect it to a remote NN. Given the above you can still use URI to check if the NN is local or remote instead of using fs instanceof DistributedFileSystem . The URI could be a local variable in the run() method.
          Hide
          liuml07 Mingliang Liu added a comment -

          Hi Konstantin Shvachko, Thanks for your insightful comments.

          1. The previous patch files I uploaded were generated by git format-patch, so perhaps you need git am to apply them locally. The Jenkins is able to use (see reports above) the patch format so I did not realize it was a problem. The v4 patch was generated by git diff and hopefully it is applicable via git apply.

          2. I agree with you that another parameter is desirable to switch to a remote NN. Perhaps Vinayakumar B has further comments.

          3.As you commented, either fs instanceof DistributedFileSystem or nnUri == "file:///" will work. The latest patch v4 uses the nnUri for simplicity.

          Show
          liuml07 Mingliang Liu added a comment - Hi Konstantin Shvachko , Thanks for your insightful comments. 1. The previous patch files I uploaded were generated by git format-patch , so perhaps you need git am to apply them locally. The Jenkins is able to use (see reports above) the patch format so I did not realize it was a problem. The v4 patch was generated by git diff and hopefully it is applicable via git apply . 2. I agree with you that another parameter is desirable to switch to a remote NN. Perhaps Vinayakumar B has further comments. 3.As you commented, either fs instanceof DistributedFileSystem or nnUri == "file:///" will work. The latest patch v4 uses the nnUri for simplicity.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 6m 53s trunk passed
          +1 compile 0m 38s trunk passed with JDK v1.8.0_66
          +1 compile 0m 40s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 20s trunk passed
          +1 mvnsite 0m 50s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 52s trunk passed
          +1 javadoc 1m 2s trunk passed with JDK v1.8.0_66
          +1 javadoc 1m 44s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 44s the patch passed
          +1 compile 0m 35s the patch passed with JDK v1.8.0_66
          +1 javac 0m 35s the patch passed
          +1 compile 0m 38s the patch passed with JDK v1.7.0_91
          +1 javac 0m 38s the patch passed
          +1 checkstyle 0m 18s the patch passed
          +1 mvnsite 0m 49s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 7s the patch passed
          +1 javadoc 1m 5s the patch passed with JDK v1.8.0_66
          +1 javadoc 1m 44s the patch passed with JDK v1.7.0_91
          -1 unit 53m 39s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          +1 unit 54m 9s hadoop-hdfs in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 25s Patch does not generate ASF License warnings.
          132m 56s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.datanode.TestBlockScanner



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12785308/HDFS-9053.004.patch
          JIRA Issue HDFS-9503
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 5ffcb0a3f7f4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 772ea7b
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14297/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14297/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14297/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Max memory used 77MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14297/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 6m 53s trunk passed +1 compile 0m 38s trunk passed with JDK v1.8.0_66 +1 compile 0m 40s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 20s trunk passed +1 mvnsite 0m 50s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 52s trunk passed +1 javadoc 1m 2s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 44s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 44s the patch passed +1 compile 0m 35s the patch passed with JDK v1.8.0_66 +1 javac 0m 35s the patch passed +1 compile 0m 38s the patch passed with JDK v1.7.0_91 +1 javac 0m 38s the patch passed +1 checkstyle 0m 18s the patch passed +1 mvnsite 0m 49s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 7s the patch passed +1 javadoc 1m 5s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 44s the patch passed with JDK v1.7.0_91 -1 unit 53m 39s hadoop-hdfs in the patch failed with JDK v1.8.0_66. +1 unit 54m 9s hadoop-hdfs in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 25s Patch does not generate ASF License warnings. 132m 56s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.datanode.TestBlockScanner Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12785308/HDFS-9053.004.patch JIRA Issue HDFS-9503 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 5ffcb0a3f7f4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 772ea7b Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/14297/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14297/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14297/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 77MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14297/console This message was automatically generated.
          Hide
          shv Konstantin Shvachko added a comment -

          Sorry, it was my fault, I used some old eclipse, which couldn't apply the patch. The patch was fine.
          I tested this with -fs option locally. Minor comment on the last patch:

          1. The message about limits
                System.err.println(" If you use remote namenode by -fs option, its config" +
                    "dfs.namenode.fs-limits.min-block-size should be set 16.");
            

            should say something like

                System.err.println("If connecting to a remote NameNode with -fs option " +
                "dfs.namenode.fs-limits.min-block-size should be set to 16.");
            
          2. In the test cluster.shutdown(); should be in the finally section. Otherwise in case of failure of the benchmark we will have an unclosed cluster, which usually affects subsequent tests.
          Show
          shv Konstantin Shvachko added a comment - Sorry, it was my fault, I used some old eclipse, which couldn't apply the patch. The patch was fine. I tested this with -fs option locally. Minor comment on the last patch: The message about limits System .err.println( " If you use remote namenode by -fs option, its config" + "dfs.namenode.fs-limits.min-block-size should be set 16." ); should say something like System .err.println( "If connecting to a remote NameNode with -fs option " + "dfs.namenode.fs-limits.min-block-size should be set to 16." ); In the test cluster.shutdown(); should be in the finally section. Otherwise in case of failure of the benchmark we will have an unclosed cluster, which usually affects subsequent tests.
          Hide
          liuml07 Mingliang Liu added a comment -

          s/desirable/not desirable/

          Sorry about this.

          Show
          liuml07 Mingliang Liu added a comment - s/desirable/not desirable/ Sorry about this.
          Hide
          liuml07 Mingliang Liu added a comment -

          Nice catch, Konstantin Shvachko. The v5 patch is to address your latest two comments. Thanks!

          Show
          liuml07 Mingliang Liu added a comment - Nice catch, Konstantin Shvachko . The v5 patch is to address your latest two comments. Thanks!
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 8s Maven dependency ordering for branch
          +1 mvninstall 7m 9s trunk passed
          +1 compile 0m 42s trunk passed with JDK v1.8.0_66
          +1 compile 0m 41s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 22s trunk passed
          +1 mvnsite 1m 1s trunk passed
          +1 mvneclipse 0m 16s trunk passed
          +1 findbugs 1m 55s trunk passed
          +1 javadoc 1m 4s trunk passed with JDK v1.8.0_66
          +1 javadoc 1m 41s trunk passed with JDK v1.7.0_91
          0 mvndep 0m 8s Maven dependency ordering for patch
          +1 mvninstall 0m 43s the patch passed
          +1 compile 0m 35s the patch passed with JDK v1.8.0_66
          +1 javac 0m 35s the patch passed
          +1 compile 0m 37s the patch passed with JDK v1.7.0_91
          +1 javac 0m 37s the patch passed
          +1 checkstyle 0m 18s the patch passed
          +1 mvnsite 0m 48s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 8s the patch passed
          +1 javadoc 1m 3s the patch passed with JDK v1.8.0_66
          +1 javadoc 1m 43s the patch passed with JDK v1.7.0_91
          -1 unit 65m 21s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          -1 unit 62m 32s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
          +1 asflicense 0m 21s Patch does not generate ASF License warnings.
          153m 49s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestFileAppend
            hadoop.hdfs.server.datanode.TestBlockScanner
            hadoop.hdfs.shortcircuit.TestShortCircuitCache
          JDK v1.7.0_91 Failed junit tests hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot
            hadoop.hdfs.TestRollingUpgrade



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12785662/HDFS-9053.005.patch
          JIRA Issue HDFS-9503
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux c3b6db6eba0e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / ed55950
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14332/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14332/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14332/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14332/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14332/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Max memory used 77MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14332/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 8s Maven dependency ordering for branch +1 mvninstall 7m 9s trunk passed +1 compile 0m 42s trunk passed with JDK v1.8.0_66 +1 compile 0m 41s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 22s trunk passed +1 mvnsite 1m 1s trunk passed +1 mvneclipse 0m 16s trunk passed +1 findbugs 1m 55s trunk passed +1 javadoc 1m 4s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 41s trunk passed with JDK v1.7.0_91 0 mvndep 0m 8s Maven dependency ordering for patch +1 mvninstall 0m 43s the patch passed +1 compile 0m 35s the patch passed with JDK v1.8.0_66 +1 javac 0m 35s the patch passed +1 compile 0m 37s the patch passed with JDK v1.7.0_91 +1 javac 0m 37s the patch passed +1 checkstyle 0m 18s the patch passed +1 mvnsite 0m 48s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 8s the patch passed +1 javadoc 1m 3s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 43s the patch passed with JDK v1.7.0_91 -1 unit 65m 21s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 62m 32s hadoop-hdfs in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 21s Patch does not generate ASF License warnings. 153m 49s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestFileAppend   hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.shortcircuit.TestShortCircuitCache JDK v1.7.0_91 Failed junit tests hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot   hadoop.hdfs.TestRollingUpgrade Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12785662/HDFS-9053.005.patch JIRA Issue HDFS-9503 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux c3b6db6eba0e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ed55950 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/14332/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14332/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14332/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14332/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14332/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 77MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14332/console This message was automatically generated.
          Hide
          vinayrpet Vinayakumar B added a comment -

          This is incompatible change, not because of replacing -namenode with -fs ( because, 2.8 which has -namenode not released yet),
          But because, current patch will change the startup behavior.

          Before patch, if '-namenode' was not given, then benchmark used to run in standalone mode, still using the 'fs.defaultFS' conf for namenode's RPC address.
          Now with patch, if 'fs.defaultFS' is provided, either via configuration or '-fs', benchmark expects that as Remote NameNode.
          And to run standalone mode to test Replication op specifically, 'fs.defaultFS' conf needs to be removed from site xmls.

          If this is okay, may be its better to update the corresponding DOCs and Release Notes.

          Other than this, current changes in patch looks good to me.

          Show
          vinayrpet Vinayakumar B added a comment - This is incompatible change, not because of replacing -namenode with -fs ( because, 2.8 which has -namenode not released yet), But because, current patch will change the startup behavior. Before patch, if '-namenode' was not given, then benchmark used to run in standalone mode, still using the 'fs.defaultFS' conf for namenode's RPC address. Now with patch, if 'fs.defaultFS' is provided, either via configuration or '-fs', benchmark expects that as Remote NameNode. And to run standalone mode to test Replication op specifically, 'fs.defaultFS' conf needs to be removed from site xmls. If this is okay, may be its better to update the corresponding DOCs and Release Notes. Other than this, current changes in patch looks good to me.
          Hide
          shv Konstantin Shvachko added a comment -

          Hey Vinayakumar. The patch will indeed take fs.defaultFS from a config file and treat it as a remote NameNode, which is different from current behaviour. Although, it is easy to override the configuration value with the same -fs option just introduced. The following will use local NN no matter what is in the core-site.xml

          NNThroughtputBenchmark -conf core-site.xml -fs local -op mkdirs
          NNThroughtputBenchmark -conf core-site.xml -fs file:/// -op mkdirs
          

          I don't know if it is incompatible, just saying everything can be controlled on the command line for the benchmark.

          As I mentioned in HDFS-7847 I don't understand the use case for a remote NN in the context of NNThroughtputBenchmark, but at least now we will avoid introducing unnecessary options with this patch.

          +1 for the patch

          Show
          shv Konstantin Shvachko added a comment - Hey Vinayakumar. The patch will indeed take fs.defaultFS from a config file and treat it as a remote NameNode, which is different from current behaviour. Although, it is easy to override the configuration value with the same -fs option just introduced. The following will use local NN no matter what is in the core-site.xml NNThroughtputBenchmark -conf core-site.xml -fs local -op mkdirs NNThroughtputBenchmark -conf core-site.xml -fs file: /// -op mkdirs I don't know if it is incompatible, just saying everything can be controlled on the command line for the benchmark. As I mentioned in HDFS-7847 I don't understand the use case for a remote NN in the context of NNThroughtputBenchmark, but at least now we will avoid introducing unnecessary options with this patch. +1 for the patch
          Hide
          vinayrpet Vinayakumar B added a comment -

          I don't know if it is incompatible, just saying everything can be controlled on the command line for the benchmark.

          Yes, thats looks clean and easy.
          I am not opposing the change. Just conveying, its also better to update corresponding docs and release notes to reflect the behavior change.

          cc Arpit Agarwal, Colin P. McCabe, as they were involved in HDFS-7847

          Show
          vinayrpet Vinayakumar B added a comment - I don't know if it is incompatible, just saying everything can be controlled on the command line for the benchmark. Yes, thats looks clean and easy. I am not opposing the change. Just conveying, its also better to update corresponding docs and release notes to reflect the behavior change. cc Arpit Agarwal , Colin P. McCabe , as they were involved in HDFS-7847
          Hide
          shv Konstantin Shvachko added a comment -

          Hey Mingliang Liu I was trying to commit this to branch-2.8, but the test is failing for me with the following error:

          Running org.apache.hadoop.hdfs.server.namenode.TestNNThroughputBenchmark
          Tests run: 4, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 35.989 sec <<< FAILURE! - in org.apache.hadoop.hdfs.server.namenode.TestNNThroughputBenchmark
          testNNThroughput(org.apache.hadoop.hdfs.server.namenode.TestNNThroughputBenchmark)  Time elapsed: 1.838 sec  <<< ERROR!
          org.apache.hadoop.hdfs.server.namenode.NotReplicatedYetException: Not replicated yet: /nnThroughputBenchmark/blockReport/ThroughputBenchDir0/ThroughputBench8
          	at org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.validateAddBlock(FSDirWriteFileOp.java:188)
          	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2405)
          	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:793)
          	at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$BlockReportStats.addBlocks(NNThroughputBenchmark.java:1175)
          	at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$BlockReportStats.generateInputs(NNThroughputBenchmark.java:1162)
          	at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$OperationStatsBase.benchmark(NNThroughputBenchmark.java:280)
          	at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.run(NNThroughputBenchmark.java:1515)
          	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
          	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
          	at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.runBenchmark(NNThroughputBenchmark.java:1417)
          	at org.apache.hadoop.hdfs.server.namenode.TestNNThroughputBenchmark.testNNThroughput(TestNNThroughputBenchmark.java:56)
          

          Could you please verify this.

          Show
          shv Konstantin Shvachko added a comment - Hey Mingliang Liu I was trying to commit this to branch-2.8, but the test is failing for me with the following error: Running org.apache.hadoop.hdfs.server.namenode.TestNNThroughputBenchmark Tests run: 4, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 35.989 sec <<< FAILURE! - in org.apache.hadoop.hdfs.server.namenode.TestNNThroughputBenchmark testNNThroughput(org.apache.hadoop.hdfs.server.namenode.TestNNThroughputBenchmark) Time elapsed: 1.838 sec <<< ERROR! org.apache.hadoop.hdfs.server.namenode.NotReplicatedYetException: Not replicated yet: /nnThroughputBenchmark/blockReport/ThroughputBenchDir0/ThroughputBench8 at org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.validateAddBlock(FSDirWriteFileOp.java:188) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2405) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:793) at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$BlockReportStats.addBlocks(NNThroughputBenchmark.java:1175) at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$BlockReportStats.generateInputs(NNThroughputBenchmark.java:1162) at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$OperationStatsBase.benchmark(NNThroughputBenchmark.java:280) at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.run(NNThroughputBenchmark.java:1515) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90) at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.runBenchmark(NNThroughputBenchmark.java:1417) at org.apache.hadoop.hdfs.server.namenode.TestNNThroughputBenchmark.testNNThroughput(TestNNThroughputBenchmark.java:56) Could you please verify this.
          Hide
          liuml07 Mingliang Liu added a comment -

          Hi Konstantin Shvachko, I believe this is an independent issue which was addressed in HDFS-9601, which is not cherry-picked in branch-2.8.

          Show
          liuml07 Mingliang Liu added a comment - Hi Konstantin Shvachko , I believe this is an independent issue which was addressed in HDFS-9601 , which is not cherry-picked in branch-2.8 .
          Hide
          shv Konstantin Shvachko added a comment -

          Committing HDFS-9601 solved the problem I saw on branch-2.8.

          I just committed this to trunk, branch-2, and branch-2.8. Thank you Mingliang Liu.

          Show
          shv Konstantin Shvachko added a comment - Committing HDFS-9601 solved the problem I saw on branch-2.8. I just committed this to trunk, branch-2, and branch-2.8. Thank you Mingliang Liu .
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9242 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9242/)
          HDFS-9503. Use generic option -fs for NNThroughputBenchmark instead of (shv: rev 81ef1a1f55ee0104f267316e44596463a78f87ea)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNNThroughputBenchmark.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9242 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9242/ ) HDFS-9503 . Use generic option -fs for NNThroughputBenchmark instead of (shv: rev 81ef1a1f55ee0104f267316e44596463a78f87ea) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNNThroughputBenchmark.java
          Hide
          liuml07 Mingliang Liu added a comment -

          Thank you Konstantin Shvachko for your review and commit. Thanks Vinayakumar B for discussion and review.

          p. s. I added brief release notes. Feel free to correct/refine it if needed.

          Show
          liuml07 Mingliang Liu added a comment - Thank you Konstantin Shvachko for your review and commit. Thanks Vinayakumar B for discussion and review. p. s. I added brief release notes. Feel free to correct/refine it if needed.
          Hide
          shv Konstantin Shvachko added a comment -

          Adding HDFS-7847, HDFS-9421, and HDFS-9503 to branch-2.7.

          Show
          shv Konstantin Shvachko added a comment - Adding HDFS-7847 , HDFS-9421 , and HDFS-9503 to branch-2.7.

            People

            • Assignee:
              liuml07 Mingliang Liu
              Reporter:
              shv Konstantin Shvachko
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development