Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9557

Reduce object allocation in PB conversion

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.8.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: hdfs-client
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      PB conversions use ByteString.copyFrom to populate the builder. Unfortunately this creates unique instances for empty arrays instead of returning the singleton ByteString.EMPTY.

      1. HDFS-9557.patch
        7 kB
        Daryn Sharp
      2. HDFS-9557.patch
        7 kB
        Daryn Sharp

        Activity

        Hide
        cmccabe Colin P. McCabe added a comment -

        Thanks, Daryn Sharp.

        Show
        cmccabe Colin P. McCabe added a comment - Thanks, Daryn Sharp .
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-trunk-Commit #8976 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8976/)
        HDFS-9557. Reduce object allocation in PB conversion. Contributed by (cnauroth: rev c470c8953d4927043b6383fad8e792289c634c09)

        • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelperClient.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocolPB/TestPBHelper.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #8976 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8976/ ) HDFS-9557 . Reduce object allocation in PB conversion. Contributed by (cnauroth: rev c470c8953d4927043b6383fad8e792289c634c09) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelperClient.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocolPB/TestPBHelper.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
        Hide
        cnauroth Chris Nauroth added a comment -

        I took a closer look, and those tests are very unpredictable, with or without this patch. sigh

        +1, and committed to trunk, branch-2 and branch-2.8. Thank you, Daryn.

        Show
        cnauroth Chris Nauroth added a comment - I took a closer look, and those tests are very unpredictable, with or without this patch. sigh +1, and committed to trunk, branch-2 and branch-2.8. Thank you, Daryn.
        Hide
        daryn Daryn Sharp added a comment -

        Seem to work for me. It didn't find classes and got permission denied for libc???

        Show
        daryn Daryn Sharp added a comment - Seem to work for me. It didn't find classes and got permission denied for libc???
        Hide
        cnauroth Chris Nauroth added a comment -

        Daryn, could you please review the test failures? I quickly ruled out the javac and Checkstyle warnings. They appear to be unrelated/unaddressable within the scope of this patch.

        Show
        cnauroth Chris Nauroth added a comment - Daryn, could you please review the test failures? I quickly ruled out the javac and Checkstyle warnings. They appear to be unrelated/unaddressable within the scope of this patch.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 11m 3s trunk passed
        +1 compile 2m 40s trunk passed with JDK v1.8.0_66
        +1 compile 2m 16s trunk passed with JDK v1.7.0_91
        +1 checkstyle 0m 30s trunk passed
        +1 mvnsite 2m 6s trunk passed
        +1 mvneclipse 0m 40s trunk passed
        +1 findbugs 5m 6s trunk passed
        +1 javadoc 2m 14s trunk passed with JDK v1.8.0_66
        +1 javadoc 3m 9s trunk passed with JDK v1.7.0_91
        +1 mvninstall 1m 57s the patch passed
        +1 compile 2m 33s the patch passed with JDK v1.8.0_66
        -1 javac 12m 29s hadoop-hdfs-project-jdk1.8.0_66 with JDK v1.8.0_66 generated 3 new issues (was 48, now 48).
        +1 javac 2m 33s the patch passed
        +1 compile 2m 18s the patch passed with JDK v1.7.0_91
        -1 javac 14m 48s hadoop-hdfs-project-jdk1.7.0_91 with JDK v1.7.0_91 generated 3 new issues (was 50, now 50).
        +1 javac 2m 18s the patch passed
        -1 checkstyle 0m 27s Patch generated 1 new checkstyle issues in hadoop-hdfs-project (total was 91, now 91).
        +1 mvnsite 2m 0s the patch passed
        +1 mvneclipse 0m 39s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 5m 26s the patch passed
        +1 javadoc 2m 21s the patch passed with JDK v1.8.0_66
        +1 javadoc 3m 14s the patch passed with JDK v1.7.0_91
        +1 unit 1m 25s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66.
        -1 unit 94m 34s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
        +1 unit 1m 23s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91.
        -1 unit 80m 42s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
        -1 asflicense 0m 31s Patch generated 56 ASF License warnings.
        233m 30s



        Reason Tests
        JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestHFlush
          hadoop.hdfs.TestReadStripedFileWithDecoding
          hadoop.hdfs.security.TestDelegationTokenForProxyUser
          hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
          hadoop.hdfs.server.datanode.TestDirectoryScanner
          hadoop.hdfs.server.namenode.ha.TestEditLogTailer
        JDK v1.7.0_91 Failed junit tests hadoop.hdfs.TestFileCreationDelete
          hadoop.hdfs.server.datanode.TestDirectoryScanner
          hadoop.hdfs.server.blockmanagement.TestReplicationPolicy



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12777848/HDFS-9557.patch
        JIRA Issue HDFS-9557
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 513e95b001e9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / ae67c84
        findbugs v3.0.0
        javac hadoop-hdfs-project-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-HDFS-Build/13888/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project-jdk1.8.0_66.txt
        javac hadoop-hdfs-project-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-HDFS-Build/13888/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project-jdk1.7.0_91.txt
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13888/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/13888/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/13888/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13888/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13888/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13888/testReport/
        asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13888/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project
        Max memory used 75MB
        Powered by Apache Yetus 0.1.0 http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13888/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 11m 3s trunk passed +1 compile 2m 40s trunk passed with JDK v1.8.0_66 +1 compile 2m 16s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 30s trunk passed +1 mvnsite 2m 6s trunk passed +1 mvneclipse 0m 40s trunk passed +1 findbugs 5m 6s trunk passed +1 javadoc 2m 14s trunk passed with JDK v1.8.0_66 +1 javadoc 3m 9s trunk passed with JDK v1.7.0_91 +1 mvninstall 1m 57s the patch passed +1 compile 2m 33s the patch passed with JDK v1.8.0_66 -1 javac 12m 29s hadoop-hdfs-project-jdk1.8.0_66 with JDK v1.8.0_66 generated 3 new issues (was 48, now 48). +1 javac 2m 33s the patch passed +1 compile 2m 18s the patch passed with JDK v1.7.0_91 -1 javac 14m 48s hadoop-hdfs-project-jdk1.7.0_91 with JDK v1.7.0_91 generated 3 new issues (was 50, now 50). +1 javac 2m 18s the patch passed -1 checkstyle 0m 27s Patch generated 1 new checkstyle issues in hadoop-hdfs-project (total was 91, now 91). +1 mvnsite 2m 0s the patch passed +1 mvneclipse 0m 39s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 5m 26s the patch passed +1 javadoc 2m 21s the patch passed with JDK v1.8.0_66 +1 javadoc 3m 14s the patch passed with JDK v1.7.0_91 +1 unit 1m 25s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66. -1 unit 94m 34s hadoop-hdfs in the patch failed with JDK v1.8.0_66. +1 unit 1m 23s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91. -1 unit 80m 42s hadoop-hdfs in the patch failed with JDK v1.7.0_91. -1 asflicense 0m 31s Patch generated 56 ASF License warnings. 233m 30s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestHFlush   hadoop.hdfs.TestReadStripedFileWithDecoding   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes   hadoop.hdfs.server.datanode.TestDirectoryScanner   hadoop.hdfs.server.namenode.ha.TestEditLogTailer JDK v1.7.0_91 Failed junit tests hadoop.hdfs.TestFileCreationDelete   hadoop.hdfs.server.datanode.TestDirectoryScanner   hadoop.hdfs.server.blockmanagement.TestReplicationPolicy Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12777848/HDFS-9557.patch JIRA Issue HDFS-9557 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 513e95b001e9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ae67c84 findbugs v3.0.0 javac hadoop-hdfs-project-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-HDFS-Build/13888/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project-jdk1.8.0_66.txt javac hadoop-hdfs-project-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-HDFS-Build/13888/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project-jdk1.7.0_91.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13888/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13888/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13888/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13888/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13888/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13888/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13888/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project Max memory used 75MB Powered by Apache Yetus 0.1.0 http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13888/console This message was automatically generated.
        Hide
        daryn Daryn Sharp added a comment -

        You'll soon be delighted to know I'll have a patch up this week that uses PB decoding in a far more efficient manner.

        Show
        daryn Daryn Sharp added a comment - You'll soon be delighted to know I'll have a patch up this week that uses PB decoding in a far more efficient manner.
        Hide
        daryn Daryn Sharp added a comment -

        Ug, when one last search-n-replace goes horribly wrong... Yes. Will repost.

        Show
        daryn Daryn Sharp added a comment - Ug, when one last search-n-replace goes horribly wrong... Yes. Will repost.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 7m 28s trunk passed
        +1 compile 1m 27s trunk passed with JDK v1.8.0_66
        +1 compile 1m 36s trunk passed with JDK v1.7.0_91
        +1 checkstyle 0m 21s trunk passed
        +1 mvnsite 1m 23s trunk passed
        +1 mvneclipse 0m 25s trunk passed
        +1 findbugs 3m 41s trunk passed
        +1 javadoc 1m 23s trunk passed with JDK v1.8.0_66
        +1 javadoc 2m 15s trunk passed with JDK v1.7.0_91
        +1 mvninstall 1m 20s the patch passed
        +1 compile 1m 28s the patch passed with JDK v1.8.0_66
        -1 javac 8m 19s hadoop-hdfs-project-jdk1.8.0_66 with JDK v1.8.0_66 generated 3 new issues (was 48, now 48).
        +1 javac 1m 28s the patch passed
        +1 compile 1m 34s the patch passed with JDK v1.7.0_91
        -1 javac 9m 53s hadoop-hdfs-project-jdk1.7.0_91 with JDK v1.7.0_91 generated 3 new issues (was 50, now 50).
        +1 javac 1m 34s the patch passed
        -1 checkstyle 0m 21s Patch generated 1 new checkstyle issues in hadoop-hdfs-project (total was 91, now 91).
        +1 mvnsite 1m 25s the patch passed
        +1 mvneclipse 0m 26s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        -1 findbugs 2m 3s hadoop-hdfs-project/hadoop-hdfs-client introduced 1 new FindBugs issues.
        +1 javadoc 1m 26s the patch passed with JDK v1.8.0_66
        +1 javadoc 2m 11s the patch passed with JDK v1.7.0_91
        +1 unit 0m 51s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66.
        -1 unit 49m 34s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
        +1 unit 0m 58s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91.
        -1 unit 48m 37s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
        -1 asflicense 0m 19s Patch generated 58 ASF License warnings.
        137m 24s



        Reason Tests
        FindBugs module:hadoop-hdfs-project/hadoop-hdfs-client
          There is an apparent infinite recursive loop in org.apache.hadoop.hdfs.protocolPB.PBHelperClient.getByteString(byte[]) At PBHelperClient.java:recursive loop in org.apache.hadoop.hdfs.protocolPB.PBHelperClient.getByteString(byte[]) At PBHelperClient.java:[line 198]
        JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.namenode.TestRecoverStripedBlocks
          hadoop.fs.viewfs.TestViewFsAtHdfsRoot
          hadoop.cli.TestCryptoAdminCLI
          hadoop.hdfs.qjournal.client.TestQuorumJournalManager
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060
          hadoop.hdfs.TestDFSStripedOutputStream
          hadoop.hdfs.TestEncryptionZonesWithKMS
          hadoop.fs.contract.hdfs.TestHDFSContractRename
          hadoop.fs.viewfs.TestViewFsWithXAttrs
          hadoop.fs.viewfs.TestViewFileSystemWithXAttrs
          hadoop.hdfs.server.namenode.TestCheckpoint
          hadoop.hdfs.server.namenode.TestQuotaWithStripedBlocks
          hadoop.hdfs.TestErasureCodeBenchmarkThroughput
          hadoop.hdfs.server.namenode.snapshot.TestXAttrWithSnapshot
          hadoop.cli.TestErasureCodingCLI
          hadoop.hdfs.server.namenode.TestAddStripedBlocks
          hadoop.hdfs.TestHDFSTrash
          hadoop.hdfs.tools.TestDelegationTokenFetcher
          hadoop.hdfs.TestAclsEndToEnd
          hadoop.hdfs.server.namenode.TestQuotaByStorageType
          hadoop.hdfs.TestSnapshotCommands
          hadoop.hdfs.TestSafeModeWithStripedFile
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050
          hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped
          hadoop.hdfs.TestEncryptionZonesWithHA
          hadoop.hdfs.qjournal.server.TestJournalNode
          hadoop.hdfs.TestReservedRawPaths
          hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer
          hadoop.hdfs.server.namenode.TestFSImageWithXAttr
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130
          hadoop.hdfs.server.namenode.snapshot.TestSnapshottableDirListing
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure170
          hadoop.hdfs.qjournal.TestSecureNNWithQJM
          hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA
          hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks
          hadoop.hdfs.server.namenode.ha.TestDelegationTokensWithHA
          hadoop.hdfs.server.mover.TestStorageMover
          hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
          hadoop.hdfs.web.TestWebHDFS
          hadoop.fs.contract.hdfs.TestHDFSContractRootDirectory
          hadoop.hdfs.server.mover.TestMover
          hadoop.hdfs.security.TestDelegationToken
          hadoop.hdfs.server.namenode.snapshot.TestSnapshotMetrics
          hadoop.fs.TestHDFSFileContextMainOperations
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure210
          hadoop.hdfs.TestDistributedFileSystem
          hadoop.hdfs.TestEncryptionZones
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180
          hadoop.hdfs.qjournal.TestNNWithQJM
          hadoop.hdfs.server.namenode.TestNameNodeXAttr
          hadoop.hdfs.server.namenode.snapshot.TestSetQuotaWithSnapshot
          hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA
          hadoop.hdfs.TestErasureCodingPolicyWithSnapshot
          hadoop.hdfs.server.namenode.TestFsck
          hadoop.hdfs.qjournal.client.TestQJMWithFaults
          hadoop.fs.permission.TestStickyBit
          hadoop.hdfs.server.namenode.TestStripedINodeFile
          hadoop.hdfs.TestRollingUpgrade
          hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock
          hadoop.hdfs.server.balancer.TestBalancer
          hadoop.hdfs.TestBlockStoragePolicy
          hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
          hadoop.hdfs.TestReadStripedFileWithMissingBlocks
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140
          hadoop.fs.TestResolveHdfsSymlink
          hadoop.cli.TestXAttrCLI
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090
          hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits
          hadoop.hdfs.server.namenode.TestFSImage
          hadoop.hdfs.TestDFSStripedInputStream
          hadoop.hdfs.server.namenode.ha.TestXAttrsWithHA
          hadoop.hdfs.TestFileStatus
          hadoop.hdfs.server.namenode.ha.TestBootstrapStandbyWithQJM
          hadoop.hdfs.TestDatanodeLayoutUpgrade
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure040
          hadoop.hdfs.TestReadStripedFileWithDecoding
          hadoop.hdfs.TestRecoverStripedFile
          hadoop.fs.viewfs.TestViewFileSystemAtHdfsRoot
          hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport
          hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForXAttr
          hadoop.hdfs.TestListFilesInFileContext
          hadoop.hdfs.TestErasureCodingPolicies
          hadoop.hdfs.server.namenode.ha.TestFailoverWithBlockTokensEnabled
          hadoop.hdfs.TestAppendSnapshotTruncate
          hadoop.fs.viewfs.TestViewFileSystemHdfs
          hadoop.security.TestPermissionSymlinks
          hadoop.hdfs.TestRollingUpgradeRollback
          hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS
          hadoop.hdfs.TestDFSUpgradeFromImage
          hadoop.hdfs.TestHDFSFileSystemContract
          hadoop.hdfs.server.namenode.TestFSImageWithSnapshot
          hadoop.hdfs.TestEncryptedTransfer
          hadoop.hdfs.server.namenode.TestINodeAttributeProvider
          hadoop.hdfs.server.blockmanagement.TestSequentialBlockGroupId
          hadoop.fs.viewfs.TestViewFsHdfs
          hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer
          hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
          hadoop.hdfs.TestFileStatusWithECPolicy
          hadoop.hdfs.server.namenode.TestFileContextXAttr
          hadoop.hdfs.security.token.block.TestBlockToken
          hadoop.hdfs.protocolPB.TestPBHelper
          hadoop.hdfs.server.namenode.snapshot.TestSnapshotListing
          hadoop.hdfs.security.TestDelegationTokenForProxyUser
          hadoop.cli.TestHDFSCLI
          hadoop.hdfs.web.TestWebHDFSXAttr
          hadoop.hdfs.TestListFilesInDFS
          hadoop.hdfs.server.namenode.TestNamenodeRetryCache
          hadoop.hdfs.server.namenode.TestFSEditLogLoader
          hadoop.fs.loadGenerator.TestLoadGenerator
          hadoop.cli.TestAclCLI
          hadoop.fs.TestGlobPaths
          hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerWithStripedBlocks
          hadoop.hdfs.TestDatanodeStartupFixesLegacyStorageIDs
          hadoop.hdfs.server.namenode.TestBackupNode
          hadoop.hdfs.server.namenode.TestINodeFile
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070
          hadoop.hdfs.TestParallelShortCircuitReadUnCached
          hadoop.hdfs.TestDFSShell
          hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer
          hadoop.hdfs.TestDFSRename
          hadoop.fs.TestSymlinkHdfsFileSystem
          hadoop.hdfs.TestWriteReadStripedFile
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure190
          hadoop.hdfs.server.namenode.snapshot.TestSnapshot
          hadoop.hdfs.TestRollingUpgradeDowngrade
          hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure
          hadoop.hdfs.TestDFSInotifyEventInputStream
          hadoop.fs.TestSymlinkHdfsFileContext
        JDK v1.7.0_91 Failed junit tests hadoop.hdfs.server.namenode.TestRecoverStripedBlocks
          hadoop.fs.viewfs.TestViewFsAtHdfsRoot
          hadoop.cli.TestCryptoAdminCLI
          hadoop.hdfs.qjournal.client.TestQuorumJournalManager
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060
          hadoop.hdfs.TestDFSStripedOutputStream
          hadoop.hdfs.TestEncryptionZonesWithKMS
          hadoop.fs.contract.hdfs.TestHDFSContractRename
          hadoop.fs.viewfs.TestViewFsWithXAttrs
          hadoop.fs.viewfs.TestViewFileSystemWithXAttrs
          hadoop.hdfs.server.namenode.TestCheckpoint
          hadoop.hdfs.server.namenode.TestQuotaWithStripedBlocks
          hadoop.hdfs.TestErasureCodeBenchmarkThroughput
          hadoop.hdfs.server.namenode.snapshot.TestXAttrWithSnapshot
          hadoop.cli.TestErasureCodingCLI
          hadoop.hdfs.server.namenode.TestAddStripedBlocks
          hadoop.hdfs.TestHDFSTrash
          hadoop.hdfs.tools.TestDelegationTokenFetcher
          hadoop.hdfs.TestAclsEndToEnd
          hadoop.hdfs.server.namenode.TestQuotaByStorageType
          hadoop.hdfs.TestSnapshotCommands
          hadoop.hdfs.TestSafeModeWithStripedFile
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050
          hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped
          hadoop.hdfs.TestEncryptionZonesWithHA
          hadoop.hdfs.qjournal.server.TestJournalNode
          hadoop.hdfs.TestReservedRawPaths
          hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer
          hadoop.hdfs.server.namenode.TestFSImageWithXAttr
          hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130
          hadoop.hdfs.server.namenode.snapshot.TestSnapshottableDirListing
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure170
          hadoop.hdfs.qjournal.TestSecureNNWithQJM
          hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA
          hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks
          hadoop.hdfs.server.namenode.ha.TestDelegationTokensWithHA
          hadoop.hdfs.server.mover.TestStorageMover
          hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
          hadoop.hdfs.web.TestWebHDFS
          hadoop.fs.contract.hdfs.TestHDFSContractRootDirectory
          hadoop.hdfs.server.mover.TestMover
          hadoop.hdfs.security.TestDelegationToken
          hadoop.hdfs.server.namenode.snapshot.TestSnapshotMetrics
          hadoop.fs.TestHDFSFileContextMainOperations
          hadoop.hdfs.TestDistributedFileSystem
          hadoop.hdfs.TestEncryptionZones
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180
          hadoop.hdfs.qjournal.TestNNWithQJM
          hadoop.hdfs.server.namenode.TestNameNodeXAttr
          hadoop.hdfs.server.namenode.snapshot.TestSetQuotaWithSnapshot
          hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA
          hadoop.hdfs.TestErasureCodingPolicyWithSnapshot
          hadoop.hdfs.server.namenode.TestFsck
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure030
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160
          hadoop.hdfs.qjournal.client.TestQJMWithFaults
          hadoop.fs.permission.TestStickyBit
          hadoop.fs.TestFcHdfsSetUMask
          hadoop.hdfs.server.namenode.TestStripedINodeFile
          hadoop.hdfs.TestRollingUpgrade
          hadoop.hdfs.server.balancer.TestBalancer
          hadoop.hdfs.TestBlockStoragePolicy
          hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
          hadoop.hdfs.TestReadStripedFileWithMissingBlocks
          hadoop.fs.TestResolveHdfsSymlink
          hadoop.cli.TestXAttrCLI
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090
          hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits
          hadoop.hdfs.server.namenode.TestFSImage
          hadoop.hdfs.TestDFSStripedInputStream
          hadoop.hdfs.server.namenode.ha.TestXAttrsWithHA
          hadoop.hdfs.TestFileStatus
          hadoop.hdfs.server.namenode.ha.TestBootstrapStandbyWithQJM
          hadoop.hdfs.TestDatanodeLayoutUpgrade
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure040
          hadoop.hdfs.TestReadStripedFileWithDecoding
          hadoop.hdfs.TestRecoverStripedFile
          hadoop.fs.viewfs.TestViewFileSystemAtHdfsRoot
          hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport
          hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForXAttr
          hadoop.hdfs.TestListFilesInFileContext
          hadoop.hdfs.TestErasureCodingPolicies
          hadoop.hdfs.server.namenode.ha.TestFailoverWithBlockTokensEnabled
          hadoop.hdfs.TestAppendSnapshotTruncate
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010
          hadoop.fs.viewfs.TestViewFileSystemHdfs
          hadoop.security.TestPermissionSymlinks
          hadoop.hdfs.TestRollingUpgradeRollback
          hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS
          hadoop.hdfs.TestDFSUpgradeFromImage
          hadoop.hdfs.TestHDFSFileSystemContract
          hadoop.hdfs.server.namenode.TestFSImageWithSnapshot
          hadoop.hdfs.TestEncryptedTransfer
          hadoop.hdfs.server.namenode.TestINodeAttributeProvider
          hadoop.hdfs.server.blockmanagement.TestSequentialBlockGroupId
          hadoop.fs.viewfs.TestViewFsHdfs
          hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer
          hadoop.hdfs.TestFileStatusWithECPolicy
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure120
          hadoop.hdfs.server.namenode.TestFileContextXAttr
          hadoop.hdfs.security.token.block.TestBlockToken
          hadoop.hdfs.protocolPB.TestPBHelper
          hadoop.hdfs.server.namenode.snapshot.TestSnapshotListing
          hadoop.hdfs.security.TestDelegationTokenForProxyUser
          hadoop.cli.TestHDFSCLI
          hadoop.hdfs.web.TestWebHDFSXAttr
          hadoop.hdfs.TestListFilesInDFS
          hadoop.hdfs.server.namenode.TestNamenodeRetryCache
          hadoop.hdfs.server.namenode.TestFSEditLogLoader
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200
          hadoop.fs.loadGenerator.TestLoadGenerator
          hadoop.cli.TestAclCLI
          hadoop.fs.TestGlobPaths
          hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerWithStripedBlocks
          hadoop.hdfs.TestDatanodeStartupFixesLegacyStorageIDs
          hadoop.hdfs.server.namenode.TestBackupNode
          hadoop.hdfs.server.namenode.TestINodeFile
          hadoop.hdfs.TestParallelShortCircuitReadUnCached
          hadoop.hdfs.TestDFSShell
          hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer
          hadoop.hdfs.TestDFSRename
          hadoop.fs.TestSymlinkHdfsFileSystem
          hadoop.hdfs.TestWriteReadStripedFile
          hadoop.hdfs.server.namenode.snapshot.TestSnapshot
          hadoop.hdfs.TestRollingUpgradeDowngrade
          hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure
          hadoop.hdfs.TestDFSInotifyEventInputStream
          hadoop.fs.TestSymlinkHdfsFileContext



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12777804/HDFS-9557.patch
        JIRA Issue HDFS-9557
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 48b1b129fac6 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 8602692
        findbugs v3.0.0
        javac hadoop-hdfs-project-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-HDFS-Build/13880/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project-jdk1.8.0_66.txt
        javac hadoop-hdfs-project-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-HDFS-Build/13880/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project-jdk1.7.0_91.txt
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13880/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
        findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13880/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs-client.html
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/13880/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/13880/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13880/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13880/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13880/testReport/
        asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13880/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project
        Max memory used 76MB
        Powered by Apache Yetus 0.1.0 http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13880/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 28s trunk passed +1 compile 1m 27s trunk passed with JDK v1.8.0_66 +1 compile 1m 36s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 21s trunk passed +1 mvnsite 1m 23s trunk passed +1 mvneclipse 0m 25s trunk passed +1 findbugs 3m 41s trunk passed +1 javadoc 1m 23s trunk passed with JDK v1.8.0_66 +1 javadoc 2m 15s trunk passed with JDK v1.7.0_91 +1 mvninstall 1m 20s the patch passed +1 compile 1m 28s the patch passed with JDK v1.8.0_66 -1 javac 8m 19s hadoop-hdfs-project-jdk1.8.0_66 with JDK v1.8.0_66 generated 3 new issues (was 48, now 48). +1 javac 1m 28s the patch passed +1 compile 1m 34s the patch passed with JDK v1.7.0_91 -1 javac 9m 53s hadoop-hdfs-project-jdk1.7.0_91 with JDK v1.7.0_91 generated 3 new issues (was 50, now 50). +1 javac 1m 34s the patch passed -1 checkstyle 0m 21s Patch generated 1 new checkstyle issues in hadoop-hdfs-project (total was 91, now 91). +1 mvnsite 1m 25s the patch passed +1 mvneclipse 0m 26s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. -1 findbugs 2m 3s hadoop-hdfs-project/hadoop-hdfs-client introduced 1 new FindBugs issues. +1 javadoc 1m 26s the patch passed with JDK v1.8.0_66 +1 javadoc 2m 11s the patch passed with JDK v1.7.0_91 +1 unit 0m 51s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66. -1 unit 49m 34s hadoop-hdfs in the patch failed with JDK v1.8.0_66. +1 unit 0m 58s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91. -1 unit 48m 37s hadoop-hdfs in the patch failed with JDK v1.7.0_91. -1 asflicense 0m 19s Patch generated 58 ASF License warnings. 137m 24s Reason Tests FindBugs module:hadoop-hdfs-project/hadoop-hdfs-client   There is an apparent infinite recursive loop in org.apache.hadoop.hdfs.protocolPB.PBHelperClient.getByteString(byte[]) At PBHelperClient.java:recursive loop in org.apache.hadoop.hdfs.protocolPB.PBHelperClient.getByteString(byte[]) At PBHelperClient.java: [line 198] JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.namenode.TestRecoverStripedBlocks   hadoop.fs.viewfs.TestViewFsAtHdfsRoot   hadoop.cli.TestCryptoAdminCLI   hadoop.hdfs.qjournal.client.TestQuorumJournalManager   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060   hadoop.hdfs.TestDFSStripedOutputStream   hadoop.hdfs.TestEncryptionZonesWithKMS   hadoop.fs.contract.hdfs.TestHDFSContractRename   hadoop.fs.viewfs.TestViewFsWithXAttrs   hadoop.fs.viewfs.TestViewFileSystemWithXAttrs   hadoop.hdfs.server.namenode.TestCheckpoint   hadoop.hdfs.server.namenode.TestQuotaWithStripedBlocks   hadoop.hdfs.TestErasureCodeBenchmarkThroughput   hadoop.hdfs.server.namenode.snapshot.TestXAttrWithSnapshot   hadoop.cli.TestErasureCodingCLI   hadoop.hdfs.server.namenode.TestAddStripedBlocks   hadoop.hdfs.TestHDFSTrash   hadoop.hdfs.tools.TestDelegationTokenFetcher   hadoop.hdfs.TestAclsEndToEnd   hadoop.hdfs.server.namenode.TestQuotaByStorageType   hadoop.hdfs.TestSnapshotCommands   hadoop.hdfs.TestSafeModeWithStripedFile   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050   hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped   hadoop.hdfs.TestEncryptionZonesWithHA   hadoop.hdfs.qjournal.server.TestJournalNode   hadoop.hdfs.TestReservedRawPaths   hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer   hadoop.hdfs.server.namenode.TestFSImageWithXAttr   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130   hadoop.hdfs.server.namenode.snapshot.TestSnapshottableDirListing   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure170   hadoop.hdfs.qjournal.TestSecureNNWithQJM   hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA   hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks   hadoop.hdfs.server.namenode.ha.TestDelegationTokensWithHA   hadoop.hdfs.server.mover.TestStorageMover   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots   hadoop.hdfs.web.TestWebHDFS   hadoop.fs.contract.hdfs.TestHDFSContractRootDirectory   hadoop.hdfs.server.mover.TestMover   hadoop.hdfs.security.TestDelegationToken   hadoop.hdfs.server.namenode.snapshot.TestSnapshotMetrics   hadoop.fs.TestHDFSFileContextMainOperations   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure210   hadoop.hdfs.TestDistributedFileSystem   hadoop.hdfs.TestEncryptionZones   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180   hadoop.hdfs.qjournal.TestNNWithQJM   hadoop.hdfs.server.namenode.TestNameNodeXAttr   hadoop.hdfs.server.namenode.snapshot.TestSetQuotaWithSnapshot   hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA   hadoop.hdfs.TestErasureCodingPolicyWithSnapshot   hadoop.hdfs.server.namenode.TestFsck   hadoop.hdfs.qjournal.client.TestQJMWithFaults   hadoop.fs.permission.TestStickyBit   hadoop.hdfs.server.namenode.TestStripedINodeFile   hadoop.hdfs.TestRollingUpgrade   hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock   hadoop.hdfs.server.balancer.TestBalancer   hadoop.hdfs.TestBlockStoragePolicy   hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer   hadoop.hdfs.TestReadStripedFileWithMissingBlocks   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140   hadoop.fs.TestResolveHdfsSymlink   hadoop.cli.TestXAttrCLI   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090   hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits   hadoop.hdfs.server.namenode.TestFSImage   hadoop.hdfs.TestDFSStripedInputStream   hadoop.hdfs.server.namenode.ha.TestXAttrsWithHA   hadoop.hdfs.TestFileStatus   hadoop.hdfs.server.namenode.ha.TestBootstrapStandbyWithQJM   hadoop.hdfs.TestDatanodeLayoutUpgrade   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure040   hadoop.hdfs.TestReadStripedFileWithDecoding   hadoop.hdfs.TestRecoverStripedFile   hadoop.fs.viewfs.TestViewFileSystemAtHdfsRoot   hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport   hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForXAttr   hadoop.hdfs.TestListFilesInFileContext   hadoop.hdfs.TestErasureCodingPolicies   hadoop.hdfs.server.namenode.ha.TestFailoverWithBlockTokensEnabled   hadoop.hdfs.TestAppendSnapshotTruncate   hadoop.fs.viewfs.TestViewFileSystemHdfs   hadoop.security.TestPermissionSymlinks   hadoop.hdfs.TestRollingUpgradeRollback   hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS   hadoop.hdfs.TestDFSUpgradeFromImage   hadoop.hdfs.TestHDFSFileSystemContract   hadoop.hdfs.server.namenode.TestFSImageWithSnapshot   hadoop.hdfs.TestEncryptedTransfer   hadoop.hdfs.server.namenode.TestINodeAttributeProvider   hadoop.hdfs.server.blockmanagement.TestSequentialBlockGroupId   hadoop.fs.viewfs.TestViewFsHdfs   hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer   hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes   hadoop.hdfs.TestFileStatusWithECPolicy   hadoop.hdfs.server.namenode.TestFileContextXAttr   hadoop.hdfs.security.token.block.TestBlockToken   hadoop.hdfs.protocolPB.TestPBHelper   hadoop.hdfs.server.namenode.snapshot.TestSnapshotListing   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.cli.TestHDFSCLI   hadoop.hdfs.web.TestWebHDFSXAttr   hadoop.hdfs.TestListFilesInDFS   hadoop.hdfs.server.namenode.TestNamenodeRetryCache   hadoop.hdfs.server.namenode.TestFSEditLogLoader   hadoop.fs.loadGenerator.TestLoadGenerator   hadoop.cli.TestAclCLI   hadoop.fs.TestGlobPaths   hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerWithStripedBlocks   hadoop.hdfs.TestDatanodeStartupFixesLegacyStorageIDs   hadoop.hdfs.server.namenode.TestBackupNode   hadoop.hdfs.server.namenode.TestINodeFile   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070   hadoop.hdfs.TestParallelShortCircuitReadUnCached   hadoop.hdfs.TestDFSShell   hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer   hadoop.hdfs.TestDFSRename   hadoop.fs.TestSymlinkHdfsFileSystem   hadoop.hdfs.TestWriteReadStripedFile   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure190   hadoop.hdfs.server.namenode.snapshot.TestSnapshot   hadoop.hdfs.TestRollingUpgradeDowngrade   hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure   hadoop.hdfs.TestDFSInotifyEventInputStream   hadoop.fs.TestSymlinkHdfsFileContext JDK v1.7.0_91 Failed junit tests hadoop.hdfs.server.namenode.TestRecoverStripedBlocks   hadoop.fs.viewfs.TestViewFsAtHdfsRoot   hadoop.cli.TestCryptoAdminCLI   hadoop.hdfs.qjournal.client.TestQuorumJournalManager   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060   hadoop.hdfs.TestDFSStripedOutputStream   hadoop.hdfs.TestEncryptionZonesWithKMS   hadoop.fs.contract.hdfs.TestHDFSContractRename   hadoop.fs.viewfs.TestViewFsWithXAttrs   hadoop.fs.viewfs.TestViewFileSystemWithXAttrs   hadoop.hdfs.server.namenode.TestCheckpoint   hadoop.hdfs.server.namenode.TestQuotaWithStripedBlocks   hadoop.hdfs.TestErasureCodeBenchmarkThroughput   hadoop.hdfs.server.namenode.snapshot.TestXAttrWithSnapshot   hadoop.cli.TestErasureCodingCLI   hadoop.hdfs.server.namenode.TestAddStripedBlocks   hadoop.hdfs.TestHDFSTrash   hadoop.hdfs.tools.TestDelegationTokenFetcher   hadoop.hdfs.TestAclsEndToEnd   hadoop.hdfs.server.namenode.TestQuotaByStorageType   hadoop.hdfs.TestSnapshotCommands   hadoop.hdfs.TestSafeModeWithStripedFile   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050   hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped   hadoop.hdfs.TestEncryptionZonesWithHA   hadoop.hdfs.qjournal.server.TestJournalNode   hadoop.hdfs.TestReservedRawPaths   hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer   hadoop.hdfs.server.namenode.TestFSImageWithXAttr   hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130   hadoop.hdfs.server.namenode.snapshot.TestSnapshottableDirListing   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure170   hadoop.hdfs.qjournal.TestSecureNNWithQJM   hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA   hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks   hadoop.hdfs.server.namenode.ha.TestDelegationTokensWithHA   hadoop.hdfs.server.mover.TestStorageMover   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots   hadoop.hdfs.web.TestWebHDFS   hadoop.fs.contract.hdfs.TestHDFSContractRootDirectory   hadoop.hdfs.server.mover.TestMover   hadoop.hdfs.security.TestDelegationToken   hadoop.hdfs.server.namenode.snapshot.TestSnapshotMetrics   hadoop.fs.TestHDFSFileContextMainOperations   hadoop.hdfs.TestDistributedFileSystem   hadoop.hdfs.TestEncryptionZones   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180   hadoop.hdfs.qjournal.TestNNWithQJM   hadoop.hdfs.server.namenode.TestNameNodeXAttr   hadoop.hdfs.server.namenode.snapshot.TestSetQuotaWithSnapshot   hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA   hadoop.hdfs.TestErasureCodingPolicyWithSnapshot   hadoop.hdfs.server.namenode.TestFsck   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure030   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160   hadoop.hdfs.qjournal.client.TestQJMWithFaults   hadoop.fs.permission.TestStickyBit   hadoop.fs.TestFcHdfsSetUMask   hadoop.hdfs.server.namenode.TestStripedINodeFile   hadoop.hdfs.TestRollingUpgrade   hadoop.hdfs.server.balancer.TestBalancer   hadoop.hdfs.TestBlockStoragePolicy   hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer   hadoop.hdfs.TestReadStripedFileWithMissingBlocks   hadoop.fs.TestResolveHdfsSymlink   hadoop.cli.TestXAttrCLI   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090   hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits   hadoop.hdfs.server.namenode.TestFSImage   hadoop.hdfs.TestDFSStripedInputStream   hadoop.hdfs.server.namenode.ha.TestXAttrsWithHA   hadoop.hdfs.TestFileStatus   hadoop.hdfs.server.namenode.ha.TestBootstrapStandbyWithQJM   hadoop.hdfs.TestDatanodeLayoutUpgrade   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure040   hadoop.hdfs.TestReadStripedFileWithDecoding   hadoop.hdfs.TestRecoverStripedFile   hadoop.fs.viewfs.TestViewFileSystemAtHdfsRoot   hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport   hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForXAttr   hadoop.hdfs.TestListFilesInFileContext   hadoop.hdfs.TestErasureCodingPolicies   hadoop.hdfs.server.namenode.ha.TestFailoverWithBlockTokensEnabled   hadoop.hdfs.TestAppendSnapshotTruncate   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010   hadoop.fs.viewfs.TestViewFileSystemHdfs   hadoop.security.TestPermissionSymlinks   hadoop.hdfs.TestRollingUpgradeRollback   hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS   hadoop.hdfs.TestDFSUpgradeFromImage   hadoop.hdfs.TestHDFSFileSystemContract   hadoop.hdfs.server.namenode.TestFSImageWithSnapshot   hadoop.hdfs.TestEncryptedTransfer   hadoop.hdfs.server.namenode.TestINodeAttributeProvider   hadoop.hdfs.server.blockmanagement.TestSequentialBlockGroupId   hadoop.fs.viewfs.TestViewFsHdfs   hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer   hadoop.hdfs.TestFileStatusWithECPolicy   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure120   hadoop.hdfs.server.namenode.TestFileContextXAttr   hadoop.hdfs.security.token.block.TestBlockToken   hadoop.hdfs.protocolPB.TestPBHelper   hadoop.hdfs.server.namenode.snapshot.TestSnapshotListing   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.cli.TestHDFSCLI   hadoop.hdfs.web.TestWebHDFSXAttr   hadoop.hdfs.TestListFilesInDFS   hadoop.hdfs.server.namenode.TestNamenodeRetryCache   hadoop.hdfs.server.namenode.TestFSEditLogLoader   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200   hadoop.fs.loadGenerator.TestLoadGenerator   hadoop.cli.TestAclCLI   hadoop.fs.TestGlobPaths   hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerWithStripedBlocks   hadoop.hdfs.TestDatanodeStartupFixesLegacyStorageIDs   hadoop.hdfs.server.namenode.TestBackupNode   hadoop.hdfs.server.namenode.TestINodeFile   hadoop.hdfs.TestParallelShortCircuitReadUnCached   hadoop.hdfs.TestDFSShell   hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer   hadoop.hdfs.TestDFSRename   hadoop.fs.TestSymlinkHdfsFileSystem   hadoop.hdfs.TestWriteReadStripedFile   hadoop.hdfs.server.namenode.snapshot.TestSnapshot   hadoop.hdfs.TestRollingUpgradeDowngrade   hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure   hadoop.hdfs.TestDFSInotifyEventInputStream   hadoop.fs.TestSymlinkHdfsFileContext Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12777804/HDFS-9557.patch JIRA Issue HDFS-9557 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 48b1b129fac6 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 8602692 findbugs v3.0.0 javac hadoop-hdfs-project-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-HDFS-Build/13880/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project-jdk1.8.0_66.txt javac hadoop-hdfs-project-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-HDFS-Build/13880/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project-jdk1.7.0_91.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13880/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13880/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs-client.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/13880/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13880/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13880/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13880/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13880/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13880/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project Max memory used 76MB Powered by Apache Yetus 0.1.0 http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13880/console This message was automatically generated.
        Hide
        liuml07 Mingliang Liu added a comment -

        Does the general case fall into ByteString.copyFrom?

        Show
        liuml07 Mingliang Liu added a comment - Does the general case fall into ByteString.copyFrom ?
        Hide
        cmccabe Colin P. McCabe added a comment - - edited

        +1 for the idea, pending fixing the issue Chris Nauroth found.

        Long-term, I think we might want to switch to a PB library that does less object allocation.

        Show
        cmccabe Colin P. McCabe added a comment - - edited +1 for the idea, pending fixing the issue Chris Nauroth found. Long-term, I think we might want to switch to a PB library that does less object allocation.
        Hide
        cnauroth Chris Nauroth added a comment -

        +1 for the idea, but I see an infinite recursion here:

           public static ByteString getByteString(byte[] bytes) {
            // return singleton to reduce object allocation
            return (bytes.length == 0) ? ByteString.EMPTY : getByteString(bytes); // <-- stack overflow for non-empty bytes
           }
        
        Show
        cnauroth Chris Nauroth added a comment - +1 for the idea, but I see an infinite recursion here: public static ByteString getByteString( byte [] bytes) { // return singleton to reduce object allocation return (bytes.length == 0) ? ByteString.EMPTY : getByteString(bytes); // <-- stack overflow for non-empty bytes }
        Hide
        daryn Daryn Sharp added a comment -

        very simple patch

        Show
        daryn Daryn Sharp added a comment - very simple patch

          People

          • Assignee:
            daryn Daryn Sharp
            Reporter:
            daryn Daryn Sharp
          • Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development