Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9805

TCP_NODELAY not set before SASL handshake in data transfer pipeline

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0-alpha1
    • Component/s: datanode
    • Labels:
      None
    • Target Version/s:

      Description

      There are a few places in the DN -> DN block transfer pipeline where TCP_NODELAY is not set before doing a SASL handshake:

      • in DataNode.DataTransfer::run()
      • in DataXceiver::replaceBlock()
      • in DataXceiver::writeBlock()
      1. HDFS-9805.002.patch
        3 kB
        Gary Helmling
      2. HDFS-9805.003.patch
        3 kB
        Gary Helmling
      3. HDFS-9805.004.patch
        23 kB
        Gary Helmling
      4. HDFS-9805.005.patch
        23 kB
        Gary Helmling
      5. HDFS-9805.branch-2.001.patch
        22 kB
        Gary Helmling

        Activity

        Hide
        ghelmling Gary Helmling added a comment -

        In trunk, it looks like there is an additional location where this happens:

        • ErasureCodingWorker.ReconstructAndTransferBlock::initTargetStreams()
        Show
        ghelmling Gary Helmling added a comment - In trunk, it looks like there is an additional location where this happens: ErasureCodingWorker.ReconstructAndTransferBlock::initTargetStreams()
        Hide
        ghelmling Gary Helmling added a comment -

        Attaching a very simple patch for trunk, which always sets TCP_NODELAY for these paths, following what is done in DFSUtilClient::peerFromSocket().

        Alternately, since HDFS-9700 added a new config option dfs.data.transfer.client.tcpnodelay (defaulting to true), we could do something similar here and introduce a new config option (ie. dfs.data.transfer.tcpnodelay) for these cases.

        Show
        ghelmling Gary Helmling added a comment - Attaching a very simple patch for trunk, which always sets TCP_NODELAY for these paths, following what is done in DFSUtilClient::peerFromSocket() . Alternately, since HDFS-9700 added a new config option dfs.data.transfer.client.tcpnodelay (defaulting to true), we could do something similar here and introduce a new config option (ie. dfs.data.transfer.tcpnodelay ) for these cases.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 24s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 10m 17s trunk passed
        +1 compile 1m 13s trunk passed with JDK v1.8.0_72
        +1 compile 0m 56s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 26s trunk passed
        +1 mvnsite 1m 8s trunk passed
        +1 mvneclipse 0m 17s trunk passed
        +1 findbugs 2m 28s trunk passed
        +1 javadoc 1m 39s trunk passed with JDK v1.8.0_72
        +1 javadoc 2m 33s trunk passed with JDK v1.7.0_95
        +1 mvninstall 1m 27s the patch passed
        +1 compile 1m 23s the patch passed with JDK v1.8.0_72
        +1 javac 1m 23s the patch passed
        +1 compile 0m 56s the patch passed with JDK v1.7.0_95
        +1 javac 0m 56s the patch passed
        -1 checkstyle 0m 27s hadoop-hdfs-project/hadoop-hdfs: patch generated 1 new + 269 unchanged - 1 fixed = 270 total (was 270)
        +1 mvnsite 1m 14s the patch passed
        +1 mvneclipse 0m 16s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 3m 33s the patch passed
        +1 javadoc 2m 26s the patch passed with JDK v1.8.0_72
        +1 javadoc 3m 31s the patch passed with JDK v1.7.0_95
        -1 unit 114m 36s hadoop-hdfs in the patch failed with JDK v1.8.0_72.
        -1 unit 124m 56s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
        +1 asflicense 0m 43s Patch does not generate ASF License warnings.
        280m 28s



        Reason Tests
        JDK v1.8.0_72 Failed junit tests hadoop.hdfs.server.namenode.ha.TestEditLogTailer
          hadoop.hdfs.security.TestDelegationTokenForProxyUser
          hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints
          hadoop.hdfs.TestDatanodeRegistration
          hadoop.hdfs.server.namenode.ha.TestHAAppend
          hadoop.hdfs.qjournal.TestSecureNNWithQJM
          hadoop.hdfs.server.datanode.TestDirectoryScanner
        JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency
          hadoop.hdfs.server.namenode.TestSecureNameNode
          hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations
          hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
          hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes
          hadoop.hdfs.server.datanode.TestBlockReplacement
          hadoop.metrics2.sink.TestRollingFileSystemSinkWithSecureHdfs
          hadoop.hdfs.server.namenode.ha.TestHAAppend
          hadoop.hdfs.server.namenode.TestFileTruncate
          hadoop.hdfs.TestEncryptionZones
          hadoop.hdfs.server.datanode.TestNNHandlesCombinedBlockReport
          hadoop.hdfs.server.datanode.TestDirectoryScanner
        JDK v1.7.0_95 Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12788110/HDFS-9805.001.patch
        JIRA Issue HDFS-9805
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux b2bcf6e1647d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 4b0e59f
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14507/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/14507/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/14507/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14507/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14507/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14507/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14507/console
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 24s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 10m 17s trunk passed +1 compile 1m 13s trunk passed with JDK v1.8.0_72 +1 compile 0m 56s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 26s trunk passed +1 mvnsite 1m 8s trunk passed +1 mvneclipse 0m 17s trunk passed +1 findbugs 2m 28s trunk passed +1 javadoc 1m 39s trunk passed with JDK v1.8.0_72 +1 javadoc 2m 33s trunk passed with JDK v1.7.0_95 +1 mvninstall 1m 27s the patch passed +1 compile 1m 23s the patch passed with JDK v1.8.0_72 +1 javac 1m 23s the patch passed +1 compile 0m 56s the patch passed with JDK v1.7.0_95 +1 javac 0m 56s the patch passed -1 checkstyle 0m 27s hadoop-hdfs-project/hadoop-hdfs: patch generated 1 new + 269 unchanged - 1 fixed = 270 total (was 270) +1 mvnsite 1m 14s the patch passed +1 mvneclipse 0m 16s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 3m 33s the patch passed +1 javadoc 2m 26s the patch passed with JDK v1.8.0_72 +1 javadoc 3m 31s the patch passed with JDK v1.7.0_95 -1 unit 114m 36s hadoop-hdfs in the patch failed with JDK v1.8.0_72. -1 unit 124m 56s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 43s Patch does not generate ASF License warnings. 280m 28s Reason Tests JDK v1.8.0_72 Failed junit tests hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints   hadoop.hdfs.TestDatanodeRegistration   hadoop.hdfs.server.namenode.ha.TestHAAppend   hadoop.hdfs.qjournal.TestSecureNNWithQJM   hadoop.hdfs.server.datanode.TestDirectoryScanner JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency   hadoop.hdfs.server.namenode.TestSecureNameNode   hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes   hadoop.hdfs.server.datanode.TestBlockReplacement   hadoop.metrics2.sink.TestRollingFileSystemSinkWithSecureHdfs   hadoop.hdfs.server.namenode.ha.TestHAAppend   hadoop.hdfs.server.namenode.TestFileTruncate   hadoop.hdfs.TestEncryptionZones   hadoop.hdfs.server.datanode.TestNNHandlesCombinedBlockReport   hadoop.hdfs.server.datanode.TestDirectoryScanner JDK v1.7.0_95 Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2 Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12788110/HDFS-9805.001.patch JIRA Issue HDFS-9805 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux b2bcf6e1647d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 4b0e59f Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14507/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14507/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14507/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14507/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14507/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14507/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14507/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        ghelmling Gary Helmling added a comment -

        The checkstyle warning is on the DataXceiver::writeBlock() method being 236 lines long, greater than max allowed of 150. This patch adds a single line to writeBlock(), which I think triggered the warning. But I don't think it makes sense to refactor the method as part of this issue.

        For java 8, the following tests failed or timed out:

        Failed tests: 
          TestDatanodeRegistration.testForcedRegistration:382 null
          TestHAAppend.testMultipleAppendsDuringCatchupTailing:125 inode should complete in ~60000 ms.
        Expected: is <true>
             but: was <false>
          TestStandbyCheckpoints.testCheckpointCancellationDuringUpload:347 expected:<0> but was:<108>
        
        Tests in error: 
          TestDelegationTokenForProxyUser.testWebHdfsDoAs:161 ?  test timed out after 50...
          TestDirectoryScanner.testThrottling:584->createFile:108->Object.wait:502->Object.wait:-2 ? 
          TestEditLogTailer.testNN1TriggersLogRolls:115->testStandbyTriggersLogRolls:147->waitForLogRollInSharedDir:168 ? Timeout
          TestSecureNNWithQJM.testSecureMode:167->doNNWithQJMTest:187->restartNameNode:205->Object.wait:502->Object.wait:-2 ? 
          TestSecureNNWithQJM.testSecondaryNameNodeHttpAddressNotNeeded:173->doNNWithQJMTest:193->restartNameNode:205 ? EditLogInput
        

        All of these pass for me locally.

        For java 7, the following tests failed or timed out:

        Failed tests: 
          TestDataNodeVolumeFailure.testUnderReplicationAfterVolFailure:412 There is no under replicated block after volume failure
          TestBlockReplacement.testDeletedBlockWhenAddBlockIsInEdit:436 The block should be only on 1 datanode  expected:<1> but was:<2>
          TestNameNodeMetadataConsistency.testGenerationStampInFuture:113 expected:<17> but was:<0>
          TestHAAppend.testMultipleAppendsDuringCatchupTailing:125 inode should complete in ~60000 ms.
        Expected: is <true>
             but: was <false>
        
        Tests in error: 
          TestRollingFileSystemSinkWithSecureHdfs.testMissingPropertiesWithSecureHDFS:145->createDirectoriesSecurely:191 ? IO
          TestDataNodeHotSwapVolumes.testRemoveVolumeBeingWritten:609->testRemoveVolumeBeingWrittenForDatanode:686 ? Timeout
          TestDataNodeMultipleRegistrations.testClusterIdMismatchAtStartupWithHA:253 ?  ...
          TestNNHandlesCombinedBlockReport>BlockReportTestBase.testOneReplicaRbwReportArrivesAfterBlockCompleted:630 ? 
          TestDirectoryScanner.testThrottling:584->createFile:108->Object.wait:503->Object.wait:-2 ? 
          TestSecureNameNode.testName:65 ? IO Failed on local exception: java.io.IOExcep...
          TestFileTruncate.testTruncateWithDataNodesRestart:704 ? Timeout Timed out wait...
          TestEncryptionZones.testStartFileRetry:1067 ?  test timed out after 120000 mil...
        

        All of these pass locally for me with java 7 as well.

        Show
        ghelmling Gary Helmling added a comment - The checkstyle warning is on the DataXceiver::writeBlock() method being 236 lines long, greater than max allowed of 150. This patch adds a single line to writeBlock() , which I think triggered the warning. But I don't think it makes sense to refactor the method as part of this issue. For java 8, the following tests failed or timed out: Failed tests: TestDatanodeRegistration.testForcedRegistration:382 null TestHAAppend.testMultipleAppendsDuringCatchupTailing:125 inode should complete in ~60000 ms. Expected: is <true> but: was <false> TestStandbyCheckpoints.testCheckpointCancellationDuringUpload:347 expected:<0> but was:<108> Tests in error: TestDelegationTokenForProxyUser.testWebHdfsDoAs:161 ? test timed out after 50... TestDirectoryScanner.testThrottling:584->createFile:108->Object.wait:502->Object.wait:-2 ? TestEditLogTailer.testNN1TriggersLogRolls:115->testStandbyTriggersLogRolls:147->waitForLogRollInSharedDir:168 ? Timeout TestSecureNNWithQJM.testSecureMode:167->doNNWithQJMTest:187->restartNameNode:205->Object.wait:502->Object.wait:-2 ? TestSecureNNWithQJM.testSecondaryNameNodeHttpAddressNotNeeded:173->doNNWithQJMTest:193->restartNameNode:205 ? EditLogInput All of these pass for me locally. For java 7, the following tests failed or timed out: Failed tests: TestDataNodeVolumeFailure.testUnderReplicationAfterVolFailure:412 There is no under replicated block after volume failure TestBlockReplacement.testDeletedBlockWhenAddBlockIsInEdit:436 The block should be only on 1 datanode expected:<1> but was:<2> TestNameNodeMetadataConsistency.testGenerationStampInFuture:113 expected:<17> but was:<0> TestHAAppend.testMultipleAppendsDuringCatchupTailing:125 inode should complete in ~60000 ms. Expected: is <true> but: was <false> Tests in error: TestRollingFileSystemSinkWithSecureHdfs.testMissingPropertiesWithSecureHDFS:145->createDirectoriesSecurely:191 ? IO TestDataNodeHotSwapVolumes.testRemoveVolumeBeingWritten:609->testRemoveVolumeBeingWrittenForDatanode:686 ? Timeout TestDataNodeMultipleRegistrations.testClusterIdMismatchAtStartupWithHA:253 ? ... TestNNHandlesCombinedBlockReport>BlockReportTestBase.testOneReplicaRbwReportArrivesAfterBlockCompleted:630 ? TestDirectoryScanner.testThrottling:584->createFile:108->Object.wait:503->Object.wait:-2 ? TestSecureNameNode.testName:65 ? IO Failed on local exception: java.io.IOExcep... TestFileTruncate.testTruncateWithDataNodesRestart:704 ? Timeout Timed out wait... TestEncryptionZones.testStartFileRetry:1067 ? test timed out after 120000 mil... All of these pass locally for me with java 7 as well.
        Hide
        ghelmling Gary Helmling added a comment -

        Ping. Any takers for this change? It's pretty straightforward, though I can add a separate config for it if necessary.

        Show
        ghelmling Gary Helmling added a comment - Ping. Any takers for this change? It's pretty straightforward, though I can add a separate config for it if necessary.
        Hide
        cmccabe Colin P. McCabe added a comment -

        Hi Gary Helmling, thanks for looking at this. Can you update the patch?

        Show
        cmccabe Colin P. McCabe added a comment - Hi Gary Helmling , thanks for looking at this. Can you update the patch?
        Hide
        ghelmling Gary Helmling added a comment -

        Here's a new patch rebased against the latest trunk.

        Show
        ghelmling Gary Helmling added a comment - Here's a new patch rebased against the latest trunk.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 11s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 6m 57s trunk passed
        +1 compile 0m 37s trunk passed with JDK v1.8.0_77
        +1 compile 0m 42s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 23s trunk passed
        +1 mvnsite 0m 53s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 1m 57s trunk passed
        +1 javadoc 1m 5s trunk passed with JDK v1.8.0_77
        +1 javadoc 1m 45s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 45s the patch passed
        +1 compile 0m 35s the patch passed with JDK v1.8.0_77
        +1 javac 0m 35s the patch passed
        +1 compile 0m 39s the patch passed with JDK v1.7.0_95
        +1 javac 0m 39s the patch passed
        -1 checkstyle 0m 21s hadoop-hdfs-project/hadoop-hdfs: patch generated 1 new + 264 unchanged - 1 fixed = 265 total (was 265)
        +1 mvnsite 0m 48s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 2m 5s the patch passed
        +1 javadoc 1m 5s the patch passed with JDK v1.8.0_77
        +1 javadoc 1m 41s the patch passed with JDK v1.7.0_95
        -1 unit 56m 47s hadoop-hdfs in the patch failed with JDK v1.8.0_77.
        -1 unit 54m 0s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
        -1 asflicense 0m 22s Patch generated 1 ASF License warnings.
        136m 3s



        Reason Tests
        JDK v1.8.0_77 Failed junit tests hadoop.hdfs.TestHFlush
          hadoop.cli.TestHDFSCLI
        JDK v1.7.0_95 Failed junit tests hadoop.hdfs.TestHFlush
          hadoop.cli.TestHDFSCLI



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:fbe3e86
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12795741/HDFS-9805.002.patch
        JIRA Issue HDFS-9805
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 5230e691db27 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 0ef8bbf
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14973/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/14973/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/14973/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14973/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14973/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14973/testReport/
        asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/14973/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14973/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 57s trunk passed +1 compile 0m 37s trunk passed with JDK v1.8.0_77 +1 compile 0m 42s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 23s trunk passed +1 mvnsite 0m 53s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 57s trunk passed +1 javadoc 1m 5s trunk passed with JDK v1.8.0_77 +1 javadoc 1m 45s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 45s the patch passed +1 compile 0m 35s the patch passed with JDK v1.8.0_77 +1 javac 0m 35s the patch passed +1 compile 0m 39s the patch passed with JDK v1.7.0_95 +1 javac 0m 39s the patch passed -1 checkstyle 0m 21s hadoop-hdfs-project/hadoop-hdfs: patch generated 1 new + 264 unchanged - 1 fixed = 265 total (was 265) +1 mvnsite 0m 48s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 5s the patch passed +1 javadoc 1m 5s the patch passed with JDK v1.8.0_77 +1 javadoc 1m 41s the patch passed with JDK v1.7.0_95 -1 unit 56m 47s hadoop-hdfs in the patch failed with JDK v1.8.0_77. -1 unit 54m 0s hadoop-hdfs in the patch failed with JDK v1.7.0_95. -1 asflicense 0m 22s Patch generated 1 ASF License warnings. 136m 3s Reason Tests JDK v1.8.0_77 Failed junit tests hadoop.hdfs.TestHFlush   hadoop.cli.TestHDFSCLI JDK v1.7.0_95 Failed junit tests hadoop.hdfs.TestHFlush   hadoop.cli.TestHDFSCLI Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12795741/HDFS-9805.002.patch JIRA Issue HDFS-9805 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 5230e691db27 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0ef8bbf Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14973/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14973/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14973/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14973/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14973/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14973/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/14973/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14973/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        ghelmling Gary Helmling added a comment -

        New patch rebased against the latest trunk.

        Show
        ghelmling Gary Helmling added a comment - New patch rebased against the latest trunk.
        Hide
        ghelmling Gary Helmling added a comment -

        Colin P. McCabe I've updated the patch against the latest trunk. I can do a branch-2.8 version as well if that helps. Please let me know what you think.

        Show
        ghelmling Gary Helmling added a comment - Colin P. McCabe I've updated the patch against the latest trunk. I can do a branch-2.8 version as well if that helps. Please let me know what you think.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 10s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 6m 33s trunk passed
        +1 compile 0m 38s trunk passed with JDK v1.8.0_74
        +1 compile 0m 41s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 24s trunk passed
        +1 mvnsite 0m 53s trunk passed
        +1 mvneclipse 0m 12s trunk passed
        +1 findbugs 1m 56s trunk passed
        +1 javadoc 1m 8s trunk passed with JDK v1.8.0_74
        +1 javadoc 1m 47s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 47s the patch passed
        +1 compile 0m 38s the patch passed with JDK v1.8.0_74
        +1 javac 0m 38s the patch passed
        +1 compile 0m 39s the patch passed with JDK v1.7.0_95
        +1 javac 0m 39s the patch passed
        -1 checkstyle 0m 22s hadoop-hdfs-project/hadoop-hdfs: patch generated 1 new + 264 unchanged - 1 fixed = 265 total (was 265)
        +1 mvnsite 0m 52s the patch passed
        +1 mvneclipse 0m 12s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 2m 11s the patch passed
        +1 javadoc 1m 2s the patch passed with JDK v1.8.0_74
        +1 javadoc 1m 44s the patch passed with JDK v1.7.0_95
        -1 unit 58m 25s hadoop-hdfs in the patch failed with JDK v1.8.0_74.
        -1 unit 53m 28s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
        +1 asflicense 0m 21s Patch does not generate ASF License warnings.
        137m 8s



        Reason Tests
        JDK v1.8.0_74 Failed junit tests hadoop.TestRefreshCallQueue
          hadoop.hdfs.TestDFSClientRetries
          hadoop.hdfs.TestHFlush
          hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA
        JDK v1.7.0_95 Failed junit tests hadoop.TestRefreshCallQueue
          hadoop.hdfs.TestHFlush
          hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA
          hadoop.hdfs.shortcircuit.TestShortCircuitCache



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:fbe3e86
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12796423/HDFS-9805.003.patch
        JIRA Issue HDFS-9805
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux accfe83502e1 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / aac4d65
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15031/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/15031/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/15031/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15031/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15031/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15031/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15031/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 33s trunk passed +1 compile 0m 38s trunk passed with JDK v1.8.0_74 +1 compile 0m 41s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 24s trunk passed +1 mvnsite 0m 53s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 56s trunk passed +1 javadoc 1m 8s trunk passed with JDK v1.8.0_74 +1 javadoc 1m 47s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 47s the patch passed +1 compile 0m 38s the patch passed with JDK v1.8.0_74 +1 javac 0m 38s the patch passed +1 compile 0m 39s the patch passed with JDK v1.7.0_95 +1 javac 0m 39s the patch passed -1 checkstyle 0m 22s hadoop-hdfs-project/hadoop-hdfs: patch generated 1 new + 264 unchanged - 1 fixed = 265 total (was 265) +1 mvnsite 0m 52s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 11s the patch passed +1 javadoc 1m 2s the patch passed with JDK v1.8.0_74 +1 javadoc 1m 44s the patch passed with JDK v1.7.0_95 -1 unit 58m 25s hadoop-hdfs in the patch failed with JDK v1.8.0_74. -1 unit 53m 28s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 21s Patch does not generate ASF License warnings. 137m 8s Reason Tests JDK v1.8.0_74 Failed junit tests hadoop.TestRefreshCallQueue   hadoop.hdfs.TestDFSClientRetries   hadoop.hdfs.TestHFlush   hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA JDK v1.7.0_95 Failed junit tests hadoop.TestRefreshCallQueue   hadoop.hdfs.TestHFlush   hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA   hadoop.hdfs.shortcircuit.TestShortCircuitCache Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12796423/HDFS-9805.003.patch JIRA Issue HDFS-9805 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux accfe83502e1 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / aac4d65 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15031/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15031/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15031/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15031/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15031/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15031/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15031/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        cmccabe Colin P. McCabe added a comment -

        Hi Gary Helmling,

        Thanks for looking at this. However, it seems like we should be checking to see if ipc.server.tcpnodelay is true rather than unconditionally setting TCP_NODELAY.

        Show
        cmccabe Colin P. McCabe added a comment - Hi Gary Helmling , Thanks for looking at this. However, it seems like we should be checking to see if ipc.server.tcpnodelay is true rather than unconditionally setting TCP_NODELAY .
        Hide
        ghelmling Gary Helmling added a comment -

        Colin P. McCabe, is ipc.server.tcpnodelay the right config to use? Or do we need a dfs.data.transfer.server.tcpnodelay config similar to the dfs.data.transfer.client.tcpnodelay added for HDFS-9700? I don't want to introduce another config, but since these changes all seem to be on the data transfer side, that seems consistent.

        Show
        ghelmling Gary Helmling added a comment - Colin P. McCabe , is ipc.server.tcpnodelay the right config to use? Or do we need a dfs.data.transfer.server.tcpnodelay config similar to the dfs.data.transfer.client.tcpnodelay added for HDFS-9700 ? I don't want to introduce another config, but since these changes all seem to be on the data transfer side, that seems consistent.
        Hide
        cmccabe Colin P. McCabe added a comment -

        Good point. For consistency's sake, we should have a dfs.data.transfer.server.tcpnodelay configuration matching dfs.data.transfer.client.tcpnodelay. In general, "IPC" configuration keys in Hadoop tend to refer to Hadoop RPC v9 endpoints, even though technically DataTransferProtocol is also a form of IPC (inter-process communication).

        I don't want to introduce another config, but since these changes all seem to be on the data transfer side, that seems consistent.

        I think it's OK.

        Can we add a unit test that ensures that TCP_NODELAY is set or cleared, as the configuration key dictates? Otherwise we may run into a regression here where the setting gets lost.

        Show
        cmccabe Colin P. McCabe added a comment - Good point. For consistency's sake, we should have a dfs.data.transfer.server.tcpnodelay configuration matching dfs.data.transfer.client.tcpnodelay . In general, "IPC" configuration keys in Hadoop tend to refer to Hadoop RPC v9 endpoints, even though technically DataTransferProtocol is also a form of IPC (inter-process communication). I don't want to introduce another config, but since these changes all seem to be on the data transfer side, that seems consistent. I think it's OK. Can we add a unit test that ensures that TCP_NODELAY is set or cleared, as the configuration key dictates? Otherwise we may run into a regression here where the setting gets lost.
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Thanks Gary Helmling for the patch! I believe we've hit the same issue and am interested to move this forward.
        Would you mind if I rebase the patch and add the configs and the unit tests as Colin suggested?

        Show
        jojochuang Wei-Chiu Chuang added a comment - Thanks Gary Helmling for the patch! I believe we've hit the same issue and am interested to move this forward. Would you mind if I rebase the patch and add the configs and the unit tests as Colin suggested?
        Hide
        ghelmling Gary Helmling added a comment -

        Wei-Chiu Chuang, I have a version that adds in the config already. I'll take a stab at the unit test and post an update tomorrow.

        Show
        ghelmling Gary Helmling added a comment - Wei-Chiu Chuang , I have a version that adds in the config already. I'll take a stab at the unit test and post an update tomorrow.
        Hide
        ghelmling Gary Helmling added a comment -

        Updated patch adding:

        • a new configuration property (dfs.data.transfer.server.tcpnodelay), defaulting to true, controlling the TCP_NODELAY setting in the DN -> DN transfer path
        • a test case check the TCP_NODELAY was enabled on all used sockets when the relevant config settings are enabled

        Note that I had to modify DataNode#newSocket() in this patch in order to support the test case. Prior to this, newSocket() was not using the configured socket factory, instead creating sockets directly. This seems like a change we would want anyway, just calling in out.

        Show
        ghelmling Gary Helmling added a comment - Updated patch adding: a new configuration property (dfs.data.transfer.server.tcpnodelay), defaulting to true, controlling the TCP_NODELAY setting in the DN -> DN transfer path a test case check the TCP_NODELAY was enabled on all used sockets when the relevant config settings are enabled Note that I had to modify DataNode#newSocket() in this patch in order to support the test case. Prior to this, newSocket() was not using the configured socket factory, instead creating sockets directly. This seems like a change we would want anyway, just calling in out.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 26s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 13s Maven dependency ordering for branch
        +1 mvninstall 6m 21s trunk passed
        +1 compile 1m 30s trunk passed
        +1 checkstyle 0m 42s trunk passed
        +1 mvnsite 1m 35s trunk passed
        +1 mvneclipse 0m 28s trunk passed
        +1 findbugs 3m 12s trunk passed
        +1 javadoc 1m 16s trunk passed
        0 mvndep 0m 7s Maven dependency ordering for patch
        +1 mvninstall 1m 22s the patch passed
        +1 compile 1m 26s the patch passed
        +1 javac 1m 26s the patch passed
        -0 checkstyle 0m 41s hadoop-hdfs-project: The patch generated 27 new + 679 unchanged - 2 fixed = 706 total (was 681)
        +1 mvnsite 1m 47s the patch passed
        +1 mvneclipse 0m 25s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 3m 55s the patch passed
        +1 javadoc 1m 14s the patch passed
        +1 unit 0m 55s hadoop-hdfs-client in the patch passed.
        -1 unit 74m 53s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 19s The patch does not generate ASF License warnings.
        104m 20s



        Reason Tests
        Failed junit tests hadoop.tools.TestHdfsConfigFields
          hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
          hadoop.metrics2.sink.TestRollingFileSystemSinkWithHdfs
          hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:85209cc
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12813805/HDFS-9805.004.patch
        JIRA Issue HDFS-9805
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux df7ac815c41f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 9683eab
        Default Java 1.8.0_91
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15922/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/15922/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15922/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15922/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 26s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 13s Maven dependency ordering for branch +1 mvninstall 6m 21s trunk passed +1 compile 1m 30s trunk passed +1 checkstyle 0m 42s trunk passed +1 mvnsite 1m 35s trunk passed +1 mvneclipse 0m 28s trunk passed +1 findbugs 3m 12s trunk passed +1 javadoc 1m 16s trunk passed 0 mvndep 0m 7s Maven dependency ordering for patch +1 mvninstall 1m 22s the patch passed +1 compile 1m 26s the patch passed +1 javac 1m 26s the patch passed -0 checkstyle 0m 41s hadoop-hdfs-project: The patch generated 27 new + 679 unchanged - 2 fixed = 706 total (was 681) +1 mvnsite 1m 47s the patch passed +1 mvneclipse 0m 25s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 55s the patch passed +1 javadoc 1m 14s the patch passed +1 unit 0m 55s hadoop-hdfs-client in the patch passed. -1 unit 74m 53s hadoop-hdfs in the patch failed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 104m 20s Reason Tests Failed junit tests hadoop.tools.TestHdfsConfigFields   hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer   hadoop.metrics2.sink.TestRollingFileSystemSinkWithHdfs   hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot Subsystem Report/Notes Docker Image:yetus/hadoop:85209cc JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12813805/HDFS-9805.004.patch JIRA Issue HDFS-9805 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux df7ac815c41f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 9683eab Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15922/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15922/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15922/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15922/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        ghelmling Gary Helmling added a comment -

        Another update to test and checkstyle issues:

        • adds dfs.data.transfer.server.tcpnodelay to hdfs-default.xml to fix TestHdfsConfigFields
        • fixes checkstyle line length issues

        Of the other reported test failures:

        • TestOpenFilesWithSnapshot, TestRollingFileSystemSinkWithHdfs both pass for me locally
        • TestOfflineEditsViewer seems to be already failing on trunk
        Show
        ghelmling Gary Helmling added a comment - Another update to test and checkstyle issues: adds dfs.data.transfer.server.tcpnodelay to hdfs-default.xml to fix TestHdfsConfigFields fixes checkstyle line length issues Of the other reported test failures: TestOpenFilesWithSnapshot, TestRollingFileSystemSinkWithHdfs both pass for me locally TestOfflineEditsViewer seems to be already failing on trunk
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 32s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 7s Maven dependency ordering for branch
        +1 mvninstall 6m 14s trunk passed
        +1 compile 1m 23s trunk passed
        +1 checkstyle 0m 38s trunk passed
        +1 mvnsite 1m 25s trunk passed
        +1 mvneclipse 0m 25s trunk passed
        +1 findbugs 3m 6s trunk passed
        +1 javadoc 1m 16s trunk passed
        0 mvndep 0m 7s Maven dependency ordering for patch
        +1 mvninstall 1m 17s the patch passed
        +1 compile 1m 23s the patch passed
        +1 javac 1m 23s the patch passed
        -0 checkstyle 0m 35s hadoop-hdfs-project: The patch generated 3 new + 679 unchanged - 2 fixed = 682 total (was 681)
        +1 mvnsite 1m 20s the patch passed
        +1 mvneclipse 0m 20s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 findbugs 3m 20s the patch passed
        +1 javadoc 1m 10s the patch passed
        +1 unit 0m 53s hadoop-hdfs-client in the patch passed.
        -1 unit 72m 38s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 21s The patch does not generate ASF License warnings.
        99m 51s



        Reason Tests
        Failed junit tests hadoop.hdfs.server.datanode.TestBlockScanner
          hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency
          hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:85209cc
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12813826/HDFS-9805.005.patch
        JIRA Issue HDFS-9805
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux e8bb3f61e9df 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 9683eab
        Default Java 1.8.0_91
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15924/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/15924/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15924/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15924/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 32s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 7s Maven dependency ordering for branch +1 mvninstall 6m 14s trunk passed +1 compile 1m 23s trunk passed +1 checkstyle 0m 38s trunk passed +1 mvnsite 1m 25s trunk passed +1 mvneclipse 0m 25s trunk passed +1 findbugs 3m 6s trunk passed +1 javadoc 1m 16s trunk passed 0 mvndep 0m 7s Maven dependency ordering for patch +1 mvninstall 1m 17s the patch passed +1 compile 1m 23s the patch passed +1 javac 1m 23s the patch passed -0 checkstyle 0m 35s hadoop-hdfs-project: The patch generated 3 new + 679 unchanged - 2 fixed = 682 total (was 681) +1 mvnsite 1m 20s the patch passed +1 mvneclipse 0m 20s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 3m 20s the patch passed +1 javadoc 1m 10s the patch passed +1 unit 0m 53s hadoop-hdfs-client in the patch passed. -1 unit 72m 38s hadoop-hdfs in the patch failed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 99m 51s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency   hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer Subsystem Report/Notes Docker Image:yetus/hadoop:85209cc JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12813826/HDFS-9805.005.patch JIRA Issue HDFS-9805 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux e8bb3f61e9df 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 9683eab Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15924/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15924/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15924/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15924/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        ghelmling Gary Helmling added a comment -

        For the test failures:

        • TestBlockScanner passes for me locally
        • TestNameNodeMetadataConsistency seems flaky. It failed 1 time out of 10 for me locally.
        • TestOfflineEditsViewer is failing in trunk

        Colin P. McCabe, Wei-Chiu Chuang: could you take a look at the latest patch?

        Show
        ghelmling Gary Helmling added a comment - For the test failures: TestBlockScanner passes for me locally TestNameNodeMetadataConsistency seems flaky. It failed 1 time out of 10 for me locally. TestOfflineEditsViewer is failing in trunk Colin P. McCabe , Wei-Chiu Chuang : could you take a look at the latest patch?
        Hide
        jzhuge John Zhuge added a comment -

        Colin P. McCabe: You committed the patch into trunk on 6/29. Do you plan to resolve the jira?

        Show
        jzhuge John Zhuge added a comment - Colin P. McCabe : You committed the patch into trunk on 6/29. Do you plan to resolve the jira?
        Hide
        cmccabe Colin P. McCabe added a comment -

        Thanks for the reminder, John Zhuge. I committed the patch last week, but JIRA went down before I could mark the ticket as resolved.

        I have committed this to trunk only for the moment. The backport to branch-2 looks like it might be a little tricky, and our next release will be 3.0 anyway. If anyone is interested in backporting to branch-2, please do and update the ticket. Cheers.

        Show
        cmccabe Colin P. McCabe added a comment - Thanks for the reminder, John Zhuge . I committed the patch last week, but JIRA went down before I could mark the ticket as resolved. I have committed this to trunk only for the moment. The backport to branch-2 looks like it might be a little tricky, and our next release will be 3.0 anyway. If anyone is interested in backporting to branch-2, please do and update the ticket. Cheers.
        Hide
        ghelmling Gary Helmling added a comment -

        Colin P. McCabe Here is a patch backported to branch-2. It should apply cleanly. Do I need to re-open the issue to get the Yetus build to run?

        Show
        ghelmling Gary Helmling added a comment - Colin P. McCabe Here is a patch backported to branch-2. It should apply cleanly. Do I need to re-open the issue to get the Yetus build to run?

          People

          • Assignee:
            ghelmling Gary Helmling
            Reporter:
            ghelmling Gary Helmling
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development