Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9669

TcpPeerServer should respect ipc.server.listen.queue.size

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.2
    • Fix Version/s: 2.8.0, 2.7.3, 2.6.5, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None

      Description

      On periods of high traffic we are seeing:

      16/01/19 23:40:40 WARN hdfs.DFSClient: Connection failure: Failed to connect to /10.138.178.47:50010 for file /MYPATH/MYFILE for block BP-1935559084-10.138.112.27-1449689748174:blk_1080898601_7375294:java.io.IOException: Connection reset by peer
      java.io.IOException: Connection reset by peer
      	at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
      	at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
      	at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
      	at sun.nio.ch.IOUtil.write(IOUtil.java:65)
      	at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)
      	at org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:63)
      	at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
      	at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:159)
      	at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:117)
      	at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:109)
      	at java.io.DataOutputStream.writeInt(DataOutputStream.java:197)
      

      At the time that this happens there are way less xceivers than configured.

      On most JDK's this will make 50 the total backlog at any time. This effectively means that any GC + Busy time willl result in tcp resets.

      http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/tip/src/share/classes/java/net/ServerSocket.java#l370

      1. HDFS-9669.0.patch
        5 kB
        Elliott Clark
      2. HDFS-9669.1.patch
        5 kB
        Elliott Clark
      3. HDFS-9669.1.patch
        5 kB
        Elliott Clark

        Activity

        Hide
        eclark Elliott Clark added a comment -

        Straight forward patch to make sure that all the places that bind use the listen backlog setting.

        Show
        eclark Elliott Clark added a comment - Straight forward patch to make sure that all the places that bind use the listen backlog setting.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 7m 40s trunk passed
        +1 compile 0m 37s trunk passed with JDK v1.8.0_66
        +1 compile 0m 41s trunk passed with JDK v1.7.0_91
        +1 checkstyle 0m 17s trunk passed
        +1 mvnsite 0m 52s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 1m 53s trunk passed
        +1 javadoc 1m 3s trunk passed with JDK v1.8.0_66
        +1 javadoc 1m 47s trunk passed with JDK v1.7.0_91
        +1 mvninstall 0m 44s the patch passed
        +1 compile 0m 35s the patch passed with JDK v1.8.0_66
        +1 javac 0m 35s the patch passed
        +1 compile 0m 38s the patch passed with JDK v1.7.0_91
        +1 javac 0m 38s the patch passed
        -1 checkstyle 0m 17s hadoop-hdfs-project/hadoop-hdfs: patch generated 1 new + 128 unchanged - 0 fixed = 129 total (was 128)
        +1 mvnsite 0m 48s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 2m 2s the patch passed
        +1 javadoc 1m 0s the patch passed with JDK v1.8.0_66
        +1 javadoc 1m 46s the patch passed with JDK v1.7.0_91
        -1 unit 65m 23s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
        -1 unit 65m 19s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
        +1 asflicense 0m 20s Patch does not generate ASF License warnings.
        156m 26s



        Reason Tests
        JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeMetrics
          hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot
        JDK v1.7.0_91 Failed junit tests hadoop.hdfs.server.namenode.TestRecoverStripedBlocks



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12783313/HDFS-9669.0.patch
        JIRA Issue HDFS-9669
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 68b06eecc00b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 57d0a94
        Default Java 1.7.0_91
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14171/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/14171/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/14171/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14171/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14171/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14171/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Max memory used 77MB
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14171/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 40s trunk passed +1 compile 0m 37s trunk passed with JDK v1.8.0_66 +1 compile 0m 41s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 17s trunk passed +1 mvnsite 0m 52s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 53s trunk passed +1 javadoc 1m 3s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 47s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 44s the patch passed +1 compile 0m 35s the patch passed with JDK v1.8.0_66 +1 javac 0m 35s the patch passed +1 compile 0m 38s the patch passed with JDK v1.7.0_91 +1 javac 0m 38s the patch passed -1 checkstyle 0m 17s hadoop-hdfs-project/hadoop-hdfs: patch generated 1 new + 128 unchanged - 0 fixed = 129 total (was 128) +1 mvnsite 0m 48s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 2s the patch passed +1 javadoc 1m 0s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 46s the patch passed with JDK v1.7.0_91 -1 unit 65m 23s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 65m 19s hadoop-hdfs in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 20s Patch does not generate ASF License warnings. 156m 26s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeMetrics   hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot JDK v1.7.0_91 Failed junit tests hadoop.hdfs.server.namenode.TestRecoverStripedBlocks Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12783313/HDFS-9669.0.patch JIRA Issue HDFS-9669 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 68b06eecc00b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 57d0a94 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14171/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14171/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14171/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14171/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14171/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14171/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 77MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14171/console This message was automatically generated.
        Hide
        eclark Elliott Clark added a comment -

        Checkstyle nit.

        Show
        eclark Elliott Clark added a comment - Checkstyle nit.
        Hide
        ghelmling Gary Helmling added a comment -

        Patch looks good to me (non-binding). Seems like it closes the only hole in applying ipc.server.listen.queue.size.

        Show
        ghelmling Gary Helmling added a comment - Patch looks good to me (non-binding). Seems like it closes the only hole in applying ipc.server.listen.queue.size.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 7m 27s trunk passed
        +1 compile 0m 47s trunk passed with JDK v1.8.0_66
        +1 compile 0m 41s trunk passed with JDK v1.7.0_91
        +1 checkstyle 0m 17s trunk passed
        +1 mvnsite 0m 51s trunk passed
        +1 mvneclipse 0m 12s trunk passed
        +1 findbugs 1m 50s trunk passed
        +1 javadoc 1m 12s trunk passed with JDK v1.8.0_66
        +1 javadoc 1m 51s trunk passed with JDK v1.7.0_91
        +1 mvninstall 0m 45s the patch passed
        +1 compile 0m 44s the patch passed with JDK v1.8.0_66
        +1 javac 0m 44s the patch passed
        +1 compile 0m 39s the patch passed with JDK v1.7.0_91
        +1 javac 0m 39s the patch passed
        +1 checkstyle 0m 17s the patch passed
        +1 mvnsite 0m 49s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 2m 3s the patch passed
        +1 javadoc 1m 9s the patch passed with JDK v1.8.0_66
        +1 javadoc 1m 53s the patch passed with JDK v1.7.0_91
        -1 unit 71m 1s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
        -1 unit 73m 42s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
        +1 asflicense 0m 24s Patch does not generate ASF License warnings.
        171m 20s



        Reason Tests
        JDK v1.8.0_66 Failed junit tests hadoop.hdfs.shortcircuit.TestShortCircuitCache
          hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation
          hadoop.hdfs.server.namenode.TestNNThroughputBenchmark
          hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA
          hadoop.hdfs.TestDFSUpgradeFromImage
          hadoop.hdfs.TestEncryptionZonesWithKMS
          hadoop.hdfs.server.datanode.TestBlockScanner
          hadoop.hdfs.TestBlockStoragePolicy
        JDK v1.7.0_91 Failed junit tests hadoop.hdfs.TestDFSClientRetries
          hadoop.hdfs.server.datanode.TestBlockScanner



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12783371/HDFS-9669.1.patch
        JIRA Issue HDFS-9669
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 99d8d01ff7c5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 1425578
        Default Java 1.7.0_91
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/14176/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/14176/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14176/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14176/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14176/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Max memory used 77MB
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14176/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 27s trunk passed +1 compile 0m 47s trunk passed with JDK v1.8.0_66 +1 compile 0m 41s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 17s trunk passed +1 mvnsite 0m 51s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 50s trunk passed +1 javadoc 1m 12s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 51s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 45s the patch passed +1 compile 0m 44s the patch passed with JDK v1.8.0_66 +1 javac 0m 44s the patch passed +1 compile 0m 39s the patch passed with JDK v1.7.0_91 +1 javac 0m 39s the patch passed +1 checkstyle 0m 17s the patch passed +1 mvnsite 0m 49s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 3s the patch passed +1 javadoc 1m 9s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 53s the patch passed with JDK v1.7.0_91 -1 unit 71m 1s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 73m 42s hadoop-hdfs in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 24s Patch does not generate ASF License warnings. 171m 20s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.shortcircuit.TestShortCircuitCache   hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation   hadoop.hdfs.server.namenode.TestNNThroughputBenchmark   hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA   hadoop.hdfs.TestDFSUpgradeFromImage   hadoop.hdfs.TestEncryptionZonesWithKMS   hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.TestBlockStoragePolicy JDK v1.7.0_91 Failed junit tests hadoop.hdfs.TestDFSClientRetries   hadoop.hdfs.server.datanode.TestBlockScanner Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12783371/HDFS-9669.1.patch JIRA Issue HDFS-9669 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 99d8d01ff7c5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 1425578 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/14176/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14176/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14176/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14176/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14176/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 77MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14176/console This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 10m 58s trunk passed
        +1 compile 1m 18s trunk passed with JDK v1.8.0_66
        +1 compile 0m 59s trunk passed with JDK v1.7.0_91
        +1 checkstyle 0m 23s trunk passed
        +1 mvnsite 1m 13s trunk passed
        +1 mvneclipse 0m 16s trunk passed
        +1 findbugs 2m 26s trunk passed
        +1 javadoc 1m 40s trunk passed with JDK v1.8.0_66
        +1 javadoc 2m 25s trunk passed with JDK v1.7.0_91
        +1 mvninstall 1m 3s the patch passed
        +1 compile 1m 9s the patch passed with JDK v1.8.0_66
        +1 javac 1m 9s the patch passed
        +1 compile 0m 58s the patch passed with JDK v1.7.0_91
        +1 javac 0m 58s the patch passed
        +1 checkstyle 0m 22s the patch passed
        +1 mvnsite 1m 10s the patch passed
        +1 mvneclipse 0m 14s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 2m 46s the patch passed
        +1 javadoc 1m 30s the patch passed with JDK v1.8.0_66
        +1 javadoc 2m 28s the patch passed with JDK v1.7.0_91
        -1 unit 96m 22s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
        -1 unit 88m 7s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
        +1 asflicense 0m 28s Patch does not generate ASF License warnings.
        221m 43s



        Reason Tests
        JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.namenode.TestStartup
          hadoop.hdfs.TestLeaseRecovery2
          hadoop.hdfs.server.datanode.TestBlockScanner
          hadoop.hdfs.server.namenode.ha.TestEditLogTailer
          hadoop.hdfs.security.TestDelegationTokenForProxyUser
          hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
          hadoop.hdfs.server.datanode.TestBlockReplacement
          hadoop.hdfs.server.namenode.ha.TestHAAppend
          hadoop.hdfs.server.datanode.TestDirectoryScanner
        JDK v1.7.0_91 Failed junit tests hadoop.hdfs.server.datanode.TestBlockScanner
          hadoop.hdfs.shortcircuit.TestShortCircuitCache
          hadoop.hdfs.security.TestDelegationTokenForProxyUser
          hadoop.hdfs.TestDecommission
          hadoop.hdfs.server.balancer.TestBalancer
          hadoop.hdfs.server.datanode.TestDirectoryScanner



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12783370/HDFS-9669.1.patch
        JIRA Issue HDFS-9669
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux a068136477f1 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 1425578
        Default Java 1.7.0_91
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/14175/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/14175/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14175/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14175/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14175/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Max memory used 77MB
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14175/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 10m 58s trunk passed +1 compile 1m 18s trunk passed with JDK v1.8.0_66 +1 compile 0m 59s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 23s trunk passed +1 mvnsite 1m 13s trunk passed +1 mvneclipse 0m 16s trunk passed +1 findbugs 2m 26s trunk passed +1 javadoc 1m 40s trunk passed with JDK v1.8.0_66 +1 javadoc 2m 25s trunk passed with JDK v1.7.0_91 +1 mvninstall 1m 3s the patch passed +1 compile 1m 9s the patch passed with JDK v1.8.0_66 +1 javac 1m 9s the patch passed +1 compile 0m 58s the patch passed with JDK v1.7.0_91 +1 javac 0m 58s the patch passed +1 checkstyle 0m 22s the patch passed +1 mvnsite 1m 10s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 46s the patch passed +1 javadoc 1m 30s the patch passed with JDK v1.8.0_66 +1 javadoc 2m 28s the patch passed with JDK v1.7.0_91 -1 unit 96m 22s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 88m 7s hadoop-hdfs in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 28s Patch does not generate ASF License warnings. 221m 43s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.namenode.TestStartup   hadoop.hdfs.TestLeaseRecovery2   hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.server.datanode.TestBlockReplacement   hadoop.hdfs.server.namenode.ha.TestHAAppend   hadoop.hdfs.server.datanode.TestDirectoryScanner JDK v1.7.0_91 Failed junit tests hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.shortcircuit.TestShortCircuitCache   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.TestDecommission   hadoop.hdfs.server.balancer.TestBalancer   hadoop.hdfs.server.datanode.TestDirectoryScanner Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12783370/HDFS-9669.1.patch JIRA Issue HDFS-9669 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux a068136477f1 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 1425578 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/14175/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14175/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14175/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14175/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14175/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 77MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14175/console This message was automatically generated.
        Hide
        liuml07 Mingliang Liu added a comment -

        +1 (non-binding)

        Default backlog value should be used here. Failing tests seem unrelated.

        Show
        liuml07 Mingliang Liu added a comment - +1 (non-binding) Default backlog value should be used here. Failing tests seem unrelated.
        Hide
        liuml07 Mingliang Liu added a comment -

        I mean configured value. Sorry.

        Show
        liuml07 Mingliang Liu added a comment - I mean configured value . Sorry.
        Hide
        eclark Elliott Clark added a comment -

        Ping?

        This is running in production and removes thousands of tcp resets.

        Show
        eclark Elliott Clark added a comment - Ping? This is running in production and removes thousands of tcp resets.
        Hide
        cmccabe Colin P. McCabe added a comment -

        +1. Thanks, Elliott Clark.

        Show
        cmccabe Colin P. McCabe added a comment - +1. Thanks, Elliott Clark .
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #9226 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9226/)
        HDFS-9669. TcpPeerServer should respect ipc.server.listen.queue.size (cmccabe: rev 2da03b48eba53d4dec2a77209ad9835d808171d1)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/net/TcpPeerServer.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/SecureDataNodeStarter.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9226 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9226/ ) HDFS-9669 . TcpPeerServer should respect ipc.server.listen.queue.size (cmccabe: rev 2da03b48eba53d4dec2a77209ad9835d808171d1) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/net/TcpPeerServer.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/SecureDataNodeStarter.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        eclark Elliott Clark added a comment -

        Thanks Colin P. McCabe any chance of getting this on branch-2 ?

        Show
        eclark Elliott Clark added a comment - Thanks Colin P. McCabe any chance of getting this on branch-2 ?
        Hide
        cmccabe Colin P. McCabe added a comment -

        Hmm. I committed this to trunk, branch-2, branch-2.8, and branch-2.7 (isn't branching fun?), for a fix version of 2.7.3 Did I miss a backport?

        Show
        cmccabe Colin P. McCabe added a comment - Hmm. I committed this to trunk, branch-2, branch-2.8, and branch-2.7 (isn't branching fun?), for a fix version of 2.7.3 Did I miss a backport?
        Hide
        eclark Elliott Clark added a comment -

        Nope I just missed it on a pull. Sorry about the confusion. Thanks for the many many backports Colin P. McCabe

        Show
        eclark Elliott Clark added a comment - Nope I just missed it on a pull. Sorry about the confusion. Thanks for the many many backports Colin P. McCabe
        Hide
        ctrezzo Chris Trezzo added a comment -

        Adding 2.6.5 to the target versions with the intention of backporting this to branch-2.6. Please let me know if you think otherwise. Thanks!

        Show
        ctrezzo Chris Trezzo added a comment - Adding 2.6.5 to the target versions with the intention of backporting this to branch-2.6. Please let me know if you think otherwise. Thanks!
        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        Closing the JIRA as part of 2.7.3 release.

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - Closing the JIRA as part of 2.7.3 release.
        Hide
        sjlee0 Sangjin Lee added a comment -

        Cherry-picked it to 2.6.5 (trivial).

        Show
        sjlee0 Sangjin Lee added a comment - Cherry-picked it to 2.6.5 (trivial).
        Hide
        ctrezzo Chris Trezzo added a comment -

        Thanks!

        Show
        ctrezzo Chris Trezzo added a comment - Thanks!

          People

          • Assignee:
            eclark Elliott Clark
            Reporter:
            eclark Elliott Clark
          • Votes:
            0 Vote for this issue
            Watchers:
            12 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development