Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13351

TestDFSClientSocketSize buffer size tests are flaky

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.8.0, 3.0.0-alpha1
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None

      Description

      TestDFSClientSocketSize has two tests that assert that a value that was set via java.net.Socket#setSendBufferSize is equal to the value subsequently returned by java.net.Socket#getSendBufferSize.

      These tests are flaky when we run them. The occasionally fail.

      This is expected behavior, actually, because Socket#setSendBufferSize()is only a hint. (Similar to how the underlying libc setsockopt(SO_SNDBUF) works).

      1. HADOOP-13351.005.patch
        5 kB
        Mingliang Liu
      2. HADOOP-13351.006.patch
        5 kB
        Mingliang Liu
      3. HADOOP-13351.007.patch
        5 kB
        Mingliang Liu
      4. HADOOP-13551.001.patch
        2 kB
        Aaron Fabbri
      5. HADOOP-13551.002.patch
        3 kB
        Aaron Fabbri
      6. HADOOP-13551.003.patch
        3 kB
        Aaron Fabbri
      7. HADOOP-13551.004.patch
        3 kB
        Aaron Fabbri

        Activity

        Hide
        liuml07 Mingliang Liu added a comment -

        Thanks for reporting this, Aaron Fabbri. I can remember my adding this UT. I assign this to myself. Feel free to grab it if you have a plan to fix.

        Show
        liuml07 Mingliang Liu added a comment - Thanks for reporting this, Aaron Fabbri . I can remember my adding this UT. I assign this to myself. Feel free to grab it if you have a plan to fix.
        Hide
        fabbri Aaron Fabbri added a comment -

        Example error message (from Xiao Chen):

        Send buffer size should be the default value. expected:<131072> but was:<131071>
        

        Stacktrace

        java.lang.AssertionError: Send buffer size should be the default value. expected:<131072> but was:<131071>
        	at org.junit.Assert.fail(Assert.java:88)
        	at org.junit.Assert.failNotEquals(Assert.java:743)
        	at org.junit.Assert.assertEquals(Assert.java:118)
        	at org.junit.Assert.assertEquals(Assert.java:555)
        	at org.apache.hadoop.hdfs.TestDFSClientSocketSize.testDefaultSendBufferSize(TestDFSClientSocketSize.java:53)
        
        Show
        fabbri Aaron Fabbri added a comment - Example error message (from Xiao Chen ): Send buffer size should be the default value. expected:<131072> but was:<131071> Stacktrace java.lang.AssertionError: Send buffer size should be the default value. expected:<131072> but was:<131071> at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:555) at org.apache.hadoop.hdfs.TestDFSClientSocketSize.testDefaultSendBufferSize(TestDFSClientSocketSize.java:53)
        Hide
        fabbri Aaron Fabbri added a comment -

        Mingliang Liu I'm already working on this.. Attaching patch.

        Show
        fabbri Aaron Fabbri added a comment - Mingliang Liu I'm already working on this.. Attaching patch.
        Hide
        liuml07 Mingliang Liu added a comment -

        Thanks Aaron Fabbri for working on this. I can not assign this to you, so please pick it up.

        Show
        liuml07 Mingliang Liu added a comment - Thanks Aaron Fabbri for working on this. I can not assign this to you, so please pick it up.
        Hide
        liuml07 Mingliang Liu added a comment - - edited

        The v1 patch is also the best effort I can figure out so far. Thanks for the patch.

        void testSpecifiedSendBufferSize() throws IOException {
            final int mySendBufferSize = 64 * 1024;  // 64 KB
        

        To make the test covers the specified value, I'd suggest we change mySendBufferSize to be a different value (say, 512KB). The default value is 128KB >= 64KB. As a result, the test can pass without the following the statement which sets the specific value:

        conf.setInt(DFS_CLIENT_SOCKET_SEND_BUFFER_SIZE_KEY, mySendBufferSize);
        
        Show
        liuml07 Mingliang Liu added a comment - - edited The v1 patch is also the best effort I can figure out so far. Thanks for the patch. void testSpecifiedSendBufferSize() throws IOException { final int mySendBufferSize = 64 * 1024; // 64 KB To make the test covers the specified value, I'd suggest we change mySendBufferSize to be a different value (say, 512KB). The default value is 128KB >= 64KB. As a result, the test can pass without the following the statement which sets the specific value: conf.setInt(DFS_CLIENT_SOCKET_SEND_BUFFER_SIZE_KEY, mySendBufferSize);
        Hide
        fabbri Aaron Fabbri added a comment -

        Thank you for review Mingliang Liu. I do not have ability to assign JIRA issues here.. Looking into that.

        On the code, I understand what you are saying. My thinking here is that we should at least be getting a minimal send buffer, so let's test that. Since the socket option is only a hint, I'm concerned increasing it too much could add another cause of test flakiness: the OS declining to set a large send buffer size for a socket. So I'd personally lean towards keeping it as is in the patch or removing the two test cases.

        Show
        fabbri Aaron Fabbri added a comment - Thank you for review Mingliang Liu . I do not have ability to assign JIRA issues here.. Looking into that. On the code, I understand what you are saying. My thinking here is that we should at least be getting a minimal send buffer, so let's test that. Since the socket option is only a hint, I'm concerned increasing it too much could add another cause of test flakiness: the OS declining to set a large send buffer size for a socket. So I'd personally lean towards keeping it as is in the patch or removing the two test cases.
        Hide
        liuml07 Mingliang Liu added a comment - - edited

        the OS declining to set a large send buffer size for a socket.

        Thanks Aaron Fabbri for the explanation. In StandardSocketOptions, an attempt to set the socket send buffer to larger than its maximum size causes it to be set to its maximum size (refer to link here). I think the value for Socket would work in the same way. This way, a larger value like 512KB would not cause problems for set operation. However, the test may still fail since the maximum size, as the default size, is system dependent.

        That being said, testing the specified value is indeed tricky. To make the test assertion covers this case, how about comparing the hinted buffer size? I see very limited chance that the test fails, but it does confirm that, the config key works as expected.

        Sample code as following:

          @Test
          public void testSpecifiedSendBufferSize() throws IOException {
            conf.setInt(DFS_CLIENT_SOCKET_SEND_BUFFER_SIZE_KEY, 256 * 1024); // a big hint
            final Socket s1 = createSocket();
            conf.setInt(DFS_CLIENT_SOCKET_SEND_BUFFER_SIZE_KEY, 1024);          // a small hint
            final Socket s2 = createSocket();
            LOG.info("The real big buffer size is {}, small buffer size is {}",
                s1.getSendBufferSize(), s2.getSendBufferSize());
            assertTrue(s1.getSendBufferSize() > s2.getSendBufferSize());
        }
        

        What's your thought?

        Show
        liuml07 Mingliang Liu added a comment - - edited the OS declining to set a large send buffer size for a socket. Thanks Aaron Fabbri for the explanation. In StandardSocketOptions , an attempt to set the socket send buffer to larger than its maximum size causes it to be set to its maximum size (refer to link here ). I think the value for Socket would work in the same way. This way, a larger value like 512KB would not cause problems for set operation. However, the test may still fail since the maximum size, as the default size, is system dependent. That being said, testing the specified value is indeed tricky. To make the test assertion covers this case, how about comparing the hinted buffer size? I see very limited chance that the test fails, but it does confirm that, the config key works as expected. Sample code as following: @Test public void testSpecifiedSendBufferSize() throws IOException { conf.setInt(DFS_CLIENT_SOCKET_SEND_BUFFER_SIZE_KEY, 256 * 1024); // a big hint final Socket s1 = createSocket(); conf.setInt(DFS_CLIENT_SOCKET_SEND_BUFFER_SIZE_KEY, 1024); // a small hint final Socket s2 = createSocket(); LOG.info( "The real big buffer size is {}, small buffer size is {}" , s1.getSendBufferSize(), s2.getSendBufferSize()); assertTrue(s1.getSendBufferSize() > s2.getSendBufferSize()); } What's your thought?
        Hide
        fabbri Aaron Fabbri added a comment -

        Mingliang Liu seems like your suggestion would usually work. Still, behavior of get/set on this socket option is system dependent. e.g. on Linux see /proc/sys/net/core/wmem_max etc.. Going through the system's setsockopt/getsockopt to unit test hadoop socket option plumbing seems questionable. I'd suggest removing the tests, but I would be ok with using your suggestion in a patch that includes a comment that the tests will be removed if found to still be flaky. I'll attach a v2 patch like that.. Tell me what you think.

        Show
        fabbri Aaron Fabbri added a comment - Mingliang Liu seems like your suggestion would usually work. Still, behavior of get/set on this socket option is system dependent. e.g. on Linux see /proc/sys/net/core/wmem_max etc.. Going through the system's setsockopt/getsockopt to unit test hadoop socket option plumbing seems questionable. I'd suggest removing the tests, but I would be ok with using your suggestion in a patch that includes a comment that the tests will be removed if found to still be flaky. I'll attach a v2 patch like that.. Tell me what you think.
        Hide
        fabbri Aaron Fabbri added a comment -

        Attaching v2 patch. I also noticed and fixed other issues: 1. testDefaultSendBufferSize() assumes that it is executed first. I believe JUnit does not guarantee test execution order. I addressed this by always creating a fresh Configuration in a new @Before setUp() function.
        2. tearDown() prints a message that it is closing the DFSClient socket, but does not close it. I added a close() there.

        Show
        fabbri Aaron Fabbri added a comment - Attaching v2 patch. I also noticed and fixed other issues: 1. testDefaultSendBufferSize() assumes that it is executed first. I believe JUnit does not guarantee test execution order. I addressed this by always creating a fresh Configuration in a new @Before setUp() function. 2. tearDown() prints a message that it is closing the DFSClient socket, but does not close it. I added a close() there.
        Hide
        liuml07 Mingliang Liu added a comment -

        Thanks for updating the v2 patch. I like it.

        One minor comment is in testSpecifiedSendBufferSize we also close the s1/s2 sockets.

        Show
        liuml07 Mingliang Liu added a comment - Thanks for updating the v2 patch. I like it. One minor comment is in testSpecifiedSendBufferSize we also close the s1/s2 sockets.
        Hide
        fabbri Aaron Fabbri added a comment -

        Attaching v3 patch that also closes the two sockets from the new test.

        Show
        fabbri Aaron Fabbri added a comment - Attaching v3 patch that also closes the two sockets from the new test.
        Hide
        liuml07 Mingliang Liu added a comment -

        Thanks for updating the patch, Aaron Fabbri.

        Is it better to use try-with-resource for the s1/s2 sockets? My concern is that, if there is any exception thrown before closing or if there is IOE thrown when closing s1, the s2 might not be closed correctly.

        Otherwise +1 (non-binding) pending on Jenkins.

        Show
        liuml07 Mingliang Liu added a comment - Thanks for updating the patch, Aaron Fabbri . Is it better to use try-with-resource for the s1/s2 sockets? My concern is that, if there is any exception thrown before closing or if there is IOE thrown when closing s1 , the s2 might not be closed correctly. Otherwise +1 (non-binding) pending on Jenkins.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 21s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 7m 39s trunk passed
        +1 compile 0m 50s trunk passed
        +1 checkstyle 0m 27s trunk passed
        +1 mvnsite 0m 59s trunk passed
        +1 mvneclipse 0m 12s trunk passed
        +1 findbugs 1m 50s trunk passed
        +1 javadoc 0m 57s trunk passed
        +1 mvninstall 0m 57s the patch passed
        +1 compile 0m 49s the patch passed
        +1 javac 0m 49s the patch passed
        -0 checkstyle 0m 24s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 1 unchanged - 0 fixed = 3 total (was 1)
        +1 mvnsite 0m 55s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 55s the patch passed
        +1 javadoc 0m 55s the patch passed
        -1 unit 62m 53s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 18s The patch does not generate ASF License warnings.
        84m 31s



        Reason Tests
        Failed junit tests hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation
          hadoop.hdfs.server.datanode.TestLargeBlockReport



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12817218/HADOOP-13551.003.patch
        JIRA Issue HADOOP-13351
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux deb6d5228988 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 06c56ff
        Default Java 1.8.0_91
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9975/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9975/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9975/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9975/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 39s trunk passed +1 compile 0m 50s trunk passed +1 checkstyle 0m 27s trunk passed +1 mvnsite 0m 59s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 50s trunk passed +1 javadoc 0m 57s trunk passed +1 mvninstall 0m 57s the patch passed +1 compile 0m 49s the patch passed +1 javac 0m 49s the patch passed -0 checkstyle 0m 24s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 1 unchanged - 0 fixed = 3 total (was 1) +1 mvnsite 0m 55s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 55s the patch passed +1 javadoc 0m 55s the patch passed -1 unit 62m 53s hadoop-hdfs in the patch failed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 84m 31s Reason Tests Failed junit tests hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation   hadoop.hdfs.server.datanode.TestLargeBlockReport Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12817218/HADOOP-13551.003.patch JIRA Issue HADOOP-13351 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux deb6d5228988 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 06c56ff Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9975/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9975/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9975/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9975/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        ajisakaa Akira Ajisaka added a comment -

        In addition to Mingliang's comment, would you fix the checkstyle warnings?

        Show
        ajisakaa Akira Ajisaka added a comment - In addition to Mingliang's comment, would you fix the checkstyle warnings?
        Hide
        fabbri Aaron Fabbri added a comment -

        Attaching v4 patch with checkstyle fixes and try-with-resources for new test code

        Show
        fabbri Aaron Fabbri added a comment - Attaching v4 patch with checkstyle fixes and try-with-resources for new test code
        Hide
        liuml07 Mingliang Liu added a comment -

        Sorry I missed one point in my last comments. The cluster should also be closed when we create a new cluster for s2. I refactored the code based on Aaron Fabbri's v4 patch, and attached the v5 patch. Please kindly review and update it if Jenkins fails. I assigned this JIRA back to Aaron Fabbri. Thanks!

        Show
        liuml07 Mingliang Liu added a comment - Sorry I missed one point in my last comments. The cluster should also be closed when we create a new cluster for s2 . I refactored the code based on Aaron Fabbri 's v4 patch, and attached the v5 patch. Please kindly review and update it if Jenkins fails. I assigned this JIRA back to Aaron Fabbri . Thanks!
        Hide
        liuml07 Mingliang Liu added a comment -

        The License header should not be changed any way in the v5 patch. My IntelliJ boldly did it for me without my grant. Sorry about that.

        Show
        liuml07 Mingliang Liu added a comment - The License header should not be changed any way in the v5 patch. My IntelliJ boldly did it for me without my grant. Sorry about that.
        Hide
        fabbri Aaron Fabbri added a comment -

        Mingliang Liu v6 looks good to me +1 (non-binding). Glad we could get some cleanup improvements in, in addition to making the test failures less likely.

        Show
        fabbri Aaron Fabbri added a comment - Mingliang Liu v6 looks good to me +1 (non-binding). Glad we could get some cleanup improvements in, in addition to making the test failures less likely.
        Hide
        liuml07 Mingliang Liu added a comment -

        Thanks Aaron Fabbri. Akira Ajisaka hopefully will give it another review and commit it.

        Show
        liuml07 Mingliang Liu added a comment - Thanks Aaron Fabbri . Akira Ajisaka hopefully will give it another review and commit it.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 26s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 7m 3s trunk passed
        +1 compile 0m 46s trunk passed
        +1 checkstyle 0m 26s trunk passed
        +1 mvnsite 0m 52s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 1m 42s trunk passed
        +1 javadoc 0m 56s trunk passed
        +1 mvninstall 0m 47s the patch passed
        +1 compile 0m 42s the patch passed
        +1 javac 0m 42s the patch passed
        +1 checkstyle 0m 23s the patch passed
        +1 mvnsite 0m 49s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 46s the patch passed
        +1 javadoc 0m 54s the patch passed
        -1 unit 60m 26s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 22s The patch does not generate ASF License warnings.
        80m 46s



        Reason Tests
        Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12817805/HADOOP-13551.004.patch
        JIRA Issue HADOOP-13351
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 7a1306f2ff55 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / d180505
        Default Java 1.8.0_91
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9994/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9994/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9994/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 26s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 3s trunk passed +1 compile 0m 46s trunk passed +1 checkstyle 0m 26s trunk passed +1 mvnsite 0m 52s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 42s trunk passed +1 javadoc 0m 56s trunk passed +1 mvninstall 0m 47s the patch passed +1 compile 0m 42s the patch passed +1 javac 0m 42s the patch passed +1 checkstyle 0m 23s the patch passed +1 mvnsite 0m 49s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 46s the patch passed +1 javadoc 0m 54s the patch passed -1 unit 60m 26s hadoop-hdfs in the patch failed. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 80m 46s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12817805/HADOOP-13551.004.patch JIRA Issue HADOOP-13351 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 7a1306f2ff55 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d180505 Default Java 1.8.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9994/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9994/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9994/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 6m 51s trunk passed
        +1 compile 0m 46s trunk passed
        +1 checkstyle 0m 25s trunk passed
        +1 mvnsite 0m 51s trunk passed
        +1 mvneclipse 0m 12s trunk passed
        +1 findbugs 1m 46s trunk passed
        +1 javadoc 0m 56s trunk passed
        +1 mvninstall 0m 48s the patch passed
        +1 compile 0m 43s the patch passed
        +1 javac 0m 43s the patch passed
        +1 checkstyle 0m 24s the patch passed
        +1 mvnsite 0m 49s the patch passed
        +1 mvneclipse 0m 9s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 52s the patch passed
        +1 javadoc 0m 54s the patch passed
        -1 unit 58m 24s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 26s The patch does not generate ASF License warnings.
        78m 28s



        Reason Tests
        Failed junit tests hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12817816/HADOOP-13351.005.patch
        JIRA Issue HADOOP-13351
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux cfeaea5012ff 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / d180505
        Default Java 1.8.0_91
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9995/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9995/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9995/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 51s trunk passed +1 compile 0m 46s trunk passed +1 checkstyle 0m 25s trunk passed +1 mvnsite 0m 51s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 46s trunk passed +1 javadoc 0m 56s trunk passed +1 mvninstall 0m 48s the patch passed +1 compile 0m 43s the patch passed +1 javac 0m 43s the patch passed +1 checkstyle 0m 24s the patch passed +1 mvnsite 0m 49s the patch passed +1 mvneclipse 0m 9s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 52s the patch passed +1 javadoc 0m 54s the patch passed -1 unit 58m 24s hadoop-hdfs in the patch failed. +1 asflicense 0m 26s The patch does not generate ASF License warnings. 78m 28s Reason Tests Failed junit tests hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12817816/HADOOP-13351.005.patch JIRA Issue HADOOP-13351 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux cfeaea5012ff 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d180505 Default Java 1.8.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9995/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9995/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9995/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 7m 59s trunk passed
        +1 compile 0m 54s trunk passed
        +1 checkstyle 0m 28s trunk passed
        +1 mvnsite 1m 4s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 2m 1s trunk passed
        +1 javadoc 1m 2s trunk passed
        +1 mvninstall 0m 55s the patch passed
        +1 compile 0m 49s the patch passed
        +1 javac 0m 49s the patch passed
        -0 checkstyle 0m 25s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1)
        +1 mvnsite 0m 59s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 2m 10s the patch passed
        +1 javadoc 0m 58s the patch passed
        -1 unit 64m 18s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 21s The patch does not generate ASF License warnings.
        87m 11s



        Reason Tests
        Failed junit tests hadoop.hdfs.server.namenode.web.resources.TestWebHdfsDataLocality
          hadoop.hdfs.server.blockmanagement.TestBlockManager
          hadoop.hdfs.TestRollingUpgrade
          hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12817819/HADOOP-13351.006.patch
        JIRA Issue HADOOP-13351
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 081c53604a3b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / d180505
        Default Java 1.8.0_91
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9996/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9996/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9996/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9996/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 59s trunk passed +1 compile 0m 54s trunk passed +1 checkstyle 0m 28s trunk passed +1 mvnsite 1m 4s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 2m 1s trunk passed +1 javadoc 1m 2s trunk passed +1 mvninstall 0m 55s the patch passed +1 compile 0m 49s the patch passed +1 javac 0m 49s the patch passed -0 checkstyle 0m 25s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1) +1 mvnsite 0m 59s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 10s the patch passed +1 javadoc 0m 58s the patch passed -1 unit 64m 18s hadoop-hdfs in the patch failed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 87m 11s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.web.resources.TestWebHdfsDataLocality   hadoop.hdfs.server.blockmanagement.TestBlockManager   hadoop.hdfs.TestRollingUpgrade   hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12817819/HADOOP-13351.006.patch JIRA Issue HADOOP-13351 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 081c53604a3b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d180505 Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9996/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9996/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9996/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9996/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        ajisakaa Akira Ajisaka added a comment -

        Hi Mingliang Liu, would you fix the checkstyle warning? I'm +1 if that is addressed.

        Show
        ajisakaa Akira Ajisaka added a comment - Hi Mingliang Liu , would you fix the checkstyle warning? I'm +1 if that is addressed.
        Hide
        liuml07 Mingliang Liu added a comment -

        Sure, thanks for the review.

        Show
        liuml07 Mingliang Liu added a comment - Sure, thanks for the review.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 7m 27s trunk passed
        +1 compile 0m 48s trunk passed
        +1 checkstyle 0m 27s trunk passed
        +1 mvnsite 0m 56s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 1m 48s trunk passed
        +1 javadoc 0m 55s trunk passed
        +1 mvninstall 0m 48s the patch passed
        +1 compile 0m 45s the patch passed
        +1 javac 0m 45s the patch passed
        +1 checkstyle 0m 23s the patch passed
        +1 mvnsite 0m 53s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 51s the patch passed
        +1 javadoc 0m 53s the patch passed
        -1 unit 59m 49s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 22s The patch does not generate ASF License warnings.
        80m 41s



        Reason Tests
        Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics
          hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12818004/HADOOP-13351.007.patch
        JIRA Issue HADOOP-13351
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux a2d753649b45 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 6cf0175
        Default Java 1.8.0_91
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10001/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10001/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10001/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 27s trunk passed +1 compile 0m 48s trunk passed +1 checkstyle 0m 27s trunk passed +1 mvnsite 0m 56s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 48s trunk passed +1 javadoc 0m 55s trunk passed +1 mvninstall 0m 48s the patch passed +1 compile 0m 45s the patch passed +1 javac 0m 45s the patch passed +1 checkstyle 0m 23s the patch passed +1 mvnsite 0m 53s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 51s the patch passed +1 javadoc 0m 53s the patch passed -1 unit 59m 49s hadoop-hdfs in the patch failed. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 80m 41s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12818004/HADOOP-13351.007.patch JIRA Issue HADOOP-13351 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux a2d753649b45 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6cf0175 Default Java 1.8.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10001/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10001/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10001/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        ajisakaa Akira Ajisaka added a comment -

        +1, committing this.

        Show
        ajisakaa Akira Ajisaka added a comment - +1, committing this.
        Hide
        ajisakaa Akira Ajisaka added a comment -

        Committed this to trunk, branch-2, and branch-2.8. Thanks Aaron Fabbri and Mingliang Liu for the contribution!

        Show
        ajisakaa Akira Ajisaka added a comment - Committed this to trunk, branch-2, and branch-2.8. Thanks Aaron Fabbri and Mingliang Liu for the contribution!
        Hide
        liuml07 Mingliang Liu added a comment -

        Thanks Akira Ajisaka for the review and commit, thanks Aaron Fabbri for working on this.

        Show
        liuml07 Mingliang Liu added a comment - Thanks Akira Ajisaka for the review and commit, thanks Aaron Fabbri for working on this.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-trunk-Commit #10101 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10101/)
        HADOOP-13351. TestDFSClientSocketSize buffer size tests are flaky. (aajisaka: rev 5537c6b23430285ebee33c6d9b69d3ec1e9b17b1)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientSocketSize.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #10101 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10101/ ) HADOOP-13351 . TestDFSClientSocketSize buffer size tests are flaky. (aajisaka: rev 5537c6b23430285ebee33c6d9b69d3ec1e9b17b1) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientSocketSize.java

          People

          • Assignee:
            fabbri Aaron Fabbri
            Reporter:
            fabbri Aaron Fabbri
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development