Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-3738

TestDFSClientRetries#testFailuresArePerOperation sets incorrect timeout config

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: 2.0.2-alpha
    • Component/s: test
    • Labels:
      None

      Description

      TestDFSClientRetries#testFailuresArePerOperation involves testing retries by making use of expected timeouts. However, this test sets the wrong config to lower the timeout, and thus takes far longer than it should.

      1. HDFS-3738.patch
        2 kB
        Aaron T. Myers
      2. HDFS-3738.patch
        1 kB
        Aaron T. Myers

        Activity

        Aaron T. Myers created issue -
        Hide
        Aaron T. Myers added a comment -

        The attached patch lowers the timeout config that matters to 2 seconds instead of 1 minute. Without this change, this test runs in 13 minutes on my machine. With this change the test runs in 30 seconds.

        Show
        Aaron T. Myers added a comment - The attached patch lowers the timeout config that matters to 2 seconds instead of 1 minute. Without this change, this test runs in 13 minutes on my machine. With this change the test runs in 30 seconds.
        Aaron T. Myers made changes -
        Field Original Value New Value
        Attachment HDFS-3738.patch [ 12538411 ]
        Aaron T. Myers made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Todd Lipcon added a comment -

        Was this a regression? Or did we commit a test that runs for 13 minutes? I'm nervous that we might have changed the semantics of some of the configs, perhaps in HDFS-3504?

        Show
        Todd Lipcon added a comment - Was this a regression? Or did we commit a test that runs for 13 minutes? I'm nervous that we might have changed the semantics of some of the configs, perhaps in HDFS-3504 ?
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12538411/HDFS-3738.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2922//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2922//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12538411/HDFS-3738.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2922//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2922//console This message is automatically generated.
        Hide
        Aaron T. Myers added a comment -

        I took a look into this and concluded that the performance of this test is indeed a regression, inadvertently caused by HDFS-3401. Before that change, the test was using the broadcast address as its fake address for DNs, which is resolvable but will return an error immediately upon attempting a connection. HDFS-3401 changed this to the IP address "1.2.3.4" which actually requires the socket timeout be hit, which slowed down the test. So, setting the timeout low (as this patch does) is a perfectly reasonable thing to do, it just so happens that it wasn't necessary before.

        I also checked to see if any of the other tests touched by HDFS-3401 had been inadvertently slowed down, and found that one had. This updated patch improves the running time of TestPBHelper from ~11.5 seconds to ~0.5 seconds, which is similar to the running time before HDFS-3401.

        Show
        Aaron T. Myers added a comment - I took a look into this and concluded that the performance of this test is indeed a regression, inadvertently caused by HDFS-3401 . Before that change, the test was using the broadcast address as its fake address for DNs, which is resolvable but will return an error immediately upon attempting a connection. HDFS-3401 changed this to the IP address "1.2.3.4" which actually requires the socket timeout be hit, which slowed down the test. So, setting the timeout low (as this patch does) is a perfectly reasonable thing to do, it just so happens that it wasn't necessary before. I also checked to see if any of the other tests touched by HDFS-3401 had been inadvertently slowed down, and found that one had. This updated patch improves the running time of TestPBHelper from ~11.5 seconds to ~0.5 seconds, which is similar to the running time before HDFS-3401 .
        Aaron T. Myers made changes -
        Attachment HDFS-3738.patch [ 12538478 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12538478/HDFS-3738.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 2 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

        org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS
        org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics
        org.apache.hadoop.hdfs.TestPersistBlocks

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2925//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2925//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12538478/HDFS-3738.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics org.apache.hadoop.hdfs.TestPersistBlocks +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2925//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2925//console This message is automatically generated.
        Hide
        Aaron T. Myers added a comment -

        I'm confident that the test failures are unrelated.

        Show
        Aaron T. Myers added a comment - I'm confident that the test failures are unrelated.
        Hide
        Eli Collins added a comment -

        +1

        And thanks for the spelunking that led to HDFS-3401.

        Show
        Eli Collins added a comment - +1 And thanks for the spelunking that led to HDFS-3401 .
        Hide
        Aaron T. Myers added a comment -

        Thanks a lot for the review, Eli. I've just committed this to trunk and branch-2.

        Show
        Aaron T. Myers added a comment - Thanks a lot for the review, Eli. I've just committed this to trunk and branch-2.
        Aaron T. Myers made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags Reviewed [ 10343 ]
        Fix Version/s 2.2.0-alpha [ 12322472 ]
        Resolution Fixed [ 1 ]
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #2611 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2611/)
        HDFS-3738. TestDFSClientRetries#testFailuresArePerOperation sets incorrect timeout config. Contributed by Aaron T. Myers. (Revision 1367844)

        Result = SUCCESS
        atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1367844
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocolPB/TestPBHelper.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2611 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2611/ ) HDFS-3738 . TestDFSClientRetries#testFailuresArePerOperation sets incorrect timeout config. Contributed by Aaron T. Myers. (Revision 1367844) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1367844 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocolPB/TestPBHelper.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #2546 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2546/)
        HDFS-3738. TestDFSClientRetries#testFailuresArePerOperation sets incorrect timeout config. Contributed by Aaron T. Myers. (Revision 1367844)

        Result = SUCCESS
        atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1367844
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocolPB/TestPBHelper.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2546 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2546/ ) HDFS-3738 . TestDFSClientRetries#testFailuresArePerOperation sets incorrect timeout config. Contributed by Aaron T. Myers. (Revision 1367844) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1367844 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocolPB/TestPBHelper.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #2564 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2564/)
        HDFS-3738. TestDFSClientRetries#testFailuresArePerOperation sets incorrect timeout config. Contributed by Aaron T. Myers. (Revision 1367844)

        Result = FAILURE
        atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1367844
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocolPB/TestPBHelper.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2564 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2564/ ) HDFS-3738 . TestDFSClientRetries#testFailuresArePerOperation sets incorrect timeout config. Contributed by Aaron T. Myers. (Revision 1367844) Result = FAILURE atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1367844 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocolPB/TestPBHelper.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1122 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1122/)
        HDFS-3738. TestDFSClientRetries#testFailuresArePerOperation sets incorrect timeout config. Contributed by Aaron T. Myers. (Revision 1367844)

        Result = FAILURE
        atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1367844
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocolPB/TestPBHelper.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1122 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1122/ ) HDFS-3738 . TestDFSClientRetries#testFailuresArePerOperation sets incorrect timeout config. Contributed by Aaron T. Myers. (Revision 1367844) Result = FAILURE atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1367844 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocolPB/TestPBHelper.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1154 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1154/)
        HDFS-3738. TestDFSClientRetries#testFailuresArePerOperation sets incorrect timeout config. Contributed by Aaron T. Myers. (Revision 1367844)

        Result = FAILURE
        atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1367844
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocolPB/TestPBHelper.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1154 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1154/ ) HDFS-3738 . TestDFSClientRetries#testFailuresArePerOperation sets incorrect timeout config. Contributed by Aaron T. Myers. (Revision 1367844) Result = FAILURE atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1367844 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocolPB/TestPBHelper.java
        Arun C Murthy made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        2m 20s 1 Aaron T. Myers 30/Jul/12 21:52
        Patch Available Patch Available Resolved Resolved
        1d 5h 31m 1 Aaron T. Myers 01/Aug/12 03:23
        Resolved Resolved Closed Closed
        71d 15h 22m 1 Arun C Murthy 11/Oct/12 18:46

          People

          • Assignee:
            Aaron T. Myers
            Reporter:
            Aaron T. Myers
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development