Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-3462

TestDFSClientRetries.busyTest() should restore default xceiver count in the config.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha1
    • Fix Version/s: 3.0.0-alpha1
    • Component/s: test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      TestDFSClientRetries.busyTest() sets maximum xceiver count in the config to a small value, but does not restore it back to the default. As the result subsequent tests can fail, especially on slower or busy machines.

      1. testGetFileChecksum.log.htm
        586 kB
        Konstantin Shvachko
      2. HDFS-3462.patch
        1 kB
        madhukara phatak

        Issue Links

          Activity

          Hide
          shv Konstantin Shvachko added a comment -

          Attaching log from Jenkins build for HDFS-3368 with failed testGetFileChecksum().
          Suggested fix is to restore the default value for max xceiver count after all busyTest() are done.

          Show
          shv Konstantin Shvachko added a comment - Attaching log from Jenkins build for HDFS-3368 with failed testGetFileChecksum(). Suggested fix is to restore the default value for max xceiver count after all busyTest() are done.
          Hide
          phatak.dev madhukara phatak added a comment -

          Fixed busyTest method to restore the original value of xceivers.

          Show
          phatak.dev madhukara phatak added a comment - Fixed busyTest method to restore the original value of xceivers.
          Hide
          qwertymaniac Harsh J added a comment -

          +1 pending Jenkins. Thanks!

          Show
          qwertymaniac Harsh J added a comment - +1 pending Jenkins. Thanks!
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12529680/HDFS-3462.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2530//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2530//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12529680/HDFS-3462.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2530//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2530//console This message is automatically generated.
          Hide
          qwertymaniac Harsh J added a comment -

          Committed to trunk. Thanks for your continued contributions Madhukara!

          Show
          qwertymaniac Harsh J added a comment - Committed to trunk. Thanks for your continued contributions Madhukara!
          Hide
          hudson Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2296 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2296/)
          HDFS-3462. TestDFSClientRetries.busyTest() should restore default xceiver count in the config. Contributed by Madhukara Phatak. (harsh) (Revision 1343798)

          Result = SUCCESS
          harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1343798
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java
          Show
          hudson Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2296 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2296/ ) HDFS-3462 . TestDFSClientRetries.busyTest() should restore default xceiver count in the config. Contributed by Madhukara Phatak. (harsh) (Revision 1343798) Result = SUCCESS harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1343798 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java
          Hide
          hudson Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2369 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2369/)
          HDFS-3462. TestDFSClientRetries.busyTest() should restore default xceiver count in the config. Contributed by Madhukara Phatak. (harsh) (Revision 1343798)

          Result = SUCCESS
          harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1343798
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java
          Show
          hudson Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2369 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2369/ ) HDFS-3462 . TestDFSClientRetries.busyTest() should restore default xceiver count in the config. Contributed by Madhukara Phatak. (harsh) (Revision 1343798) Result = SUCCESS harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1343798 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java
          Hide
          hudson Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2315 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2315/)
          HDFS-3462. TestDFSClientRetries.busyTest() should restore default xceiver count in the config. Contributed by Madhukara Phatak. (harsh) (Revision 1343798)

          Result = FAILURE
          harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1343798
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java
          Show
          hudson Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2315 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2315/ ) HDFS-3462 . TestDFSClientRetries.busyTest() should restore default xceiver count in the config. Contributed by Madhukara Phatak. (harsh) (Revision 1343798) Result = FAILURE harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1343798 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java
          Hide
          shv Konstantin Shvachko added a comment -

          + int originalXcievers = conf.getInt(DFSConfigKeys.DFS_DATANODE_MAX_RECEIVER_THREADS_KEY,0);

          should use DFS_DATANODE_MAX_RECEIVER_THREADS_DEFAULT rather than 0.

          Show
          shv Konstantin Shvachko added a comment - + int originalXcievers = conf.getInt(DFSConfigKeys.DFS_DATANODE_MAX_RECEIVER_THREADS_KEY,0); should use DFS_DATANODE_MAX_RECEIVER_THREADS_DEFAULT rather than 0.
          Hide
          hudson Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1061 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1061/)
          HDFS-3462. TestDFSClientRetries.busyTest() should restore default xceiver count in the config. Contributed by Madhukara Phatak. (harsh) (Revision 1343798)

          Result = SUCCESS
          harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1343798
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java
          Show
          hudson Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1061 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1061/ ) HDFS-3462 . TestDFSClientRetries.busyTest() should restore default xceiver count in the config. Contributed by Madhukara Phatak. (harsh) (Revision 1343798) Result = SUCCESS harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1343798 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java
          Hide
          qwertymaniac Harsh J added a comment -

          Konstantin, I thought it to be fine for that test case but I see now what it could cause if there were no defaults. I've filed HDFS-3476 for addressing this. Sorry about this!

          Show
          qwertymaniac Harsh J added a comment - Konstantin, I thought it to be fine for that test case but I see now what it could cause if there were no defaults. I've filed HDFS-3476 for addressing this. Sorry about this!
          Hide
          hudson Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1095 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1095/)
          HDFS-3462. TestDFSClientRetries.busyTest() should restore default xceiver count in the config. Contributed by Madhukara Phatak. (harsh) (Revision 1343798)

          Result = SUCCESS
          harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1343798
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java
          Show
          hudson Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1095 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1095/ ) HDFS-3462 . TestDFSClientRetries.busyTest() should restore default xceiver count in the config. Contributed by Madhukara Phatak. (harsh) (Revision 1343798) Result = SUCCESS harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1343798 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java
          Hide
          hudson Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2374 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2374/)
          HDFS-3476. Correct the default used in TestDFSClientRetries.busyTest() after HDFS-3462. (harsh) (Revision 1344394)

          Result = SUCCESS
          harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1344394
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java
          Show
          hudson Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2374 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2374/ ) HDFS-3476 . Correct the default used in TestDFSClientRetries.busyTest() after HDFS-3462 . (harsh) (Revision 1344394) Result = SUCCESS harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1344394 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java
          Hide
          hudson Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2301 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2301/)
          HDFS-3476. Correct the default used in TestDFSClientRetries.busyTest() after HDFS-3462. (harsh) (Revision 1344394)

          Result = SUCCESS
          harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1344394
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java
          Show
          hudson Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2301 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2301/ ) HDFS-3476 . Correct the default used in TestDFSClientRetries.busyTest() after HDFS-3462 . (harsh) (Revision 1344394) Result = SUCCESS harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1344394 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java
          Hide
          hudson Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2320 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2320/)
          HDFS-3476. Correct the default used in TestDFSClientRetries.busyTest() after HDFS-3462. (harsh) (Revision 1344394)

          Result = FAILURE
          harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1344394
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java
          Show
          hudson Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2320 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2320/ ) HDFS-3476 . Correct the default used in TestDFSClientRetries.busyTest() after HDFS-3462 . (harsh) (Revision 1344394) Result = FAILURE harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1344394 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java
          Hide
          hudson Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1062 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1062/)
          HDFS-3476. Correct the default used in TestDFSClientRetries.busyTest() after HDFS-3462. (harsh) (Revision 1344394)

          Result = SUCCESS
          harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1344394
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java
          Show
          hudson Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1062 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1062/ ) HDFS-3476 . Correct the default used in TestDFSClientRetries.busyTest() after HDFS-3462 . (harsh) (Revision 1344394) Result = SUCCESS harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1344394 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java
          Hide
          hudson Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1096 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1096/)
          HDFS-3476. Correct the default used in TestDFSClientRetries.busyTest() after HDFS-3462. (harsh) (Revision 1344394)

          Result = SUCCESS
          harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1344394
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java
          Show
          hudson Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1096 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1096/ ) HDFS-3476 . Correct the default used in TestDFSClientRetries.busyTest() after HDFS-3462 . (harsh) (Revision 1344394) Result = SUCCESS harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1344394 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java

            People

            • Assignee:
              phatak.dev madhukara phatak
              Reporter:
              shv Konstantin Shvachko
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development