Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1245

TestFairScheduler fails with "too many open files" error

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.21.0
    • Component/s: test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      This was caused by MAPREDUCE-1103 and was observed after MAPREDUCE-1239.

      1. 1245_v1.patch
        2 kB
        Sharad Agarwal

        Issue Links

          Activity

          Hide
          Vinod Kumar Vavilapalli added a comment -

          Here's the problem as mentioned here:

          [junit] 09/11/25 16:34:08 INFO mapred.JobTracker: JobTracker up at: 32801
              [junit] 09/11/25 16:34:08 INFO mapred.JobTracker: JobTracker webserver: 44348
              [junit] 09/11/25 16:34:08 INFO mapred.JobTracker: Cleaning up the system directory
              [junit] 09/11/25 16:34:08 INFO mapred.JobTracker: problem cleaning system directory: file:/tmp/hadoop-sreerama/mapred/system
              [junit] java.io.IOException: Cannot run program "chmod": java.io.IOException: error=24, Too many open files
              [junit] 	at java.lang.ProcessBuilder.start(ProcessBuilder.java:474)
              [junit] 	at org.apache.hadoop.util.Shell.runCommand(Shell.java:188)
              [junit] 	at org.apache.hadoop.util.Shell.run(Shell.java:170)
              [junit] 	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:363)
              [junit] 	at org.apache.hadoop.util.Shell.execCommand(Shell.java:449)
              [junit] 	at org.apache.hadoop.util.Shell.execCommand(Shell.java:432)
              [junit] 	at org.apache.hadoop.fs.RawLocalFileSystem.execCommand(RawLocalFileSystem.java:540)
              [junit] 	at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:532)
              [junit] 	at org.apache.hadoop.fs.FilterFileSystem.setPermission(FilterFileSystem.java:281)
              [junit] 	at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:295)
              [junit] 	at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1477)
              [junit] 	at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1306)
              [junit] 	at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1299)
              [junit] 	at org.apache.hadoop.mapred.UtilsForTests.getJobTracker(UtilsForTests.java:712)
              [junit] 	at org.apache.hadoop.mapred.TestFairScheduler.testPoolAssignment(TestFairScheduler.java:2546)
              [junit] 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              [junit] 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
              [junit] 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              [junit] 	at java.lang.reflect.Method.invoke(Method.java:616)
              [junit] 	at junit.framework.TestCase.runTest(TestCase.java:168)
              [junit] 	at junit.framework.TestCase.runBare(TestCase.java:134)
              [junit] 	at junit.framework.TestResult$1.protect(TestResult.java:110)
              [junit] 	at junit.framework.TestResult.runProtected(TestResult.java:128)
              [junit] 	at junit.framework.TestResult.run(TestResult.java:113)
              [junit] 	at junit.framework.TestCase.run(TestCase.java:124)
              [junit] 	at junit.framework.TestSuite.runTest(TestSuite.java:232)
              [junit] 	at junit.framework.TestSuite.run(TestSuite.java:227)
              [junit] 	at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:79)
              [junit] 	at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39)
              [junit] 	at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:420)
              [junit] 	at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:911)
              [junit] 	at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:768)
              [junit] Caused by: java.io.IOException: java.io.IOException: error=24, Too many open files
              [junit] 	at java.lang.UNIXProcess.<init>(UNIXProcess.java:164)
              [junit] 	at java.lang.ProcessImpl.start(ProcessImpl.java:81)
              [junit] 	at java.lang.ProcessBuilder.start(ProcessBuilder.java:467)
              [junit] 	... 31 more
          

          This happens because every submitJob() creates a new JobTracker object. The same problem plagues other tests too and drastically slows them down. Creating a single object and use it repetitively should suffice.

          Show
          Vinod Kumar Vavilapalli added a comment - Here's the problem as mentioned here : [junit] 09/11/25 16:34:08 INFO mapred.JobTracker: JobTracker up at: 32801 [junit] 09/11/25 16:34:08 INFO mapred.JobTracker: JobTracker webserver: 44348 [junit] 09/11/25 16:34:08 INFO mapred.JobTracker: Cleaning up the system directory [junit] 09/11/25 16:34:08 INFO mapred.JobTracker: problem cleaning system directory: file:/tmp/hadoop-sreerama/mapred/system [junit] java.io.IOException: Cannot run program "chmod" : java.io.IOException: error=24, Too many open files [junit] at java.lang.ProcessBuilder.start(ProcessBuilder.java:474) [junit] at org.apache.hadoop.util.Shell.runCommand(Shell.java:188) [junit] at org.apache.hadoop.util.Shell.run(Shell.java:170) [junit] at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:363) [junit] at org.apache.hadoop.util.Shell.execCommand(Shell.java:449) [junit] at org.apache.hadoop.util.Shell.execCommand(Shell.java:432) [junit] at org.apache.hadoop.fs.RawLocalFileSystem.execCommand(RawLocalFileSystem.java:540) [junit] at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:532) [junit] at org.apache.hadoop.fs.FilterFileSystem.setPermission(FilterFileSystem.java:281) [junit] at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:295) [junit] at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1477) [junit] at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1306) [junit] at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1299) [junit] at org.apache.hadoop.mapred.UtilsForTests.getJobTracker(UtilsForTests.java:712) [junit] at org.apache.hadoop.mapred.TestFairScheduler.testPoolAssignment(TestFairScheduler.java:2546) [junit] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [junit] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [junit] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [junit] at java.lang.reflect.Method.invoke(Method.java:616) [junit] at junit.framework.TestCase.runTest(TestCase.java:168) [junit] at junit.framework.TestCase.runBare(TestCase.java:134) [junit] at junit.framework.TestResult$1.protect(TestResult.java:110) [junit] at junit.framework.TestResult.runProtected(TestResult.java:128) [junit] at junit.framework.TestResult.run(TestResult.java:113) [junit] at junit.framework.TestCase.run(TestCase.java:124) [junit] at junit.framework.TestSuite.runTest(TestSuite.java:232) [junit] at junit.framework.TestSuite.run(TestSuite.java:227) [junit] at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:79) [junit] at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39) [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:420) [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:911) [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:768) [junit] Caused by: java.io.IOException: java.io.IOException: error=24, Too many open files [junit] at java.lang.UNIXProcess.<init>(UNIXProcess.java:164) [junit] at java.lang.ProcessImpl.start(ProcessImpl.java:81) [junit] at java.lang.ProcessBuilder.start(ProcessBuilder.java:467) [junit] ... 31 more This happens because every submitJob() creates a new JobTracker object. The same problem plagues other tests too and drastically slows them down. Creating a single object and use it repetitively should suffice.
          Hide
          Sharad Agarwal added a comment -

          Use a lightweight JT for tests.

          Show
          Sharad Agarwal added a comment - Use a lightweight JT for tests.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12426216/1245_v1.patch
          against trunk revision 884531.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/271/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/271/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/271/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/271/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12426216/1245_v1.patch against trunk revision 884531. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/271/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/271/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/271/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/271/console This message is automatically generated.
          Hide
          Sharad Agarwal added a comment -

          Retrying hudson

          Show
          Sharad Agarwal added a comment - Retrying hudson
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12426216/1245_v1.patch
          against trunk revision 884628.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/273/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/273/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/273/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/273/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12426216/1245_v1.patch against trunk revision 884628. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/273/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/273/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/273/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/273/console This message is automatically generated.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Patch looks good. +1.

          Show
          Vinod Kumar Vavilapalli added a comment - Patch looks good. +1.
          Hide
          Sharad Agarwal added a comment -

          I just committed this. Thanks Vinod for catching this.

          Show
          Sharad Agarwal added a comment - I just committed this. Thanks Vinod for catching this.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #140 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/140/)
          . Fix TestFairScheduler failures by instantiating lightweight Jobtracker. Contributed by Sharad Agarwal.

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #140 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/140/ ) . Fix TestFairScheduler failures by instantiating lightweight Jobtracker. Contributed by Sharad Agarwal.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #162 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/162/)
          . Fix TestFairScheduler failures by instantiating lightweight Jobtracker. Contributed by Sharad Agarwal.

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #162 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/162/ ) . Fix TestFairScheduler failures by instantiating lightweight Jobtracker. Contributed by Sharad Agarwal.

            People

            • Assignee:
              Sharad Agarwal
              Reporter:
              Vinod Kumar Vavilapalli
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development