Details

    • Type: Test Test
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Using mock objects in unit tests can improve code quality (see e.g. http://www.mockobjects.com/). Hadoop would benefit from having a mock object framework for developers to write unit tests with. Doing so will allow a wider range of failure conditions to be tested and the tests will run faster.

      1. MAPREDUCE-1050.patch
        3 kB
        Tom White
      2. MAPREDUCE-1050.patch
        13 kB
        Tom White
      3. MAPREDUCE-1050.patch
        13 kB
        Tom White
      4. MAPREDUCE-1050.patch
        16 kB
        Konstantin Boudnik
      5. MAPREDUCE-1050.patch
        16 kB
        Tom White

        Issue Links

          Activity

          Hide
          Konstantin Boudnik added a comment -

          I'm wondering if mock objects would benefit from (or might be expressed by means of) the injection framework we currently have in HDFS (HDFS-435) and will have in big Hadoop (HADOOP-6204) soon?

          Show
          Konstantin Boudnik added a comment - I'm wondering if mock objects would benefit from (or might be expressed by means of) the injection framework we currently have in HDFS ( HDFS-435 ) and will have in big Hadoop ( HADOOP-6204 ) soon?
          Hide
          Tom White added a comment -

          I have attached a patch that illustrates a very simple application of mock objects to test the JobControl API. JobControl depends on the Job class (in org.apache.hadoop.mapreduce), but its logic is independent of Job's. To unit test JobControl's logic we shouldn't have to run a full MapReduce Job. To do so is time consuming, and introduces more dependencies than are strictly needed. (In some situations it is difficult - or at best awkward - to simulate the condition we are trying to test. E.g. https://issues.apache.org/jira/browse/MAPREDUCE-270?focusedCommentId=12759441&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12759441.)

          Instead we merely need to check that for various Job behaviours JobControl acts as expected. For example, if a Job succeeds then JobControl should put it in the successful jobs collection. (The test could obviously be expanded to cover failing jobs and dependent jobs, like TestMapReduceJobControl. Also, we still need system level tests that run full MapReduce jobs using the JobControl API - this JIRA is about unit tests.)

          The test uses Mockito (http://mockito.org/), a mocking framework which is gaining a lot of traction. The other contenders are jMock (http://www.jmock.org/) and EasyMock (http://easymock.org/, http://code.google.com/p/mockito/wiki/MockitoVSEasyMock). In my opinion, Mockito has the clearest syntax and is easiest to use.

          Obviously the patch shows a single test. I propose that the scope of this JIRA is to write a handful of different tests to show different cases (e.g. verification of method invocation, not just stubbing), so we can evaluate the framework for inclusion.

          Thoughts?

          Show
          Tom White added a comment - I have attached a patch that illustrates a very simple application of mock objects to test the JobControl API. JobControl depends on the Job class (in org.apache.hadoop.mapreduce), but its logic is independent of Job's. To unit test JobControl's logic we shouldn't have to run a full MapReduce Job. To do so is time consuming, and introduces more dependencies than are strictly needed. (In some situations it is difficult - or at best awkward - to simulate the condition we are trying to test. E.g. https://issues.apache.org/jira/browse/MAPREDUCE-270?focusedCommentId=12759441&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12759441 .) Instead we merely need to check that for various Job behaviours JobControl acts as expected. For example, if a Job succeeds then JobControl should put it in the successful jobs collection. (The test could obviously be expanded to cover failing jobs and dependent jobs, like TestMapReduceJobControl. Also, we still need system level tests that run full MapReduce jobs using the JobControl API - this JIRA is about unit tests.) The test uses Mockito ( http://mockito.org/ ), a mocking framework which is gaining a lot of traction. The other contenders are jMock ( http://www.jmock.org/ ) and EasyMock ( http://easymock.org/ , http://code.google.com/p/mockito/wiki/MockitoVSEasyMock ). In my opinion, Mockito has the clearest syntax and is easiest to use. Obviously the patch shows a single test. I propose that the scope of this JIRA is to write a handful of different tests to show different cases (e.g. verification of method invocation, not just stubbing), so we can evaluate the framework for inclusion. Thoughts?
          Hide
          Nigel Daley added a comment -

          +1. We should explore the utility of dynamic mocks, like those produced by http://easymock.org/ or http://jmock.org/ as well as what static mocks need to be built.

          Show
          Nigel Daley added a comment - +1. We should explore the utility of dynamic mocks, like those produced by http://easymock.org/ or http://jmock.org/ as well as what static mocks need to be built.
          Hide
          Nigel Daley added a comment -

          One thing to consider with dynamic mock frameworks is how easy it it so understand failures when they occur. My experience with easymock (from a few years back) was that it's use of DynamicProxy made stack traces really difficult to figure out when a test failed. Tom, any experience with this using mockito?

          Show
          Nigel Daley added a comment - One thing to consider with dynamic mock frameworks is how easy it it so understand failures when they occur. My experience with easymock (from a few years back) was that it's use of DynamicProxy made stack traces really difficult to figure out when a test failed. Tom, any experience with this using mockito?
          Hide
          Eli Collins added a comment -

          FYI filed HDFS-669 for using mock objects for HDFS testing.

          Show
          Eli Collins added a comment - FYI filed HDFS-669 for using mock objects for HDFS testing.
          Hide
          Tom White added a comment -

          Konstantin, I think the fault injection framework is different (but complementary) to mock objects. Both are needed. Fault injection allows you to simulate controlled faults in a larger system - e.g. HDFS. Mock objects, on the other hand, are really for unit testing smaller parts of the system. They allow you to mock out the dependencies of the class under test, so that you test only a single class (or a few closely collaborating classes, like JobControl and ControllingJob in my example). Using mock objects can help foster good class design, since it reduces coupling by encouraging the developer to consider each class's responsibilities and the interface between classes.

          Nigel, in my experience the stacktraces from Mockito are helpful and clear. Mockito by default is less brittle than other frameworks since it allows redundant invocations (see http://mockito.googlecode.com/svn/branches/1.8.0/javadoc/org/mockito/Mockito.html#8).

          Show
          Tom White added a comment - Konstantin, I think the fault injection framework is different (but complementary) to mock objects. Both are needed. Fault injection allows you to simulate controlled faults in a larger system - e.g. HDFS. Mock objects, on the other hand, are really for unit testing smaller parts of the system. They allow you to mock out the dependencies of the class under test, so that you test only a single class (or a few closely collaborating classes, like JobControl and ControllingJob in my example). Using mock objects can help foster good class design, since it reduces coupling by encouraging the developer to consider each class's responsibilities and the interface between classes. Nigel, in my experience the stacktraces from Mockito are helpful and clear. Mockito by default is less brittle than other frameworks since it allows redundant invocations (see http://mockito.googlecode.com/svn/branches/1.8.0/javadoc/org/mockito/Mockito.html#8 ).
          Hide
          Konstantin Boudnik added a comment -

          Thanks for the comment, Tom. I think 'complementary' is the word I was looking for

          Show
          Konstantin Boudnik added a comment - Thanks for the comment, Tom. I think 'complementary' is the word I was looking for
          Hide
          Nigel Daley added a comment -

          I agree that injection (don't read that as just 'fault injection') is complementary to mock object testing.

          Injection can be used for faults, for enhanced controllability, and for enhanced observability when writing tests. It is useful in situations where you don't have a very testable design, but need to write unit tests that can control and observe the code-under-test.

          Show
          Nigel Daley added a comment - I agree that injection (don't read that as just 'fault injection') is complementary to mock object testing. Injection can be used for faults, for enhanced controllability, and for enhanced observability when writing tests. It is useful in situations where you don't have a very testable design, but need to write unit tests that can control and observe the code-under-test.
          Hide
          Chad Metcalf added a comment -

          +1 on mockito

          Couple of nice comparisons to easymock.

          http://blog.jteam.nl/2009/08/13/easier-mocking-with-mockito/
          http://code.google.com/p/mockito/wiki/MockitoVSEasyMock

          The jmock guys acknowledge that their tool is opinionated and the structure arcane but that they made those decisions to force test in a certain direction. That is all fine and well for a company that can devote time and resources to training. I think its less appropriate on an open source project.

          I think mockito would be a good choice in this environment (lots of developers from different backgrounds, testing training, etc). In the end we want people to use the tool and test.

          Show
          Chad Metcalf added a comment - +1 on mockito Couple of nice comparisons to easymock. http://blog.jteam.nl/2009/08/13/easier-mocking-with-mockito/ http://code.google.com/p/mockito/wiki/MockitoVSEasyMock The jmock guys acknowledge that their tool is opinionated and the structure arcane but that they made those decisions to force test in a certain direction. That is all fine and well for a company that can devote time and resources to training. I think its less appropriate on an open source project. I think mockito would be a good choice in this environment (lots of developers from different backgrounds, testing training, etc). In the end we want people to use the tool and test.
          Hide
          Nigel Daley added a comment -

          +1 on mockito too. Looks good. Would love to see some real unit tests written with this

          Show
          Nigel Daley added a comment - +1 on mockito too. Looks good. Would love to see some real unit tests written with this
          Hide
          Tom White added a comment -

          Here are a few more MapReduce tests that show more mock-testing techniques, including verification, partial mocks (good for existing code, discouraged for new code), argument capture, and argument matching using Hamcrest. There are comments in the tests that explain what is going on.

          Show
          Tom White added a comment - Here are a few more MapReduce tests that show more mock-testing techniques, including verification, partial mocks (good for existing code, discouraged for new code), argument capture, and argument matching using Hamcrest. There are comments in the tests that explain what is going on.
          Hide
          Johan Oskarsson added a comment -

          +1 on mockito. I've used it a couple of times and so far it's been a pleasure to work with.

          Show
          Johan Oskarsson added a comment - +1 on mockito. I've used it a couple of times and so far it's been a pleasure to work with.
          Hide
          Jakob Homan added a comment -

          It looks like people are pretty well agreed on Mockito (I'm also for it) and there certainly hasn't been any opposition to using mocking. Is this going to go patch available - or is it intended to? - so that the mockito libraries can be added via ivy and tests can be start to be written?

          Show
          Jakob Homan added a comment - It looks like people are pretty well agreed on Mockito (I'm also for it) and there certainly hasn't been any opposition to using mocking. Is this going to go patch available - or is it intended to? - so that the mockito libraries can be added via ivy and tests can be start to be written?
          Hide
          Tom White added a comment -

          I think this could be committed after being checked by Hudson, the only thing that I'm not sure about is the names of the test classes, which end "WithMocks" to differentiate them from the existing tests. Any suggestions?

          Show
          Tom White added a comment - I think this could be committed after being checked by Hudson, the only thing that I'm not sure about is the names of the test classes, which end "WithMocks" to differentiate them from the existing tests. Any suggestions?
          Hide
          Eli Collins added a comment -

          How about breaking out another high-level directory to separate functional tests from unit tests? eg src/unittest parallel to src/test.

          Show
          Eli Collins added a comment - How about breaking out another high-level directory to separate functional tests from unit tests? eg src/unittest parallel to src/test.
          Hide
          Konstantin Boudnik added a comment -

          I believe ...WithMocks suffix is optional, isn't it? I suggest we just drop it. I believe we have briefly discuss it during Thursday meeting that we'll create separate folders for different types of tests, i.e. src/test/unit, src/test/functional, src/test/stress and so on. Then inside of these folder we don't need to have any special tags in the class names just whatever is required by JUnit, Mockito and so on.

          Show
          Konstantin Boudnik added a comment - I believe ...WithMocks suffix is optional, isn't it? I suggest we just drop it. I believe we have briefly discuss it during Thursday meeting that we'll create separate folders for different types of tests, i.e. src/test/unit , src/test/functional , src/test/stress and so on. Then inside of these folder we don't need to have any special tags in the class names just whatever is required by JUnit, Mockito and so on.
          Hide
          Tom White added a comment -

          Yes, the WithMocks suffix is optional, and shouldn't be usual practice - I only added it to avoid naming conflicts. With this patch I've renamed TestLostTrackerWithMocks to TestLostTaskTracker, but I've left TestMapReduceJobControlWithMocks alone for the moment. It can be renamed once TestMapReduceJobControl is moved to the functional test hierarchy - this should go in another JIRA.

          (Within a hierarchy the name needs to be unique - even for tests in different packages - since otherwise Eclipse has problems running all the tests in a hierarchy, I believe.)

          Show
          Tom White added a comment - Yes, the WithMocks suffix is optional, and shouldn't be usual practice - I only added it to avoid naming conflicts. With this patch I've renamed TestLostTrackerWithMocks to TestLostTaskTracker, but I've left TestMapReduceJobControlWithMocks alone for the moment. It can be renamed once TestMapReduceJobControl is moved to the functional test hierarchy - this should go in another JIRA. (Within a hierarchy the name needs to be unique - even for tests in different packages - since otherwise Eclipse has problems running all the tests in a hierarchy, I believe.)
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12421993/MAPREDUCE-1050.patch
          against trunk revision 824750.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 9 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/162/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/162/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/162/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/162/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12421993/MAPREDUCE-1050.patch against trunk revision 824750. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/162/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/162/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/162/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/162/console This message is automatically generated.
          Hide
          Tom White added a comment -

          The test failure is unrelated.

          Show
          Tom White added a comment - The test failure is unrelated.
          Hide
          Konstantin Boudnik added a comment -

          In order to move along with this new framework current patch (and JIRA) needs to be modified.
          This JIRA is about adding new framework. Thus the patch needs to be focused on build's configuration and ivy modifications.
          Then New JIRA needs to be opened for adding Mockito based unit tests.

          Basically, the patch needs to be split in two.

          Same is true for HDFS-669

          Show
          Konstantin Boudnik added a comment - In order to move along with this new framework current patch (and JIRA) needs to be modified. This JIRA is about adding new framework. Thus the patch needs to be focused on build's configuration and ivy modifications. Then New JIRA needs to be opened for adding Mockito based unit tests. Basically, the patch needs to be split in two. Same is true for HDFS-669
          Hide
          Konstantin Boudnik added a comment -

          On the other hand, having build modification + some test cases will allow to see immediately if Mockito works. So, I think with the patch as it is.

          +1 on the patch. Let's have it.

          Show
          Konstantin Boudnik added a comment - On the other hand, having build modification + some test cases will allow to see immediately if Mockito works. So, I think with the patch as it is. +1 on the patch. Let's have it.
          Hide
          Konstantin Boudnik added a comment -

          Per conversation w/ Nigel and Tom last night, I've moved test cases from src/test/mapred/ hierarchy to src/test/unit/ for these are true unit tests.

          Also, I've created new ant target 'run-test-unit' to execute unit tests only and modified the test running macro to pickup unit tests when a testcase is specified.

          Also, one of the tests 'TestLostTaskTracker' is repeatedly failing now. Tom, could you please take a look?

          Show
          Konstantin Boudnik added a comment - Per conversation w/ Nigel and Tom last night, I've moved test cases from src/test/mapred/ hierarchy to src/test/unit/ for these are true unit tests. Also, I've created new ant target 'run-test-unit' to execute unit tests only and modified the test running macro to pickup unit tests when a testcase is specified. Also, one of the tests 'TestLostTaskTracker' is repeatedly failing now. Tom, could you please take a look?
          Hide
          Konstantin Boudnik added a comment -

          I've checked the test failure on the original patch's version and see the same failure. Apparently, something has been changed in MR

          Show
          Konstantin Boudnik added a comment - I've checked the test failure on the original patch's version and see the same failure. Apparently, something has been changed in MR
          Hide
          Konstantin Boudnik added a comment -

          Another though on this: mockito jar is included into common configuration of ivy. Should it be included into
          test' config instead?

          Also, the needed modifications for .eclipse_template/.classpath is missed

          Show
          Konstantin Boudnik added a comment - Another though on this: mockito jar is included into common configuration of ivy. Should it be included into test' config instead? Also, the needed modifications for .eclipse_template/.classpath is missed
          Hide
          Tom White added a comment -

          Here's a new patch which updates the Eclipse config, and places the Mockito jar in the test configuration for Ivy.

          Also, TestLostTaskTracker is running OK for me. I'll run it through Hudson.

          Show
          Tom White added a comment - Here's a new patch which updates the Eclipse config, and places the Mockito jar in the test configuration for Ivy. Also, TestLostTaskTracker is running OK for me. I'll run it through Hudson.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12426134/MAPREDUCE-1050.patch
          against trunk revision 887044.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 11 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/162/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/162/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/162/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/162/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12426134/MAPREDUCE-1050.patch against trunk revision 887044. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 11 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/162/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/162/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/162/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/162/console This message is automatically generated.
          Hide
          Konstantin Boudnik added a comment -

          Well, still fails on my BSD machine. The message is

          TestLostTaskTracker
          Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0.791 sec
          ------------- Standard Output ---------------
          2009-12-04 16:43:13,581 INFO  mapred.JobTracker (JobTracker.java:<init>(1334)) - Starting jobtracker with owner as cos and supergroup as supergroup
          2009-12-04 16:43:13,587 INFO  mapred.JobTracker (JobTracker.java:initializeTaskMemoryRelatedConfig(4086)) - Scheduler configured with (memSizeForMapSlotOnJT, memSizeForRedu
          ceSlotOnJT, limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1, -1)
          2009-12-04 16:43:13,590 INFO  util.HostsFileReader (HostsFileReader.java:refresh(81)) - Refreshing hosts (include/exclude) list
          2009-12-04 16:43:13,607 INFO  mapred.QueueConfigurationParser (QueueConfigurationParser.java:parseResource(170)) - Bad conf file: top-level element not <queues>
          ------------- ---------------- ---------------
          
          Testcase: testLostTaskTrackerCalledAfterExpiryTime took 0.763 sec
                  Caused an ERROR
          No queues defined 
          java.lang.RuntimeException: No queues defined 
                  at org.apache.hadoop.mapred.QueueConfigurationParser.parseResource(QueueConfigurationParser.java:171)
                  at org.apache.hadoop.mapred.QueueConfigurationParser.loadResource(QueueConfigurationParser.java:163)
                  at org.apache.hadoop.mapred.QueueConfigurationParser.<init>(QueueConfigurationParser.java:92)
                  at org.apache.hadoop.mapred.QueueManager.getQueueConfigurationParser(QueueManager.java:126)
                  at org.apache.hadoop.mapred.QueueManager.<init>(QueueManager.java:146)
                  at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1376)
                  at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1325)
                  at org.apache.hadoop.mapred.TestLostTaskTracker.setUp(TestLostTaskTracker.java:58)
          

          The other problem: TestLostTaskTracker is JUnit v3 test (it extends TestCase, etc.). Please make it to be JUnit v4 (like the other two tests are).

          Show
          Konstantin Boudnik added a comment - Well, still fails on my BSD machine. The message is TestLostTaskTracker Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0.791 sec ------------- Standard Output --------------- 2009-12-04 16:43:13,581 INFO mapred.JobTracker (JobTracker.java:<init>(1334)) - Starting jobtracker with owner as cos and supergroup as supergroup 2009-12-04 16:43:13,587 INFO mapred.JobTracker (JobTracker.java:initializeTaskMemoryRelatedConfig(4086)) - Scheduler configured with (memSizeForMapSlotOnJT, memSizeForRedu ceSlotOnJT, limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1, -1) 2009-12-04 16:43:13,590 INFO util.HostsFileReader (HostsFileReader.java:refresh(81)) - Refreshing hosts (include/exclude) list 2009-12-04 16:43:13,607 INFO mapred.QueueConfigurationParser (QueueConfigurationParser.java:parseResource(170)) - Bad conf file: top-level element not <queues> ------------- ---------------- --------------- Testcase: testLostTaskTrackerCalledAfterExpiryTime took 0.763 sec Caused an ERROR No queues defined java.lang.RuntimeException: No queues defined at org.apache.hadoop.mapred.QueueConfigurationParser.parseResource(QueueConfigurationParser.java:171) at org.apache.hadoop.mapred.QueueConfigurationParser.loadResource(QueueConfigurationParser.java:163) at org.apache.hadoop.mapred.QueueConfigurationParser.<init>(QueueConfigurationParser.java:92) at org.apache.hadoop.mapred.QueueManager.getQueueConfigurationParser(QueueManager.java:126) at org.apache.hadoop.mapred.QueueManager.<init>(QueueManager.java:146) at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1376) at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1325) at org.apache.hadoop.mapred.TestLostTaskTracker.setUp(TestLostTaskTracker.java:58) The other problem: TestLostTaskTracker is JUnit v3 test (it extends TestCase, etc.). Please make it to be JUnit v4 (like the other two tests are).
          Hide
          Konstantin Boudnik added a comment -

          Interestingly enough but I'm seeing this problem only on BSD - Linux runs just fine...

          Show
          Konstantin Boudnik added a comment - Interestingly enough but I'm seeing this problem only on BSD - Linux runs just fine...
          Hide
          Konstantin Boudnik added a comment -

          Thanks for Chris' hint: I had a stale .xml file in my conf/ directory. I've deleted it and now everything works Ok.
          +1 patch looks good!

          Show
          Konstantin Boudnik added a comment - Thanks for Chris' hint: I had a stale .xml file in my conf/ directory. I've deleted it and now everything works Ok. +1 patch looks good!
          Hide
          Tom White added a comment -

          I've just committed this.

          Show
          Tom White added a comment - I've just committed this.

            People

            • Assignee:
              Tom White
              Reporter:
              Tom White
            • Votes:
              0 Vote for this issue
              Watchers:
              28 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development