Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2051

Contribute a fair scheduler preemption system test

    Details

    • Type: Test Test
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.22.0
    • Component/s: contrib/fair-share
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      We've seen a number of bugs in the fair share scheduler not caught by its unit tests, which are heavily mock-based. This JIRA is to add an integration/system/stress test for the fairshare scheduler. This test can help identify races and deadlocks, and when run within the jcarder framework has identified several potential deadlocks for us that aren't turned up running small scale testing.

        Activity

        Hide
        Todd Lipcon added a comment -

        Oops, forgot to hit Submit Patch. This test is nothing fancy, but when run with jcarder it at least exercises the "real" code paths so will pick up inconsistent lock orderings which often creep in in the JT with respect to the scheduler (JT, Scheduler, JIP, and TIP locks in particular)

        Show
        Todd Lipcon added a comment - Oops, forgot to hit Submit Patch. This test is nothing fancy, but when run with jcarder it at least exercises the "real" code paths so will pick up inconsistent lock orderings which often creep in in the JT with respect to the scheduler (JT, Scheduler, JIP, and TIP locks in particular)
        Hide
        Tom White added a comment -

        +1 This is a useful addition. Can you post the results of test-patch for it please?

        Show
        Tom White added a comment - +1 This is a useful addition. Can you post the results of test-patch for it please?
        Hide
        Todd Lipcon added a comment -

        [exec] +1 overall.
        [exec]
        [exec] +1 @author. The patch does not contain any @author tags.
        [exec]
        [exec] +1 tests included. The patch appears to include 2 new or modified tests.
        [exec]
        [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
        [exec]
        [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
        [exec]
        [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
        [exec]
        [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
        [exec]
        [exec] +1 system tests framework. The patch passed system tests framework compile.

        Show
        Todd Lipcon added a comment - [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 2 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] +1 system tests framework. The patch passed system tests framework compile.
        Hide
        Tom White added a comment -

        I've just committed this. Thanks Todd!

        Show
        Tom White added a comment - I've just committed this. Thanks Todd!
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #525 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/525/)
        MAPREDUCE-2051. Contribute a fair scheduler preemption system test. Contributed by Todd Lipcon.

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #525 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/525/ ) MAPREDUCE-2051 . Contribute a fair scheduler preemption system test. Contributed by Todd Lipcon.

          People

          • Assignee:
            Todd Lipcon
            Reporter:
            Todd Lipcon
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development