Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2180

Add coverage of fair scheduler servlet to system test

    Details

    • Type: Test Test
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.22.0
    • Component/s: contrib/fair-share
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      MAPREDUCE-2051 added a system test for the fair scheduler which starts a minicluster and runs a couple jobs with preemption on. I recently found a deadlock in a previous version of the scheduler that was due to lock inversion between the scheduler servlet and some JT internals. I'd like to modify the existing system test to also hit the /scheduler servlet, allowing jcarder to detect such lock inversions in the future.

        Activity

        Hide
        Todd Lipcon added a comment -

        Attached patch exercises the servlet at /scheduler while running the jobs. Under jcarder, this shows the lock inversion in the previous release and shows that it is gone in trunk.

        Show
        Todd Lipcon added a comment - Attached patch exercises the servlet at /scheduler while running the jobs. Under jcarder, this shows the lock inversion in the previous release and shows that it is gone in trunk.
        Hide
        Todd Lipcon added a comment -

        [exec] -1 overall.
        [exec]
        [exec] +1 @author. The patch does not contain any @author tags.
        [exec]
        [exec] +1 tests included. The patch appears to include 3 new or modified tests.
        [exec]
        [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
        [exec]
        [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
        [exec]
        [exec] -1 findbugs. The patch appears to introduce 13 new Findbugs warnings.
        [exec]
        [exec] -1 release audit. The applied patch generated 2 release audit warnings (more than the trunk's current 1 warnings).
        [exec]
        [exec] -1 system test framework. The patch failed system test framework compile.

        test-patch on MR produces bad results - findbugs errors are all in non-fairsched code. Release-audit is also unrelated (this patch does not add any files). system test framework errors are due to a change that went into common tonight.

        Show
        Todd Lipcon added a comment - [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 3 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] -1 findbugs. The patch appears to introduce 13 new Findbugs warnings. [exec] [exec] -1 release audit. The applied patch generated 2 release audit warnings (more than the trunk's current 1 warnings). [exec] [exec] -1 system test framework. The patch failed system test framework compile. test-patch on MR produces bad results - findbugs errors are all in non-fairsched code. Release-audit is also unrelated (this patch does not add any files). system test framework errors are due to a change that went into common tonight.
        Hide
        Konstantin Boudnik added a comment -

        +1 patch looks good. Please post any coverage numbers (before and after) if you have them.

        Show
        Konstantin Boudnik added a comment - +1 patch looks good. Please post any coverage numbers (before and after) if you have them.
        Hide
        Todd Lipcon added a comment -

        Should also mention that without this patch, the servlet is not tested at all. Though it's not exactly a great validation of the servlet behavior, at least this test serves to smoke test that it isn't throwing exceptions, etc.

        Show
        Todd Lipcon added a comment - Should also mention that without this patch, the servlet is not tested at all. Though it's not exactly a great validation of the servlet behavior, at least this test serves to smoke test that it isn't throwing exceptions, etc.
        Hide
        Todd Lipcon added a comment -

        Committed to trunk and branch (since it's a test). Thanks for review, Cos.

        Show
        Todd Lipcon added a comment - Committed to trunk and branch (since it's a test). Thanks for review, Cos.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #579 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/579/)
        MAPREDUCE-2180. Add coverage of fair scheduler servlet to system test. Contributed by Todd Lipcon

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #579 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/579/ ) MAPREDUCE-2180 . Add coverage of fair scheduler servlet to system test. Contributed by Todd Lipcon
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-22-branch #33 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-22-branch/33/)

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-22-branch #33 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-22-branch/33/ )
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #643 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/643/)

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #643 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/643/ )

          People

          • Assignee:
            Todd Lipcon
            Reporter:
            Todd Lipcon
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development