Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1258

Fair scheduler event log not logging job info

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.0
    • Component/s: contrib/fair-share
    • Labels:
      None

      Description

      The MAPREDUCE-706 patch seems to have left an unfinished TODO in the Fair Scheduler - namely, in the dump() function for periodically dumping scheduler state to the event log, the part that dumps information about jobs is commented out. This makes the event log less useful than it was before.

      It should be fairly easy to update this part to use the new scheduler data structures (Schedulable etc) and print the data.

        Activity

        Hide
        Matei Zaharia added a comment -

        Here's a patch for this issue. I didn't include a unit test because it's a very simple fix. I'd appreciate if someone could review it!

        Note that the code in the patch does not print deficits, unlike the previous code, because deficits were removed as part of MAPREDUCE-706.

        Show
        Matei Zaharia added a comment - Here's a patch for this issue. I didn't include a unit test because it's a very simple fix. I'd appreciate if someone could review it! Note that the code in the patch does not print deficits, unlike the previous code, because deficits were removed as part of MAPREDUCE-706 .
        Hide
        Scott Chen added a comment -

        +1, this patch looks good to me.

        Show
        Scott Chen added a comment - +1, this patch looks good to me.
        Hide
        Matei Zaharia added a comment -

        Thanks for the review, Scott. I'll wait to see if there are any checkstyle warnings etc and commit it if there aren't.

        Show
        Matei Zaharia added a comment - Thanks for the review, Scott. I'll wait to see if there are any checkstyle warnings etc and commit it if there aren't.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12428407/mapreduce-1258-1.patch
        against trunk revision 892411.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/225/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/225/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/225/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/225/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12428407/mapreduce-1258-1.patch against trunk revision 892411. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/225/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/225/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/225/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/225/console This message is automatically generated.
        Hide
        Matei Zaharia added a comment -

        The test failure is unrelated, and I didn't included unit tests because this is a small logging fix, so I've committed this patch. I'm also going to check whether it's possible to commit it to 0.21.

        Show
        Matei Zaharia added a comment - The test failure is unrelated, and I didn't included unit tests because this is a small logging fix, so I've committed this patch. I'm also going to check whether it's possible to commit it to 0.21.
        Hide
        dhruba borthakur added a comment -

        I vote that we pull this into 0.21 as well.

        Show
        dhruba borthakur added a comment - I vote that we pull this into 0.21 as well.
        Hide
        Matei Zaharia added a comment -

        Dhruba let me know that it's fine to commit this to 0.21 too, because it's a bug fix. I'll wait a day to see if there are any objections, and otherwise I'll commit it.

        Show
        Matei Zaharia added a comment - Dhruba let me know that it's fine to commit this to 0.21 too, because it's a bug fix. I'll wait a day to see if there are any objections, and otherwise I'll commit it.
        Hide
        Matei Zaharia added a comment -

        I've committed this to 0.21 as well, to let this logging feature continue to work as in 0.20.

        Show
        Matei Zaharia added a comment - I've committed this to 0.21 as well, to let this logging feature continue to work as in 0.20.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #196 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/196/)

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #196 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/196/ )

          People

          • Assignee:
            Matei Zaharia
            Reporter:
            Matei Zaharia
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development