Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2403

MR-279: Improve job history event handling in AM to log to HDFS

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.0
    • Component/s: mrv2
    • Labels:
      None

      Description

      Improve the job history event handling in the application master to log to HDFS in the staging directory for the job and also move it to the required location for the job history server to use.

      1. mapreduce-2403.patch
        30 kB
        Krishna Ramachandran

        Activity

        Arun C Murthy made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Mahadev konar made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        Mahadev konar added a comment -

        I just pushed this to MR-279. I tried running this on a single node cluster and am not seeing complted job history files in HDFS. Can you please take a look at it?

        Show
        Mahadev konar added a comment - I just pushed this to MR-279. I tried running this on a single node cluster and am not seeing complted job history files in HDFS. Can you please take a look at it?
        Krishna Ramachandran made changes -
        Affects Version/s 0.23.0 [ 12315570 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12474459/mapreduce-2403.patch
        against trunk revision 1082703.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 15 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/146//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12474459/mapreduce-2403.patch against trunk revision 1082703. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 15 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/146//console This message is automatically generated.
        Krishna Ramachandran made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Krishna Ramachandran added a comment -

        Attached file addresses this issue
        Additional improvements will be added incrementally(indexed directory for done location)

        Show
        Krishna Ramachandran added a comment - Attached file addresses this issue Additional improvements will be added incrementally(indexed directory for done location)
        Krishna Ramachandran made changes -
        Field Original Value New Value
        Attachment mapreduce-2403.patch [ 12474459 ]
        Hide
        Krishna Ramachandran added a comment -

        Revised patch addresses all issues that came up during previous reviews

        All unit tests passed (except for the one that failed without this patch as well)

        Single node cluster tests passed

        Show
        Krishna Ramachandran added a comment - Revised patch addresses all issues that came up during previous reviews All unit tests passed (except for the one that failed without this patch as well) Single node cluster tests passed
        Mahadev konar created issue -

          People

          • Assignee:
            Krishna Ramachandran
            Reporter:
            Mahadev konar
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development