Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.1
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Hide
      Changes the Job History file format to use JSON.
      Simplifies the Job History Parsing logic
      Removes duplication of code between HistoryViewer and the JSP files
      History Files are now named as JobID_user
      Introduces a new cluster level configuration "mapreduce.cluster.jobhistory.maxage" for configuring the amount of time history files are kept before getting cleaned up
      The configuration "hadoop.job.history.user.location" is no longer supported.
      Show
      Changes the Job History file format to use JSON. Simplifies the Job History Parsing logic Removes duplication of code between HistoryViewer and the JSP files History Files are now named as JobID_user Introduces a new cluster level configuration "mapreduce.cluster.jobhistory.maxage" for configuring the amount of time history files are kept before getting cleaned up The configuration "hadoop.job.history.user.location" is no longer supported.

      Description

      Currently, parsing the job history logs with external tools is very difficult because of the format. The most critical problem is that newlines aren't escaped in the strings. That makes using tools like grep, sed, and awk very tricky.

      1. MAPREDUCE-157-avro.patch
        4 kB
        Doug Cutting
      2. MAPREDUCE-157-avro.patch
        10 kB
        Doug Cutting
      3. mapred-157-prelim.patch
        101 kB
        Jothi Padmanabhan
      4. mapred-157-7Sep-v1.patch
        454 kB
        Jothi Padmanabhan
      5. mapred-157-7Sep.patch
        454 kB
        Jothi Padmanabhan
      6. mapred-157-4Sep.patch
        451 kB
        Jothi Padmanabhan
      7. mapred-157-16Sep-v1.patch
        470 kB
        Jothi Padmanabhan
      8. mapred-157-16Sep.patch
        469 kB
        Jothi Padmanabhan
      9. mapred-157-15Sep-v1.patch
        469 kB
        Jothi Padmanabhan
      10. mapred-157-15Sep.patch
        468 kB
        Jothi Padmanabhan
      11. mapred-157-10Sep.patch
        455 kB
        Jothi Padmanabhan

        Issue Links

          Activity

          Jeff Hammerbacher made changes -
          Link This issue relates to MAPREDUCE-2251 [ MAPREDUCE-2251 ]
          Tom White made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Jothi Padmanabhan made changes -
          Release Note Changes the Job History file format to use JSON.
          Simplifies the Job History Parsing logic
          Removes duplication of code between HistoryViewer and the JSP files
          History Files are now named as JobID_user
          Introduces a new cluster level configuration "mapreduce.cluster.jobhistory.maxage" for configuring the amount of time history files are kept before getting cleaned up
          Changes the Job History file format to use JSON.
          Simplifies the Job History Parsing logic
          Removes duplication of code between HistoryViewer and the JSP files
          History Files are now named as JobID_user
          Introduces a new cluster level configuration "mapreduce.cluster.jobhistory.maxage" for configuring the amount of time history files are kept before getting cleaned up
          The configuration "hadoop.job.history.user.location" is no longer supported.
          Jothi Padmanabhan made changes -
          Release Note Changes the Job History file format to use JSON.
          Simplifies the Job History Parsing logic
          Removes duplication of code between HistoryViewer and the JSP files
          History Files are now named as JobID_user
          Introduces a new cluster level configuration "mapreduce.cluster.jobhistory.maxage" for configuring the amount of time history files are kept before getting cleaned up
          Sharad Agarwal made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Incompatible change] [Incompatible change, Reviewed]
          Resolution Fixed [ 1 ]
          Jothi Padmanabhan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Jothi Padmanabhan made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Jothi Padmanabhan made changes -
          Attachment mapred-157-16Sep-v1.patch [ 12419790 ]
          Jothi Padmanabhan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Jothi Padmanabhan made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Jothi Padmanabhan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Jothi Padmanabhan made changes -
          Attachment mapred-157-16Sep.patch [ 12419758 ]
          Jothi Padmanabhan made changes -
          Attachment mapred-157-16Sep.patch [ 12419754 ]
          Jothi Padmanabhan made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Jothi Padmanabhan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Jothi Padmanabhan made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Jothi Padmanabhan made changes -
          Attachment mapred-157-16Sep.patch [ 12419754 ]
          Jothi Padmanabhan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Jothi Padmanabhan made changes -
          Attachment mapred-157-15Sep-v1.patch [ 12419622 ]
          Jothi Padmanabhan made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Sharad Agarwal made changes -
          Hadoop Flags [Incompatible change]
          Jothi Padmanabhan made changes -
          Link This issue blocks MAPREDUCE-980 [ MAPREDUCE-980 ]
          Jothi Padmanabhan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Jothi Padmanabhan made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Jothi Padmanabhan made changes -
          Attachment mapred-157-14Sep-v1.patch [ 12419601 ]
          Jothi Padmanabhan made changes -
          Attachment mapred-157-15Sep.patch [ 12419605 ]
          Jothi Padmanabhan made changes -
          Attachment mapred-157-14Sep-v1.patch [ 12419601 ]
          Doug Cutting made changes -
          Attachment MAPREDUCE-157-avro.patch [ 12419584 ]
          Sharad Agarwal made changes -
          Link This issue blocks MAPREDUCE-975 [ MAPREDUCE-975 ]
          Amar Kamat made changes -
          Link This issue incorporates MAPREDUCE-926 [ MAPREDUCE-926 ]
          Amar Kamat made changes -
          Link This issue incorporates MAPREDUCE-881 [ MAPREDUCE-881 ]
          Jothi Padmanabhan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Jothi Padmanabhan made changes -
          Attachment mapred-157-10Sep.patch [ 12419138 ]
          Jothi Padmanabhan made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Jothi Padmanabhan made changes -
          Link This issue blocks MAPREDUCE-277 [ MAPREDUCE-277 ]
          Jothi Padmanabhan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Jothi Padmanabhan made changes -
          Attachment mapred-157-7Sep-v1.patch [ 12418824 ]
          Jothi Padmanabhan made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Jothi Padmanabhan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Jothi Padmanabhan made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Jothi Padmanabhan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Jothi Padmanabhan made changes -
          Attachment mapred-157-7Sep.patch [ 12418781 ]
          Jothi Padmanabhan made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Jothi Padmanabhan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Affects Version/s 0.20.1 [ 12314047 ]
          Fix Version/s 0.21.0 [ 12314045 ]
          Jothi Padmanabhan made changes -
          Attachment mapred-157-4Sep.patch [ 12418622 ]
          Sharad Agarwal made changes -
          Link This issue blocks MAPREDUCE-864 [ MAPREDUCE-864 ]
          Doug Cutting made changes -
          Attachment MAPREDUCE-157-avro.patch [ 12417318 ]
          Jothi Padmanabhan made changes -
          Attachment mapred-157-prelim.patch [ 12417133 ]
          Devaraj Das made changes -
          Parent MAPREDUCE-863 [ 12432997 ]
          Issue Type Bug [ 1 ] Sub-task [ 7 ]
          Jothi Padmanabhan made changes -
          Assignee Amar Kamat [ amar_kamat ] Jothi Padmanabhan [ jothipn ]
          Owen O'Malley made changes -
          Project Hadoop Common [ 12310240 ] Hadoop Map/Reduce [ 12310941 ]
          Key HADOOP-5834 MAPREDUCE-157
          Component/s mapred [ 12310690 ]
          Owen O'Malley made changes -
          Assignee Amar Kamat [ amar_kamat ]
          Amar Kamat made changes -
          Field Original Value New Value
          Link This issue blocks HADOOP-5846 [ HADOOP-5846 ]
          Owen O'Malley created issue -

            People

            • Assignee:
              Jothi Padmanabhan
              Reporter:
              Owen O'Malley
            • Votes:
              1 Vote for this issue
              Watchers:
              22 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development