Details

    • Sub-task
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.20.1
    • 0.21.0
    • None
    • None
    • Incompatible change, Reviewed
    • Hide
      Changes the Job History file format to use JSON.
      Simplifies the Job History Parsing logic
      Removes duplication of code between HistoryViewer and the JSP files
      History Files are now named as JobID_user
      Introduces a new cluster level configuration "mapreduce.cluster.jobhistory.maxage" for configuring the amount of time history files are kept before getting cleaned up
      The configuration "hadoop.job.history.user.location" is no longer supported.
      Show
      Changes the Job History file format to use JSON. Simplifies the Job History Parsing logic Removes duplication of code between HistoryViewer and the JSP files History Files are now named as JobID_user Introduces a new cluster level configuration "mapreduce.cluster.jobhistory.maxage" for configuring the amount of time history files are kept before getting cleaned up The configuration "hadoop.job.history.user.location" is no longer supported.

    Description

      Currently, parsing the job history logs with external tools is very difficult because of the format. The most critical problem is that newlines aren't escaped in the strings. That makes using tools like grep, sed, and awk very tricky.

      Attachments

        1. mapred-157-16Sep-v1.patch
          470 kB
          Jothi Padmanabhan
        2. mapred-157-16Sep.patch
          469 kB
          Jothi Padmanabhan
        3. mapred-157-15Sep-v1.patch
          469 kB
          Jothi Padmanabhan
        4. mapred-157-15Sep.patch
          468 kB
          Jothi Padmanabhan
        5. MAPREDUCE-157-avro.patch
          10 kB
          Doug Cutting
        6. mapred-157-10Sep.patch
          455 kB
          Jothi Padmanabhan
        7. mapred-157-7Sep-v1.patch
          454 kB
          Jothi Padmanabhan
        8. mapred-157-7Sep.patch
          454 kB
          Jothi Padmanabhan
        9. mapred-157-4Sep.patch
          451 kB
          Jothi Padmanabhan
        10. MAPREDUCE-157-avro.patch
          4 kB
          Doug Cutting
        11. mapred-157-prelim.patch
          101 kB
          Jothi Padmanabhan

        Issue Links

          Activity

            People

              jothipn Jothi Padmanabhan
              omalley Owen O'Malley
              Votes:
              1 Vote for this issue
              Watchers:
              22 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: