Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.1
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Hide
      Changes the Job History file format to use JSON.
      Simplifies the Job History Parsing logic
      Removes duplication of code between HistoryViewer and the JSP files
      History Files are now named as JobID_user
      Introduces a new cluster level configuration "mapreduce.cluster.jobhistory.maxage" for configuring the amount of time history files are kept before getting cleaned up
      The configuration "hadoop.job.history.user.location" is no longer supported.
      Show
      Changes the Job History file format to use JSON. Simplifies the Job History Parsing logic Removes duplication of code between HistoryViewer and the JSP files History Files are now named as JobID_user Introduces a new cluster level configuration "mapreduce.cluster.jobhistory.maxage" for configuring the amount of time history files are kept before getting cleaned up The configuration "hadoop.job.history.user.location" is no longer supported.

      Description

      Currently, parsing the job history logs with external tools is very difficult because of the format. The most critical problem is that newlines aren't escaped in the strings. That makes using tools like grep, sed, and awk very tricky.

        Attachments

        1. MAPREDUCE-157-avro.patch
          4 kB
          Doug Cutting
        2. MAPREDUCE-157-avro.patch
          10 kB
          Doug Cutting
        3. mapred-157-prelim.patch
          101 kB
          Jothi Padmanabhan
        4. mapred-157-7Sep-v1.patch
          454 kB
          Jothi Padmanabhan
        5. mapred-157-7Sep.patch
          454 kB
          Jothi Padmanabhan
        6. mapred-157-4Sep.patch
          451 kB
          Jothi Padmanabhan
        7. mapred-157-16Sep-v1.patch
          470 kB
          Jothi Padmanabhan
        8. mapred-157-16Sep.patch
          469 kB
          Jothi Padmanabhan
        9. mapred-157-15Sep-v1.patch
          469 kB
          Jothi Padmanabhan
        10. mapred-157-15Sep.patch
          468 kB
          Jothi Padmanabhan
        11. mapred-157-10Sep.patch
          455 kB
          Jothi Padmanabhan

          Issue Links

            Activity

              People

              • Assignee:
                jothipn Jothi Padmanabhan
                Reporter:
                owen.omalley Owen O'Malley
              • Votes:
                1 Vote for this issue
                Watchers:
                22 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: