VotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.20.1
    • 0.21.0
    • None
    • None
    • Incompatible change, Reviewed
    • Hide
      Changes the Job History file format to use JSON.
      Simplifies the Job History Parsing logic
      Removes duplication of code between HistoryViewer and the JSP files
      History Files are now named as JobID_user
      Introduces a new cluster level configuration "mapreduce.cluster.jobhistory.maxage" for configuring the amount of time history files are kept before getting cleaned up
      The configuration "hadoop.job.history.user.location" is no longer supported.
      Show
      Changes the Job History file format to use JSON. Simplifies the Job History Parsing logic Removes duplication of code between HistoryViewer and the JSP files History Files are now named as JobID_user Introduces a new cluster level configuration "mapreduce.cluster.jobhistory.maxage" for configuring the amount of time history files are kept before getting cleaned up The configuration "hadoop.job.history.user.location" is no longer supported.

    Description

      Currently, parsing the job history logs with external tools is very difficult because of the format. The most critical problem is that newlines aren't escaped in the strings. That makes using tools like grep, sed, and awk very tricky.

      Attachments

        1. mapred-157-16Sep-v1.patch
          470 kB
          Jothi Padmanabhan
        2. mapred-157-16Sep.patch
          469 kB
          Jothi Padmanabhan
        3. mapred-157-15Sep-v1.patch
          469 kB
          Jothi Padmanabhan
        4. mapred-157-15Sep.patch
          468 kB
          Jothi Padmanabhan
        5. MAPREDUCE-157-avro.patch
          10 kB
          Doug Cutting
        6. mapred-157-10Sep.patch
          455 kB
          Jothi Padmanabhan
        7. mapred-157-7Sep-v1.patch
          454 kB
          Jothi Padmanabhan
        8. mapred-157-7Sep.patch
          454 kB
          Jothi Padmanabhan
        9. mapred-157-4Sep.patch
          451 kB
          Jothi Padmanabhan
        10. MAPREDUCE-157-avro.patch
          4 kB
          Doug Cutting
        11. mapred-157-prelim.patch
          101 kB
          Jothi Padmanabhan

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            jothipn Jothi Padmanabhan
            omalley Owen O'Malley
            Votes:
            1 Vote for this issue
            Watchers:
            22 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Issue deployment