Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-863 Improve/standardize job history file content format and the management of the history files
  3. MAPREDUCE-157

Job History log file format is not friendly for external tools.

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.20.1
    • 0.21.0
    • None
    • None
    • Incompatible change, Reviewed
    • Hide
      Changes the Job History file format to use JSON.
      Simplifies the Job History Parsing logic
      Removes duplication of code between HistoryViewer and the JSP files
      History Files are now named as JobID_user
      Introduces a new cluster level configuration "mapreduce.cluster.jobhistory.maxage" for configuring the amount of time history files are kept before getting cleaned up
      The configuration "hadoop.job.history.user.location" is no longer supported.
      Show
      Changes the Job History file format to use JSON. Simplifies the Job History Parsing logic Removes duplication of code between HistoryViewer and the JSP files History Files are now named as JobID_user Introduces a new cluster level configuration "mapreduce.cluster.jobhistory.maxage" for configuring the amount of time history files are kept before getting cleaned up The configuration "hadoop.job.history.user.location" is no longer supported.

    Description

      Currently, parsing the job history logs with external tools is very difficult because of the format. The most critical problem is that newlines aren't escaped in the strings. That makes using tools like grep, sed, and awk very tricky.

      Attachments

        1. MAPREDUCE-157-avro.patch
          4 kB
          Doug Cutting
        2. MAPREDUCE-157-avro.patch
          10 kB
          Doug Cutting
        3. mapred-157-prelim.patch
          101 kB
          Jothi Padmanabhan
        4. mapred-157-7Sep-v1.patch
          454 kB
          Jothi Padmanabhan
        5. mapred-157-7Sep.patch
          454 kB
          Jothi Padmanabhan
        6. mapred-157-4Sep.patch
          451 kB
          Jothi Padmanabhan
        7. mapred-157-16Sep-v1.patch
          470 kB
          Jothi Padmanabhan
        8. mapred-157-16Sep.patch
          469 kB
          Jothi Padmanabhan
        9. mapred-157-15Sep-v1.patch
          469 kB
          Jothi Padmanabhan
        10. mapred-157-15Sep.patch
          468 kB
          Jothi Padmanabhan
        11. mapred-157-10Sep.patch
          455 kB
          Jothi Padmanabhan

        Issue Links

          blocks

          Sub-task - The sub-task of the issue MAPREDUCE-864 Enhance JobClient API implementations to look at history files to get information about jobs that are not in memory

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue MAPREDUCE-975 Add an API in job client to get the history file url for a given job id

          • Major - Major loss of function.
          • Closed

          New Feature - A new feature of the product, which has yet to be developed. MAPREDUCE-198 Log job history events to a common dump file

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. MAPREDUCE-980 Modify JobHistory to use Avro for serialization instead of raw JSON

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-277 Job history counters should be avaible on the UI.

          • Blocker - Blocks development and/or testing work, production could not run
          • Closed
          incorporates

          Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-881 Jobtracker continues even if History initialization fails

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-926 History viewer on web UI should filter by job-id also

          • Major - Major loss of function.
          • Resolved
          relates to

          Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-2251 Remove mapreduce.job.userhistorylocation config

          • Major - Major loss of function.
          • Closed

          Activity

            People

              jothipn Jothi Padmanabhan
              omalley Owen O'Malley
              Votes:
              1 Vote for this issue
              Watchers:
              22 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: