Uploaded image for project: 'Chukwa'
  1. Chukwa
  2. CHUKWA-78

Track map reduce cluster usage through through job history and jobconf

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Labels:
      None
    • Environment:

      Redhat 5.1, Java 6

      Description

      Job history data and jobconf contains valuable information to determine the usage of the map reduce job cluster. The request feature is to create the following components:

      • ChukwaJobTrackerInstrumentation API to stream job history and jobconf file to Chukwa.
      • parsers to collect job data.
      • MDL dictionary to load key/value pair for Job data to database.
      • Usage report of the cluster base on user, and job queues.

        Issue Links

          Activity

          Hide
          eyang Eric Yang added a comment -

          Historical data should be tracked by parsing job history and job conf files.

          Show
          eyang Eric Yang added a comment - Historical data should be tracked by parsing job history and job conf files.
          Hide
          eyang Eric Yang added a comment -

          The usage report is calculated by aggregated slot hours used.

          Show
          eyang Eric Yang added a comment - The usage report is calculated by aggregated slot hours used.
          Hide
          eyang Eric Yang added a comment -

          After job conf and job tracker data is loaded to the database, the database partition should down sample and roll up the jobs/tasks data to bigger partitions.

          Show
          eyang Eric Yang added a comment - After job conf and job tracker data is loaded to the database, the database partition should down sample and roll up the jobs/tasks data to bigger partitions.
          Hide
          eyang Eric Yang added a comment -

          Down sample script for down sample mr job data and mr task data.

          Show
          eyang Eric Yang added a comment - Down sample script for down sample mr job data and mr task data.
          Hide
          terencekwan Terence Kwan added a comment -

          +1

          Show
          terencekwan Terence Kwan added a comment - +1
          Hide
          eyang Eric Yang added a comment -

          I just committed this, thanks Terence.

          Show
          eyang Eric Yang added a comment - I just committed this, thanks Terence.

            People

            • Assignee:
              eyang Eric Yang
              Reporter:
              eyang Eric Yang
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development