Uploaded image for project: 'Chukwa'
  1. Chukwa
  2. CHUKWA-55

Collect and aggregate slot-hours usage by user, and by scheduler queues.

        Details

          Description

          The easiest model is:

          Step 1: Collect Disk usage by user
          Step 2: Collect Task Slot usage by user
          Step 3: Aggregating data by scheduler queues
          Step 4: Display data on HICC portal
          Step 5: Profit!

            Issue Links

            depends upon

            New Feature - A new feature of the product, which has yet to be developed. CHUKWA-50 Create parser for parsing job history log file and job conf file in demux

            • Blocker - Blocks development and/or testing work, production could not run
            • Resolved

            New Feature - A new feature of the product, which has yet to be developed. CHUKWA-59 Collect HDFS usage for users

            • Major - Major loss of function.
            • Resolved
            is depended upon by

            New Feature - A new feature of the product, which has yet to be developed. CHUKWA-61 Provide usage reporting widget(s)

            • Blocker - Blocks development and/or testing work, production could not run
            • Resolved

            New Feature - A new feature of the product, which has yet to be developed. CHUKWA-78 Track map reduce cluster usage through through job history and jobconf

            • Blocker - Blocks development and/or testing work, production could not run
            • Resolved

              Activity

              Ascending order - Click to sort in descending order
              Hide
              Permalink
              macyang Mac Yang added a comment -

              Updated the summary. Use the Jira to track the usage data collection only.

              Show
              macyang Mac Yang added a comment - Updated the summary. Use the Jira to track the usage data collection only.
              Hide
              Permalink
              eyang Eric Yang added a comment -

              Slot hours is calculate from sum of the task startup and end time * number of attempts, aggregated for all jobs and group by user. HDFS bytes usage is calculated from dus periodically. By doing time intersection and time grouping, we get a time series data for user usage of the slot hours, and hdfs usage.

              Show
              eyang Eric Yang added a comment - Slot hours is calculate from sum of the task startup and end time * number of attempts, aggregated for all jobs and group by user. HDFS bytes usage is calculated from dus periodically. By doing time intersection and time grouping, we get a time series data for user usage of the slot hours, and hdfs usage.
              Hide
              Permalink
              eyang Eric Yang added a comment -

              Mark as patch available.

              Show
              eyang Eric Yang added a comment - Mark as patch available.
              Hide
              Permalink
              asrabkin Ari Rabkin added a comment -

              Looks okay to me. +1

              Show
              asrabkin Ari Rabkin added a comment - Looks okay to me. +1
              Hide
              Permalink
              eyang Eric Yang added a comment -

              I just committed this, thanks Ari.

              Show
              eyang Eric Yang added a comment - I just committed this, thanks Ari.
              Hide
              Permalink
              eyang Eric Yang added a comment -

              Close as resolved.

              Show
              eyang Eric Yang added a comment - Close as resolved.

                People

                • Assignee:
                  eyang Eric Yang
                  Reporter:
                  eyang Eric Yang
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  3 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Development