Uploaded image for project: 'Chukwa'
  1. Chukwa
  2. CHUKWA-55

Collect and aggregate slot-hours usage by user, and by scheduler queues.

    Details

      Description

      The easiest model is:

      Step 1: Collect Disk usage by user
      Step 2: Collect Task Slot usage by user
      Step 3: Aggregating data by scheduler queues
      Step 4: Display data on HICC portal
      Step 5: Profit!

        Issue Links

          Activity

          Hide
          eyang Eric Yang added a comment -

          Close as resolved.

          Show
          eyang Eric Yang added a comment - Close as resolved.
          Hide
          eyang Eric Yang added a comment -

          I just committed this, thanks Ari.

          Show
          eyang Eric Yang added a comment - I just committed this, thanks Ari.
          Hide
          asrabkin Ari Rabkin added a comment -

          Looks okay to me. +1

          Show
          asrabkin Ari Rabkin added a comment - Looks okay to me. +1
          Hide
          eyang Eric Yang added a comment -

          Mark as patch available.

          Show
          eyang Eric Yang added a comment - Mark as patch available.
          Hide
          eyang Eric Yang added a comment -

          Slot hours is calculate from sum of the task startup and end time * number of attempts, aggregated for all jobs and group by user. HDFS bytes usage is calculated from dus periodically. By doing time intersection and time grouping, we get a time series data for user usage of the slot hours, and hdfs usage.

          Show
          eyang Eric Yang added a comment - Slot hours is calculate from sum of the task startup and end time * number of attempts, aggregated for all jobs and group by user. HDFS bytes usage is calculated from dus periodically. By doing time intersection and time grouping, we get a time series data for user usage of the slot hours, and hdfs usage.
          Hide
          macyang Mac Yang added a comment -

          Updated the summary. Use the Jira to track the usage data collection only.

          Show
          macyang Mac Yang added a comment - Updated the summary. Use the Jira to track the usage data collection only.

            People

            • Assignee:
              eyang Eric Yang
              Reporter:
              eyang Eric Yang
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development