Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2131

Aggregate JobCounters and TaskCounters in a background thread

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.22.0
    • Fix Version/s: None
    • Component/s: jobtracker
    • Labels:
      None

      Description

      JobTracker.getJobCounters() aggregates the counters when getting requested.
      It may consume lots of CPU if the request is too often.
      It may be good if we aggregate them in a background thread.

      Thoughts?

        Activity

        Hide
        Steve Loughran added a comment -

        I can see some use cases where eventual consistency is OK: you don't need to force an update of everything, just a "recent" value, with some data on when that value was last collected. Decoupling aggregation from querying would achieve that, though the Counters class would need to be extended with a timestamp field, and we may want the option to force aggregation; getCounters(boolean update)

        Show
        Steve Loughran added a comment - I can see some use cases where eventual consistency is OK: you don't need to force an update of everything, just a "recent" value, with some data on when that value was last collected. Decoupling aggregation from querying would achieve that, though the Counters class would need to be extended with a timestamp field, and we may want the option to force aggregation; getCounters(boolean update)

          People

          • Assignee:
            Unassigned
            Reporter:
            Scott Chen
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:

              Development