Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.12.0
    • Component/s: None
    • Labels:
      None

      Description

      It would be nice to have map / reduce job keep aggregated counts for arbitrary events occuring in its tasks – the numer of records processed, the numer of exceptions of a specific type, the number of sentences in passive voice, whatever the jobs finds useful.

      This can be implemented by tasks periodically sending <name, value> pairs to the jobtracker (in some implementations such messages are piggy-backed on the heartbeats), so that the job tracker stores all the latests values from each task and aggregates them on a request. It should also make the aggregated values available at the job end. The value for a task would be flushed when the task fails.

      #491 and #490 may be related to this one.

      1. counters3.patch
        62 kB
        David Bowen
      2. counters2.patch
        62 kB
        David Bowen
      3. counters1.patch
        58 kB
        David Bowen

        Issue Links

          Activity

          arkady borkovsky created issue -
          Doug Cutting made changes -
          Field Original Value New Value
          Link This issue is related to HADOOP-481 [ HADOOP-481 ]
          Doug Cutting made changes -
          Assignee Owen O'Malley [ owen.omalley ]
          David Bowen made changes -
          Assignee Owen O'Malley [ owen.omalley ] David Bowen [ dbowen ]
          David Bowen made changes -
          Attachment counters.patch [ 12351751 ]
          David Bowen made changes -
          Attachment counters1.patch [ 12351801 ]
          David Bowen made changes -
          Attachment counters.patch [ 12351751 ]
          David Bowen made changes -
          Attachment counters2.patch [ 12351851 ]
          David Bowen made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          David Bowen made changes -
          Attachment counters3.patch [ 12351853 ]
          Doug Cutting made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Fix Version/s 0.12.0 [ 12312293 ]
          Doug Cutting made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Owen O'Malley made changes -
          Component/s mapred [ 12310690 ]

            People

            • Assignee:
              David Bowen
              Reporter:
              arkady borkovsky
            • Votes:
              1 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development