Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.12.0
    • Component/s: None
    • Labels:
      None

      Description

      It would be nice to have map / reduce job keep aggregated counts for arbitrary events occuring in its tasks – the numer of records processed, the numer of exceptions of a specific type, the number of sentences in passive voice, whatever the jobs finds useful.

      This can be implemented by tasks periodically sending <name, value> pairs to the jobtracker (in some implementations such messages are piggy-backed on the heartbeats), so that the job tracker stores all the latests values from each task and aggregates them on a request. It should also make the aggregated values available at the job end. The value for a task would be flushed when the task fails.

      #491 and #490 may be related to this one.

      1. counters1.patch
        58 kB
        David Bowen
      2. counters2.patch
        62 kB
        David Bowen
      3. counters3.patch
        62 kB
        David Bowen

        Issue Links

          Activity

          Owen O'Malley made changes -
          Component/s mapred [ 12310690 ]
          Doug Cutting made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Doug Cutting made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Fix Version/s 0.12.0 [ 12312293 ]
          David Bowen made changes -
          Attachment counters3.patch [ 12351853 ]
          David Bowen made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          David Bowen made changes -
          Attachment counters2.patch [ 12351851 ]
          David Bowen made changes -
          Attachment counters.patch [ 12351751 ]
          David Bowen made changes -
          Attachment counters1.patch [ 12351801 ]
          David Bowen made changes -
          Attachment counters.patch [ 12351751 ]
          David Bowen made changes -
          Assignee Owen O'Malley [ owen.omalley ] David Bowen [ dbowen ]
          Doug Cutting made changes -
          Assignee Owen O'Malley [ owen.omalley ]
          Doug Cutting made changes -
          Field Original Value New Value
          Link This issue is related to HADOOP-481 [ HADOOP-481 ]
          arkady borkovsky created issue -

            People

            • Assignee:
              David Bowen
              Reporter:
              arkady borkovsky
            • Votes:
              1 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development