Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3520

Allow for metrics to be exchanged between maps and reduces

    Details

    • Type: New Feature New Feature
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: None
    • Component/s: mrv2
    • Labels:
      None

      Description

      Some people on the mailing lists have requested having read access in reducers to counters that were set by mappers. I propose that we provide a new interface for doing this. The counter data in addition to being set periodically to the Application Master would be put into the intermediate output of the mapper for each reducer and be specially marked as such. When the reducers fetch that data they can combine it together and provide a read only interface to it.

      This allows users to potentially optimize their code on the reducer side if they know some special metadata before hand.

        Activity

        Hide
        Frank Luo added a comment -

        any progress on this ticket, or there is a better way to do it now?

        Show
        Frank Luo added a comment - any progress on this ticket, or there is a better way to do it now?
        Hide
        Robert Joseph Evans added a comment -

        No better way to do it. i just have not found the time to actually make it happen.

        Show
        Robert Joseph Evans added a comment - No better way to do it. i just have not found the time to actually make it happen.

          People

          • Assignee:
            Unassigned
            Reporter:
            Robert Joseph Evans
          • Votes:
            2 Vote for this issue
            Watchers:
            12 Start watching this issue

            Dates

            • Created:
              Updated:

              Development