Uploaded image for project: 'Chukwa'
  1. Chukwa
  2. CHUKWA-647

Spread out intermediate data with the same ReduceType into different Reduce Tasks

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 0.4.0, 0.6.0
    • Fix Version/s: 0.6.0
    • Component/s: Data Processors
    • Labels:
      None

      Description

      We have found that if we partitioned the map output data according to ReduceType, we can see the data skew in some HiTune cases. Then one or two Reduce Tasks slow down the whole Demux job somehow, since those reduce tasks have to process more input-data.

        Attachments

          Activity

            People

            • Assignee:
              asrabkin Ari Rabkin
              Reporter:
              grace.huang Jie Huang
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: