Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Invalid
    • None
    • 0.14.0
    • tez
    • None

    Description

      Currently if user has no parallel clause specified, then it defaults to 1 and it is bad for performance. MR does not have this issue as for each job number of mappers are determined by input splits and number for reducers by InputSizeReducerEstimator. Automatic reducer parallelism for Tez in general will be handled in separate jiras. But a quick workaround can be done for joins and unions by setting the parallelism of the reduce task to be sum of join tasks till ARP is put in and better estimation is done.

      Attachments

        Activity

          People

            rohini Rohini Palaniswamy
            rohini Rohini Palaniswamy
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: