Details

    • Type: Sub-task Sub-task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Invalid
    • Affects Version/s: None
    • Fix Version/s: 0.14.0
    • Component/s: tez
    • Labels:
      None

      Description

      Currently if user has no parallel clause specified, then it defaults to 1 and it is bad for performance. MR does not have this issue as for each job number of mappers are determined by input splits and number for reducers by InputSizeReducerEstimator. Automatic reducer parallelism for Tez in general will be handled in separate jiras. But a quick workaround can be done for joins and unions by setting the parallelism of the reduce task to be sum of join tasks till ARP is put in and better estimation is done.

        Activity

        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Resolved Resolved
        105d 5h 10m 1 Rohini Palaniswamy 24/Jul/14 19:43
        Rohini Palaniswamy made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Invalid [ 6 ]
        Hide
        Rohini Palaniswamy added a comment -

        Not valid anymore as PIG-3846 (automatic reducer parallelism) is already done.

        Show
        Rohini Palaniswamy added a comment - Not valid anymore as PIG-3846 (automatic reducer parallelism) is already done.
        Daniel Dai made changes -
        Component/s tez [ 12321016 ]
        Daniel Dai made changes -
        Field Original Value New Value
        Fix Version/s 0.14.0 [ 12326954 ]
        Fix Version/s tez-branch [ 12324968 ]
        Rohini Palaniswamy created issue -

          People

          • Assignee:
            Rohini Palaniswamy
            Reporter:
            Rohini Palaniswamy
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development