Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-5043

Slowstart not applied in Tez with PARALLEL clause

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.17.0, 0.16.1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      We enable auto parallelism only if there is no custom partiioner, but in case of UnorderedPartitioned it always had RoundRobinPartitioner.

      VertexManager plugin configuration is not overridden if there is no auto parallelism. So slow start settings are not applied.

      1. PIG-5043-1.patch
        18 kB
        Rohini Palaniswamy
      2. PIG-5043-2.patch
        17 kB
        Rohini Palaniswamy

        Issue Links

          Activity

          Hide
          rohini Rohini Palaniswamy added a comment - - edited

          Changes done:

          • Config is set on ShuffleVertexManager even if PARALLEL is specified so that slow start applies.
          • Reduced the amount of config set on ShuffleVertexManager/PigGraceShuffleVertexManager by only setting applicable settings. Reduced payload size is good for performance.

          Also fixed two other issues

          • mapreduce.reduce.env was not being picked up and mapreduce.map.env was used
          • mapreduce.job.running.map.limit might be picked up instead of mapreduce.job.running.reduce.limit
          Show
          rohini Rohini Palaniswamy added a comment - - edited Changes done: Config is set on ShuffleVertexManager even if PARALLEL is specified so that slow start applies. Reduced the amount of config set on ShuffleVertexManager/PigGraceShuffleVertexManager by only setting applicable settings. Reduced payload size is good for performance. Also fixed two other issues mapreduce.reduce.env was not being picked up and mapreduce.map.env was used mapreduce.job.running.map.limit might be picked up instead of mapreduce.job.running.reduce.limit
          Hide
          daijy Daniel Dai added a comment -

          Do you mind adding a test case for union (HashValuePartitioner) with auto parallelism?

          Show
          daijy Daniel Dai added a comment - Do you mind adding a test case for union (HashValuePartitioner) with auto parallelism?
          Hide
          rohini Rohini Palaniswamy added a comment -

          Not able to get the test case working. Somehow auto parallelism is not being applied by Tez with unordered partitioned and it seems to be a Tez problem. I will look into that later and enable again in a separate jira. Taking that part out in this jira.

          Show
          rohini Rohini Palaniswamy added a comment - Not able to get the test case working. Somehow auto parallelism is not being applied by Tez with unordered partitioned and it seems to be a Tez problem. I will look into that later and enable again in a separate jira. Taking that part out in this jira.
          Hide
          daijy Daniel Dai added a comment -

          +1 for PIG-5043-2.patch.

          Show
          daijy Daniel Dai added a comment - +1 for PIG-5043 -2.patch.
          Hide
          rohini Rohini Palaniswamy added a comment -

          Committed to branch-0.16 and trunk. Thanks Daniel for the review.

          Show
          rohini Rohini Palaniswamy added a comment - Committed to branch-0.16 and trunk. Thanks Daniel for the review.

            People

            • Assignee:
              rohini Rohini Palaniswamy
              Reporter:
              rohini Rohini Palaniswamy
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development