Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-3840 Umbrella jira for Pig on Tez Unit Test porting
  3. PIG-3855

Turn on UnionOptimizer by default and add new e2e tests for union

        Details

        • Type: Sub-task
        • Status: Closed
        • Priority: Major
        • Resolution: Fixed
        • Affects Version/s: None
        • Fix Version/s: tez-branch
        • Component/s: None
        • Labels:
          None
        • Hadoop Flags:
          Reviewed

          Description

          We don't have e2e tests for cases like union followed by group by, join (replicate, skewed, hash), orderby, limit, etc. PIG-3835 adds optimization to those cases and we should have e2e tests for that.

          1. Text File
            PIG-3855-1.patch
            89 kB
            Rohini Palaniswamy
          2. Text File
            PIG-3855-3.patch
            95 kB
            Rohini Palaniswamy

            Issue Links

            depends upon

            Bug - A problem which impairs or prevents the functions of the product. TEZ-1003 Need a input that merges multiple ShuffleMergedInput from VertexGroup

            • Major - Major loss of function.
            • Closed
            is related to

            Sub-task - The sub-task of the issue PIG-3835 Improve performance of union

            • Major - Major loss of function.
            • Closed

              Activity

              Descending order - Click to sort in ascending order
              Hide
              Permalink
              rohini Rohini Palaniswamy added a comment -

              Committed to tez branch. Thanks Daniel and Cheolsoo for the review.

              Show
              rohini Rohini Palaniswamy added a comment - Committed to tez branch. Thanks Daniel and Cheolsoo for the review.
              Hide
              Permalink
              rohini Rohini Palaniswamy added a comment -

              Changes done:

              • Created a new input in TEZ-1003 and used that so that we can turn on UnionOptimizer by default. Without that seeing lot of performance degradation in production scripts.
              • Added lot of e2e tests for UnionOptimizer and fixed code based on the issues found.
              • Fixed couple of other minor issues like
              • default parallelism not honored
              • Serializing full store was causing problems with some UDFs on deserialize for checkOutputSpecs.

              This patch depends on TEZ-1003. So will check in once that is available as part of tez snapshot in maven.

              Show
              rohini Rohini Palaniswamy added a comment - Changes done: Created a new input in TEZ-1003 and used that so that we can turn on UnionOptimizer by default. Without that seeing lot of performance degradation in production scripts. Added lot of e2e tests for UnionOptimizer and fixed code based on the issues found. Fixed couple of other minor issues like default parallelism not honored Serializing full store was causing problems with some UDFs on deserialize for checkOutputSpecs. This patch depends on TEZ-1003 . So will check in once that is available as part of tez snapshot in maven.
              Hide
              Permalink
              rohini Rohini Palaniswamy added a comment -

              Review board - https://reviews.apache.org/r/20320

              Show
              rohini Rohini Palaniswamy added a comment - Review board - https://reviews.apache.org/r/20320

                People

                • Assignee:
                  rohini Rohini Palaniswamy
                  Reporter:
                  rohini Rohini Palaniswamy
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  1 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Development