Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-4059 Pig on Spark
  3. PIG-4709

Support combine for spark mode

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: spark-branch
    • Component/s: spark
    • Labels:

      Description

      Currently, the GROUPBY operator of PIG is mapped by Spark's CoGroup. When the grouped data is consumed by subsequent operations to perform algebraic operations, this is sub-optimal as there is lot of shuffle traffic.

      The Spark Plan must be optimized to use reduceBy, where possible, so that a combiner is used.

        Attachments

        1. PIG-4709-v3.patch
          66 kB
          Pallavi Rao
        2. PIG-4709-v2.patch
          67 kB
          Pallavi Rao
        3. TEST-org.apache.pig.test.TestCombiner.xml
          1.24 MB
          liyunzhang
        4. PIG-4709-v1.patch
          80 kB
          Pallavi Rao
        5. PIG-4709.patch
          85 kB
          Pallavi Rao

          Issue Links

            Activity

              People

              • Assignee:
                pallavi.rao Pallavi Rao
                Reporter:
                pallavi.rao Pallavi Rao
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: