Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-4266 Umbrella jira for unit tests for Spark
  3. PIG-4781

Fix remaining unit failure about "TestCollectedGroup" for spark engine

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: spark-branch
    • Component/s: spark
    • Labels:
      None

      Description

      in https://builds.apache.org/job/Pig-spark/lastUnsuccessfulBuild/#showFailuresLink, it shows that following unit test fails:
      org.apache.pig.test.TestCollectedGroup.testMapsideGroupWithMergeJoin

      This fails because currently we use regular join to implement merge join.
      the exeception is

      Caused by: org.apache.pig.backend.hadoop.executionengine.spark.plan.SparkCompilerException: ERROR 2171: Expected one but found more then one root physical operator in physical physicalPlan.
      	at org.apache.pig.backend.hadoop.executionengine.spark.plan.SparkCompiler.visitCollectedGroup(SparkCompiler.java:512)
      	at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POCollectedGroup.visit(POCollectedGroup.java:93)
      	at org.apache.pig.backend.hadoop.executionengine.spark.plan.SparkCompiler.compile(SparkCompiler.java:259)
      	at org.apache.pig.backend.hadoop.executionengine.spark.plan.SparkCompiler.compile(SparkCompiler.java:240)
      	at org.apache.pig.backend.hadoop.executionengine.spark.plan.SparkCompiler.compile(SparkCompiler.java:240)
      	at org.apache.pig.backend.hadoop.executionengine.spark.plan.SparkCompiler.compile(SparkCompiler.java:165)
      	at org.apache.pig.backend.hadoop.executionengine.spark.SparkLauncher.compile(SparkLauncher.java:425)
      	at org.apache.pig.backend.hadoop.executionengine.spark.SparkLauncher.launchPig(SparkLauncher.java:150)
      	at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:301)
      	at org.apache.pig.PigServer.launchPlan(PigServer.java:1390)
      	at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1375)
      	at org.apache.pig.PigServer.storeEx(PigServer.java:1034)
      	... 27 more
      

      After we implement Merge join, this unit test can be fixed.

        Attachments

        1. PIG-4781.patch
          4 kB
          liyunzhang

          Issue Links

            Activity

              People

              • Assignee:
                kellyzly liyunzhang
                Reporter:
                kellyzly liyunzhang
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: