Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-4059 Pig on Spark
  3. PIG-4189

Make cross join work with Spark

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: spark-branch
    • Component/s: spark
    • Labels:
      None

      Description

      Related e2e tests: Cross_1 - Cross_5

      Sample script:
      a = load '/user/pig/tests/data/singlefile/studenttab10k' as (name, age, gpa);
      b = load '/user/pig/tests/data/singlefile/votertab10k' as (name, age, registration, contributions);
      c = filter a by age < 19 and gpa < 1.0;
      d = filter b by age < 19;
      e = cross c, d;
      store e into '/user/pig/out/praveenr-1411378727-nightly.conf/Cross_1.out';

      Log:
      [Executor task launch worker-1] ERROR org.apache.spark.executor.Executor - Exception in task ID 2
      java.lang.RuntimeException: org.apache.pig.backend.executionengine.ExecException: ERROR 2078: Caught error from UDF: org.apache.pig.impl.builtin.GFCross [Unable to get parallelism hint from job conf]
      at org.apache.pig.backend.hadoop.executionengine.spark.converter.POOutputConsumerIterator.readNext(POOutputConsumerIterator.java:57)
      at org.apache.pig.backend.hadoop.executionengine.spark.converter.POOutputConsumerIterator.hasNext(POOutputConsumerIterator.java:63)
      at scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:41)
      at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
      at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
      at scala.collection.Iterator$class.foreach(Iterator.scala:727)
      at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
      at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:161)
      at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:102)
      at org.apache.spark.scheduler.Task.run(Task.scala:53)
      at org.apache.spark.executor.Executor$TaskRunner$$anonfun$run$1.apply$mcV$sp(Executor.scala:213)
      at org.apache.spark.deploy.SparkHadoopUtil.runAsUser(SparkHadoopUtil.scala:49)
      at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:178)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:744)
      Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 2078: Caught error from UDF: org.apache.pig.impl.builtin.GFCross [Unable to get parallelism hint from job conf]
      at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:372)
      at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNextDataBag(POUserFunc.java:388)
      at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.getNext(PhysicalOperator.java:331)
      at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.processPlan(POForEach.java:378)
      at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNextTuple(POForEach.java:298)
      at org.apache.pig.backend.hadoop.executionengine.spark.converter.ForEachConverter$ForEachFunction$1.getNextResult(ForEachConverter.java:53)
      at org.apache.pig.backend.hadoop.executionengine.spark.converter.POOutputConsumerIterator.readNext(POOutputConsumerIterator.java:36)
      ... 15 more
      Caused by: java.io.IOException: Unable to get parallelism hint from job conf
      at org.apache.pig.impl.builtin.GFCross.exec(GFCross.java:61)
      at org.apache.pig.impl.builtin.GFCross.exec(GFCross.java:1)
      at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:344)
      ... 21 more

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                mohitsabharwal Mohit Sabharwal
                Reporter:
                praveenr019 Praveen Rachabattuni
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: