Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-1812

Problem with DID_NOT_FIND_LOAD_ONLY_MAP_PLAN

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.7.0, 0.8.0
    • Fix Version/s: 0.8.1
    • Component/s: None
    • Labels:
      None
    • Environment:

      RHEL, Pig 0.8.0

    • Hadoop Flags:
      Reviewed

      Description

      Hi,
      I have the following input files:

      pkg.txt

      a 3

      {(123,1.0),(236,2.0)}

      a 3

      {(236,1.0)}

      model.txt

      a 123 2 0.33
      a 236 2 0.5

      My script is listed below:

      A = load 'pkg.txt' using PigStorage('\t') as (pkg:chararray, ts:int, cat_bag:

      {t:(id:chararray, wht:float)}

      );

      M = load 'model.txt' using PigStorage('\t') as (pkg:chararray, cat_id:chararray, ts:int, score:double);

      B = foreach A generate ts, pkg, flatten(cat_bag.id) as (cat_id:chararray);

      B = distinct B;

      H1 = cogroup M by (pkg, cat_id) inner, B by (pkg, cat_id);

      H2 = foreach H1 {
      I = order M by ts;
      J = order B by ts;
      generate flatten(group) as (pkg:chararray, cat_id:chararray), J.ts as tsorig, I.ts as tsmap;
      }

      dump H2;

      When running this script, I got a warning about "Encountered Warning DID_NOT_FIND_LOAD_ONLY_MAP_PLAN 1 time(s)" and pig error log as below:

      Pig Stack Trace

      ---------------

      ERROR 2043: Unexpected error during execution.
      org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias H2
      at org.apache.pig.PigServer.openIterator(PigServer.java:764)
      at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:612)
      at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
      at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
      at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:141)
      at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:90)
      at org.apache.pig.Main.run(Main.java:500)
      at org.apache.pig.Main.main(Main.java:107)
      Caused by: org.apache.pig.PigException: ERROR 1002: Unable to store alias H2
      at org.apache.pig.PigServer.storeEx(PigServer.java:888)
      at org.apache.pig.PigServer.store(PigServer.java:826)
      at org.apache.pig.PigServer.openIterator(PigServer.java:738)
      ... 7 more
      Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 2043: Unexpected error during execution.
      at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:403)
      at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1208)
      at org.apache.pig.PigServer.storeEx(PigServer.java:884)
      ... 9 more
      Caused by: java.lang.ClassCastException: org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLoad cannot be cast to org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLocalRearrange
      at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.SecondaryKeyOptimizer.visitMROp(SecondaryKeyOptimizer.java:352)
      at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceOper.visit(MapReduceOper.java:246)
      at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceOper.visit(MapReduceOper.java:41)
      at org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:69)
      at org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:71)
      at org.apache.pig.impl.plan.DepthFirstWalker.walk(DepthFirstWalker.java:52)
      at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:51)
      at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.compile(MapReduceLauncher.java:498)
      at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:117)
      at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:378)
      ... 11 more

      But, when I removed the DISTINCT statement before COGROUP, i.e. "B = distinct B;" this script can run smoothly. I have also tried other reducer side operations like ORDER, it seems that they will also trigger above error. This is really very confusing.

        Attachments

        1. PIG-1812-2.patch
          59 kB
          Jianyong Dai
        2. PIG-1812-1.patch
          7 kB
          Jianyong Dai

          Activity

            People

            • Assignee:
              daijy Jianyong Dai
              Reporter:
              xianyu.zhao xianyu
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: