Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.9.0
    • Component/s: None
    • Labels:
      None

      Description

      When query execution plan is created by pig, it assumes the type to be bytearray because there is no schema information associated with map fields.
      But at run time, the loader might return the actual type. This results in a ClassCastException.
      This issue points to the larger issue of the way pig is handling types for map-value.

      This issue should be fixed in the context of revisiting the frontend logic and pig-latin semantics.

      This is related to PIG-880 . The patch in PIG-880 changed PigStorage to always return bytearray for map values to work around this, but other loaders like BinStorage can return the actual type causing this issue.

        Issue Links

          Activity

          Hide
          thejas Thejas M Nair added a comment -
          l = load 'st_attr2.bin' using BinStorage();
          f = foreach l generate $1, $4#'origin';  --   $4#'origin is stored as chararray
          o = order f by $2;
          dump o; 
          

          It results in map-reduce failure with error -

          java.lang.ClassCastException: org.apache.pig.impl.io.NullableText cannot be cast to
          org.apache.pig.impl.io.NullableBytesWritable
          at
          org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigBytesRawComparator.compare(PigBytesRawComparator.java:94)
          at java.util.Arrays.binarySearch0(Arrays.java:2105)
          at java.util.Arrays.binarySearch(Arrays.java:2043)
          at
          org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.partitioners.WeightedRangePartitioner.getPartition(WeightedRangePartitioner.java:64)
          at
          org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.partitioners.WeightedRangePartitioner.getPartition(WeightedRangePartitioner.java:53)
          at org.apache.hadoop.mapred.MapTask$OldOutputCollector.collect(MapTask.java:466)
          at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Map.collect(PigMapReduce.java:108)
          at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:251)
          at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:240)
          at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Map.map(PigMapReduce.java:93)
          at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
          at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
          at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
          at org.apache.hadoop.mapred.Child.main(Child.java:170)

          Show
          thejas Thejas M Nair added a comment - l = load 'st_attr2.bin' using BinStorage(); f = foreach l generate $1, $4#'origin'; -- $4#'origin is stored as chararray o = order f by $2; dump o; It results in map-reduce failure with error - java.lang.ClassCastException: org.apache.pig.impl.io.NullableText cannot be cast to org.apache.pig.impl.io.NullableBytesWritable at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigBytesRawComparator.compare(PigBytesRawComparator.java:94) at java.util.Arrays.binarySearch0(Arrays.java:2105) at java.util.Arrays.binarySearch(Arrays.java:2043) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.partitioners.WeightedRangePartitioner.getPartition(WeightedRangePartitioner.java:64) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.partitioners.WeightedRangePartitioner.getPartition(WeightedRangePartitioner.java:53) at org.apache.hadoop.mapred.MapTask$OldOutputCollector.collect(MapTask.java:466) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Map.collect(PigMapReduce.java:108) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:251) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:240) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Map.map(PigMapReduce.java:93) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307) at org.apache.hadoop.mapred.Child.main(Child.java:170)
          Hide
          thejas Thejas M Nair added a comment -

          In previous comment

          o = order f by $2;
          

          should have been -

          o = order f by $1;
          
          Show
          thejas Thejas M Nair added a comment - In previous comment o = order f by $2; should have been - o = order f by $1;
          Hide
          olgan Olga Natkovich added a comment -

          Dealying - will address as part of semantic cleanup

          Show
          olgan Olga Natkovich added a comment - Dealying - will address as part of semantic cleanup
          Hide
          daijy Daniel Dai added a comment -

          Fixed along with PIG-1277.

          Show
          daijy Daniel Dai added a comment - Fixed along with PIG-1277 .

            People

            • Assignee:
              alangates Alan Gates
              Reporter:
              thejas Thejas M Nair
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development