Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.9.0
    • Component/s: None
    • Labels:
      None

      Description

      When query execution plan is created by pig, it assumes the type to be bytearray because there is no schema information associated with map fields.
      But at run time, the loader might return the actual type. This results in a ClassCastException.
      This issue points to the larger issue of the way pig is handling types for map-value.

      This issue should be fixed in the context of revisiting the frontend logic and pig-latin semantics.

      This is related to PIG-880 . The patch in PIG-880 changed PigStorage to always return bytearray for map values to work around this, but other loaders like BinStorage can return the actual type causing this issue.

        Issue Links

          Activity

          Olga Natkovich made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Daniel Dai made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          Daniel Dai added a comment -

          Fixed along with PIG-1277.

          Show
          Daniel Dai added a comment - Fixed along with PIG-1277 .
          Daniel Dai made changes -
          Link This issue duplicates PIG-1277 [ PIG-1277 ]
          Daniel Dai made changes -
          Link This issue duplicates PIG-1065 [ PIG-1065 ]
          Daniel Dai made changes -
          Link This issue duplicates PIG-1065 [ PIG-1065 ]
          Alan Gates made changes -
          Assignee Alan Gates [ alangates ]
          Olga Natkovich made changes -
          Fix Version/s 0.9.0 [ 12315191 ]
          Olga Natkovich made changes -
          Fix Version/s 0.7.0 [ 12314397 ]
          Hide
          Olga Natkovich added a comment -

          Dealying - will address as part of semantic cleanup

          Show
          Olga Natkovich added a comment - Dealying - will address as part of semantic cleanup
          Thejas M Nair made changes -
          Fix Version/s 0.7.0 [ 12314397 ]
          Hide
          Thejas M Nair added a comment -

          In previous comment

          o = order f by $2;
          

          should have been -

          o = order f by $1;
          
          Show
          Thejas M Nair added a comment - In previous comment o = order f by $2; should have been - o = order f by $1;
          Hide
          Thejas M Nair added a comment -
          l = load 'st_attr2.bin' using BinStorage();
          f = foreach l generate $1, $4#'origin';  --   $4#'origin is stored as chararray
          o = order f by $2;
          dump o; 
          

          It results in map-reduce failure with error -

          java.lang.ClassCastException: org.apache.pig.impl.io.NullableText cannot be cast to
          org.apache.pig.impl.io.NullableBytesWritable
          at
          org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigBytesRawComparator.compare(PigBytesRawComparator.java:94)
          at java.util.Arrays.binarySearch0(Arrays.java:2105)
          at java.util.Arrays.binarySearch(Arrays.java:2043)
          at
          org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.partitioners.WeightedRangePartitioner.getPartition(WeightedRangePartitioner.java:64)
          at
          org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.partitioners.WeightedRangePartitioner.getPartition(WeightedRangePartitioner.java:53)
          at org.apache.hadoop.mapred.MapTask$OldOutputCollector.collect(MapTask.java:466)
          at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Map.collect(PigMapReduce.java:108)
          at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:251)
          at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:240)
          at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Map.map(PigMapReduce.java:93)
          at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
          at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
          at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
          at org.apache.hadoop.mapred.Child.main(Child.java:170)

          Show
          Thejas M Nair added a comment - l = load 'st_attr2.bin' using BinStorage(); f = foreach l generate $1, $4#'origin'; -- $4#'origin is stored as chararray o = order f by $2; dump o; It results in map-reduce failure with error - java.lang.ClassCastException: org.apache.pig.impl.io.NullableText cannot be cast to org.apache.pig.impl.io.NullableBytesWritable at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigBytesRawComparator.compare(PigBytesRawComparator.java:94) at java.util.Arrays.binarySearch0(Arrays.java:2105) at java.util.Arrays.binarySearch(Arrays.java:2043) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.partitioners.WeightedRangePartitioner.getPartition(WeightedRangePartitioner.java:64) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.partitioners.WeightedRangePartitioner.getPartition(WeightedRangePartitioner.java:53) at org.apache.hadoop.mapred.MapTask$OldOutputCollector.collect(MapTask.java:466) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Map.collect(PigMapReduce.java:108) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:251) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:240) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Map.map(PigMapReduce.java:93) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307) at org.apache.hadoop.mapred.Child.main(Child.java:170)
          Thejas M Nair made changes -
          Field Original Value New Value
          Parent PIG-998 [ 12437538 ]
          Issue Type Bug [ 1 ] Sub-task [ 7 ]
          Thejas M Nair created issue -

            People

            • Assignee:
              Alan Gates
              Reporter:
              Thejas M Nair
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development