Pig
  1. Pig
  2. PIG-1919

order-by on bag gives error only at runtime

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.8.0, 0.9.0
    • Fix Version/s: 0.13.0
    • Component/s: None
    • Labels:
      None

      Description

      Order-by on a bag or tuple should give error at query compile time, instead of giving an error at runtime.

      1. PIG-1919-0.patch
        3 kB
        Jonathan Coveney
      2. PIG-1919-1.patch
        3 kB
        Jonathan Coveney
      3. PIG-1919-1.patch
        8 kB
        Jonathan Coveney

        Activity

        Thejas M Nair created issue -
        Thejas M Nair made changes -
        Field Original Value New Value
        Fix Version/s 0.9.0 [ 12315191 ]
        Affects Version/s 0.8.0 [ 12314562 ]
        Affects Version/s 0.9.0 [ 12315191 ]
        Hide
        Thejas M Nair added a comment -
        grunt> l = load 't.txt' as (a,b,c);
        grunt> g = group l by a;           
        grunt> describe g;                 
        g: {group: bytearray,l: {(a: bytearray,b: bytearray,c: bytearray)}}
        grunt> o = order g by $1;   -- this should have given an error message
        grunt> dump o;
        ...
        ...
        2011-03-18 14:26:23,903 [Thread-29] WARN  org.apache.hadoop.mapred.LocalJobRunner - job_local_0002
        java.lang.ClassCastException: org.apache.pig.data.DataByteArray cannot be cast to org.apache.pig.data.DataBag
                at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POProject.processInputBag(POProject.java:487)
                at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POProject.getNext(POProject.java:202)
                at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.getNext(PhysicalOperator.java:317)
                at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLocalRearrange.getNext(POLocalRearrange.java:325)
                at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:261)
                at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:256)
                at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:58)
                at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
                at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
                at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
                at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
        
        
        Show
        Thejas M Nair added a comment - grunt> l = load 't.txt' as (a,b,c); grunt> g = group l by a; grunt> describe g; g: {group: bytearray,l: {(a: bytearray,b: bytearray,c: bytearray)}} grunt> o = order g by $1; -- this should have given an error message grunt> dump o; ... ... 2011-03-18 14:26:23,903 [ Thread -29] WARN org.apache.hadoop.mapred.LocalJobRunner - job_local_0002 java.lang.ClassCastException: org.apache.pig.data.DataByteArray cannot be cast to org.apache.pig.data.DataBag at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POProject.processInputBag(POProject.java:487) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POProject.getNext(POProject.java:202) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.getNext(PhysicalOperator.java:317) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLocalRearrange.getNext(POLocalRearrange.java:325) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:261) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:256) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:58) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
        Olga Natkovich made changes -
        Fix Version/s 0.10 [ 12316246 ]
        Olga Natkovich made changes -
        Fix Version/s 0.9.0 [ 12315191 ]
        Olga Natkovich made changes -
        Fix Version/s 0.10 [ 12316246 ]
        Hide
        Wojciech Langiewicz added a comment -

        Using Pig 0.8.1 (CDH3u1) gives this stacktrace:

        java.lang.ClassCastException: java.lang.Integer cannot be cast to org.apache.pig.data.DataBag
        	at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POProject.processInputBag(POProject.java:482)
        	at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POProject.getNext(POProject.java:197)
        	at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLocalRearrange.getNext(POLocalRearrange.java:330)
        	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:237)
        	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:232)
        	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:53)
        	at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
        	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
        	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
        	at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
        	at java.security.AccessController.doPrivileged(Native Method)
        	at javax.security.auth.Subject.doAs(Subject.java:396)
        	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
        	at org.apache.hadoop.mapred.Child.main(Child.java:264)
        Show
        Wojciech Langiewicz added a comment - Using Pig 0.8.1 (CDH3u1) gives this stacktrace: java.lang.ClassCastException: java.lang. Integer cannot be cast to org.apache.pig.data.DataBag at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POProject.processInputBag(POProject.java:482) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POProject.getNext(POProject.java:197) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLocalRearrange.getNext(POLocalRearrange.java:330) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:237) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:232) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:53) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323) at org.apache.hadoop.mapred.Child$4.run(Child.java:270) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127) at org.apache.hadoop.mapred.Child.main(Child.java:264)
        Jonathan Coveney made changes -
        Fix Version/s 0.11 [ 12318878 ]
        Fix Version/s 0.10.1 [ 12320547 ]
        Hide
        Jonathan Coveney added a comment -

        Thejas, please find attached a rudimentary stab at this. The approach I took was to check the SortInfo that is generated against the Schema of the Tuple, and to ensure that it isn't a Bag, Tuple, or Map. I throw a FrontendException if that is the case. I can add tests, but first I wanted to see if this was the right approach to take (and if the error I throw is a good one).

        Operation "let's get rid of some of these older JIRA's" has commenced.

        Show
        Jonathan Coveney added a comment - Thejas, please find attached a rudimentary stab at this. The approach I took was to check the SortInfo that is generated against the Schema of the Tuple, and to ensure that it isn't a Bag, Tuple, or Map. I throw a FrontendException if that is the case. I can add tests, but first I wanted to see if this was the right approach to take (and if the error I throw is a good one). Operation "let's get rid of some of these older JIRA's" has commenced.
        Jonathan Coveney made changes -
        Attachment PIG-1919-0.patch [ 12521903 ]
        Hide
        Jonathan Coveney added a comment -

        minor fix to make tests pass. one downside of this is that describe doesnt fail, only dump or store

        Show
        Jonathan Coveney added a comment - minor fix to make tests pass. one downside of this is that describe doesnt fail, only dump or store
        Jonathan Coveney made changes -
        Attachment PIG-1919-1.patch [ 12521904 ]
        Jonathan Coveney made changes -
        Attachment PIG-1919-1.patch [ 12521905 ]
        Hide
        Olga Natkovich added a comment -

        Jonathan, should this be assigned to you? Is this going to be finished for 0.11 or should be moved to 0.12?

        Show
        Olga Natkovich added a comment - Jonathan, should this be assigned to you? Is this going to be finished for 0.11 or should be moved to 0.12?
        Hide
        Jonathan Coveney added a comment -

        Move to 0.12, and yeah, I'll assign it to myself. It could go into 0.11 but it's not that important, really.

        Show
        Jonathan Coveney added a comment - Move to 0.12, and yeah, I'll assign it to myself. It could go into 0.11 but it's not that important, really.
        Jonathan Coveney made changes -
        Assignee Jonathan Coveney [ jcoveney ]
        Jonathan Coveney made changes -
        Fix Version/s 0.12 [ 12323380 ]
        Fix Version/s 0.11 [ 12318878 ]
        Fix Version/s 0.10.1 [ 12320547 ]
        Daniel Dai made changes -
        Fix Version/s 0.13.0 [ 12324971 ]
        Fix Version/s 0.12.0 [ 12323380 ]

          People

          • Assignee:
            Jonathan Coveney
            Reporter:
            Thejas M Nair
          • Votes:
            2 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:

              Development