Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Fix Version/s: 1.0.9, 1.1.0
    • Component/s: None
    • Labels:
      None

      Description

      It appears that some changes need to be made to support CompositeColumns. Right now if you try to load and use a column family that utilizes composite columns you get the following exception[1].

      It appears to me that we need to modify the storage handler for Pig to support this scenario.

      [1]

      ================================================================================
      Backend error message
      ---------------------
      java.lang.RuntimeException: Unexpected data type -1 found in stream.
      at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:478)
      at org.apache.pig.data.BinInterSedes.writeTuple(BinInterSedes.java:541)
      at org.apache.pig.data.BinInterSedes.writeBag(BinInterSedes.java:522)
      at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:361)
      at org.apache.pig.data.BinInterSedes.writeTuple(BinInterSedes.java:541)
      at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:357)
      at org.apache.pig.data.BinSedesTuple.write(BinSedesTuple.java:57)
      at org.apache.pig.impl.io.PigNullableWritable.write(PigNullableWritable.java:123)
      at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:90)
      at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:77)
      at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1061)
      at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:691)
      at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
      at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Map.collect(PigMapReduce.java:116)
      at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:239)
      at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:232)
      at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:53)
      at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
      at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
      at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
      at org.apache.hadoop.mapred.Child$4.run(Child.java:272)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:396)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
      at org.apache.hadoop.mapred.Child.main(Child.java:266)

      Backend error message
      ---------------------
      java.lang.Throwable: Child Error
      at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)
      Caused by: java.io.IOException: Task process exit with nonzero status of 65.
      at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258)

      1. 3684-jalkanen.txt
        6 kB
        Janne Jalkanen
      2. 3684-jalkanen-test.txt
        1 kB
        Janne Jalkanen
      3. 3684-jalkanen-test-v2.txt
        2 kB
        Janne Jalkanen

        Activity

        Hide
        Brandon Williams added a comment -

        Committed, thanks!

        Show
        Brandon Williams added a comment - Committed, thanks!
        Hide
        Janne Jalkanen added a comment -

        Improved test patch against the tests; this time with also Long:Long composite type.

        Show
        Janne Jalkanen added a comment - Improved test patch against the tests; this time with also Long:Long composite type.
        Hide
        Janne Jalkanen added a comment -

        Patch to provide tests against the Composite Columns.

        Show
        Janne Jalkanen added a comment - Patch to provide tests against the Composite Columns.
        Hide
        Janne Jalkanen added a comment -

        Also, apologies for extra crap; my OCD demands that my git is configured to remove extra space at the end of the lines . If this approach looks feasible, I'll make a cleaner patch.

        Show
        Janne Jalkanen added a comment - Also, apologies for extra crap; my OCD demands that my git is configured to remove extra space at the end of the lines . If this approach looks feasible, I'll make a cleaner patch.
        Hide
        Janne Jalkanen added a comment -

        This patch (against cassandra-1.0) brings in basic support for Composite Columns.

        I needed a simple way to deconstruct an AbstractCompositeType, so I had to enhance that. Dunno if that's desireable, but it's certainly easy :-P

        Also the column name is an untyped tuple; not sure what would be the best way to extract the schema for it.

        Show
        Janne Jalkanen added a comment - This patch (against cassandra-1.0) brings in basic support for Composite Columns. I needed a simple way to deconstruct an AbstractCompositeType, so I had to enhance that. Dunno if that's desireable, but it's certainly easy :-P Also the column name is an untyped tuple; not sure what would be the best way to extract the schema for it.

          People

          • Assignee:
            Janne Jalkanen
            Reporter:
            Benjamin Coverston
            Reviewer:
            Brandon Williams
          • Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development