Pig
  1. Pig
  2. PIG-2202

AvroStorage doesn't work with Avro 1.5.1

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.10.0
    • Component/s: None
    • Labels:
      None
    • Release Note:
      AvroStorage upgraded to work with Avro 1.5.1.

      Description

      See PIG-2195. Pig jobs that use AvroStorage to write records using the 'schema' or 'data' options fail when using Avro 1.5.1 with the following exception:

      java.lang.ClassCastException: org.apache.pig.data.BinSedesTuple cannot be cast to org.apache.avro.generic.IndexedRecord
      
      1. PIG-2202_2.patch
        2 kB
        Bill Graham
      2. PIG-2202_1.patch
        2 kB
        Bill Graham

        Activity

        Daniel Dai made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Alan Gates made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Fix Version/s 0.10 [ 12316246 ]
        Resolution Fixed [ 1 ]
        Hide
        Alan Gates added a comment -

        Patch checked in. Thanks Bill.

        Show
        Alan Gates added a comment - Patch checked in. Thanks Bill.
        Hide
        Alan Gates added a comment -

        I'll be reviewing this patch.

        Show
        Alan Gates added a comment - I'll be reviewing this patch.
        Bill Graham made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Release Note AvroStorage upgraded to work with Avro 1.5.1.
        Assignee Bill Graham [ billgraham ]
        Bill Graham made changes -
        Attachment PIG-2202_2.patch [ 12489552 ]
        Hide
        Bill Graham added a comment -

        Attached is patch 2 that upgrades PigAvroDatumWriter to work with changes to it's Avro superclass, GenericDatumWriter.

        This patch is backward compatible with Avro 1.4.1.

        Show
        Bill Graham added a comment - Attached is patch 2 that upgrades PigAvroDatumWriter to work with changes to it's Avro superclass, GenericDatumWriter . This patch is backward compatible with Avro 1.4.1.
        Bill Graham made changes -
        Field Original Value New Value
        Attachment PIG-2202_1.patch [ 12489147 ]
        Hide
        Bill Graham added a comment -

        Attached is patch 1, which allows you to compile against Avro 1.5.1. This patch still results in 2 unit test failures with exceptions as shown below.

        Is it acceptable for AvroStorage to be upgraded to work with Avro 1.5.1 and not support earlier Avro releases?

        11/08/02 17:40:35 WARN mapred.LocalJobRunner: job_local_0005
        java.lang.ClassCastException: org.apache.pig.data.BinSedesTuple cannot be cast to org.apache.avro.generic.IndexedRecord
                at org.apache.avro.generic.GenericData.getField(GenericData.java:470)
                at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:102)
                at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:65)
                at org.apache.pig.piggybank.storage.avro.PigAvroDatumWriter.write(PigAvroDatumWriter.java:99)
                at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:57)
                at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:244)
                at org.apache.pig.piggybank.storage.avro.PigAvroRecordWriter.write(PigAvroRecordWriter.java:49)
                at org.apache.pig.piggybank.storage.avro.AvroStorage.putNext(AvroStorage.java:580)
                at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POStore.getNext(POStore.java:143)
                at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POSplit.runPipeline(POSplit.java:254)
                at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POSplit.processPlan(POSplit.java:236)
                at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POSplit.getNext(POSplit.java:228)
                at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.runPipeline(PigMapReduce.java:456)
                at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.processOnePackageOutput(PigMapReduce.java:424)
                at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.reduce(PigMapReduce.java:404)
                at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.reduce(PigMapReduce.java:258)
                at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:176)
                at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:566)
                at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:408)
                at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:216)
        
        Show
        Bill Graham added a comment - Attached is patch 1, which allows you to compile against Avro 1.5.1. This patch still results in 2 unit test failures with exceptions as shown below. Is it acceptable for AvroStorage to be upgraded to work with Avro 1.5.1 and not support earlier Avro releases? 11/08/02 17:40:35 WARN mapred.LocalJobRunner: job_local_0005 java.lang.ClassCastException: org.apache.pig.data.BinSedesTuple cannot be cast to org.apache.avro.generic.IndexedRecord at org.apache.avro.generic.GenericData.getField(GenericData.java:470) at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:102) at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:65) at org.apache.pig.piggybank.storage.avro.PigAvroDatumWriter.write(PigAvroDatumWriter.java:99) at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:57) at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:244) at org.apache.pig.piggybank.storage.avro.PigAvroRecordWriter.write(PigAvroRecordWriter.java:49) at org.apache.pig.piggybank.storage.avro.AvroStorage.putNext(AvroStorage.java:580) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POStore.getNext(POStore.java:143) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POSplit.runPipeline(POSplit.java:254) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POSplit.processPlan(POSplit.java:236) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POSplit.getNext(POSplit.java:228) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.runPipeline(PigMapReduce.java:456) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.processOnePackageOutput(PigMapReduce.java:424) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.reduce(PigMapReduce.java:404) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.reduce(PigMapReduce.java:258) at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:176) at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:566) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:408) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:216)
        Bill Graham created issue -

          People

          • Assignee:
            Bill Graham
            Reporter:
            Bill Graham
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development