Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-3358

Pig Avro Writer: Unsupported type in record:class java.lang.String

Add voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 0.11.1
    • None
    • piggybank
    • None

    Description

      Attached test case gives the exception below - the avro writer always fails when writing to a schema with a single field (I've only tested with string fields though).

      org.apache.avro.file.DataFileWriter$AppendWriteException: java.lang.RuntimeException: Unsupported type in record:class java.lang.String
      at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:263)
      at org.apache.pig.piggybank.storage.avro.PigAvroRecordWriter.write(PigAvroRecordWriter.java:49)
      at org.apache.pig.piggybank.storage.avro.AvroStorage.putNext(AvroStorage.java:727)
      at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:139)
      at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:1)
      at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:639)
      at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
      at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapOnly$Map.collect(PigMapOnly.java:48)
      at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.runPipeline(PigGenericMapBase.java:284)
      at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:277)
      at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:1)
      at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
      at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
      at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
      at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212)
      Caused by: java.lang.RuntimeException: Unsupported type in record:class java.lang.String
      at org.apache.pig.piggybank.storage.avro.PigAvroDatumWriter.getField(PigAvroDatumWriter.java:385)
      at org.apache.pig.piggybank.storage.avro.PigAvroDatumWriter.writeRecord(PigAvroDatumWriter.java:363)
      at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:66)
      at org.apache.pig.piggybank.storage.avro.PigAvroDatumWriter.write(PigAvroDatumWriter.java:99)
      at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:58)
      at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:257)
      ... 14 more

      Attachments

        1. TestTypeInference.java
          2 kB
          Nick White

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            njw45 Nick White

            Dates

              Created:
              Updated:

              Slack

                Issue deployment