Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-4793

AvroStorage issues during write into HDFS

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Invalid
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: piggybank
    • Labels:
      None

      Description

      Dear,

      I created the simple pig script that reads two avro files, merges the two relations and stores it into the output avro file.

      I tried to store output relation into avro file using:
      store outputSet into 'avrostorage' using AvroStorage();

      Some workaround was required because pig has problems to process schema with :: (maybe another bug?)

      Added code below the result 'avrostorage' file was generated.
      outputSet = foreach outputSet generate $0 as (name:chararray) , $1 as (customerId:chararray), $2 as (VIN:chararray) , $3 as (Birthdate:chararray), $4 as (Mileage:chararray) ,$5 as (Fuel_Consumption:chararray);

      When I tried to store avro file with the schema definition using code below,
      strange error is occurring https://bpaste.net/show/ccf0cbef06a9 (Full log).

      ...
      10.0.1.47:8050 2016-01-29 17:24:39,600 [main] ERROR org.apache.pig.tools.pigstats.mapreduce.MRPigStatsUtil - 1 map reduce job(s) failed!
      ...

      STORE outputSet INTO '/avro-dest/Test-20160129-1401822'
      USING org.apache.pig.piggybank.storage.avro.AvroStorage('no_schema_check', 'schema', '....')

      Sample data and pig script:

      https://drive.google.com/file/d/0B6RZ_9vVuTEcd01aWm9zczNUUWc/view

      I think these might be two important issues, could you please investigate?

      Thank you

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              lenovomi John Smith
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: