Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-3621

Python Avro library can't read Avros made with builtin AvroStorage

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.12.0
    • Fix Version/s: 0.12.1, 0.13.0
    • Component/s: internal-udfs
    • Labels:
      None

      Description

      Using this script:

      from avro import schema, datafile, io
      import pprint
      import sys
      import json

      field_id = None

      1. Optional key to print
        if (len(sys.argv) > 2):
        field_id = sys.argv[2]
      1. Test reading avros
        rec_reader = io.DatumReader()
      1. Create a 'data file' (avro file) reader
        df_reader = datafile.DataFileReader(
        open(sys.argv[1]),
        rec_reader
        )

      the last line fails with:

      Traceback (most recent call last):
      File "/Users/rjurney/bin/cat_avro", line 22, in <module>
      rec_reader
      File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/avro/datafile.py", line 247, in _init_
      self.datum_reader.writers_schema = schema.parse(self.get_meta(SCHEMA_KEY))
      File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/avro/schema.py", line 784, in parse
      return make_avsc_object(json_data, names)
      File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/avro/schema.py", line 740, in make_avsc_object
      return RecordSchema(name, namespace, fields, names, type, doc, other_props)
      File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/avro/schema.py", line 653, in _init_
      other_props)
      File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/avro/schema.py", line 294, in _init_
      new_name = names.add_name(name, namespace, self)
      File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/avro/schema.py", line 268, in add_name
      raise SchemaParseException(fail_msg)
      avro.schema.SchemaParseException: record is a reserved type name.

        Attachments

        1. PIG-3621-3.patch
          0.5 kB
          Cheolsoo Park
        2. PIG-3631.patch
          0.6 kB
          Russell Jurney
        3. PIG-3631-2.patch
          0.9 kB
          Cheolsoo Park

          Issue Links

            Activity

              People

              • Assignee:
                russell.jurney Russell Jurney
                Reporter:
                russell.jurney Russell Jurney
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: