Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-4326

AvroStorageSchemaConversionUtilities does not properly convert schema for maps of arrays of records

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.12.0, 0.13.0
    • Fix Version/s: 0.15.0
    • Component/s: impl
    • Labels:
      None
    • Patch Info:
      Patch Available
    • Hadoop Flags:
      Reviewed

      Description

      I tried to convert the avro schema of a map of arrays of records into the proper pig schema and got always empty map schemas in pig.

      The reason is that the AvroStorageSchemaConversionUtilities does only assume records or primitive types as content of the map. However, a map of arrays, or a map of map, could have a schema itself and requires recursive calling to derive the full schema.

      I wrote a unit test to test for maps of arrays of records which fails with every pig release since the AvroStorage was rewritten (I think this was in 0.12), and there have been no changes since then in the trunk.

      Further the attached patch contains the (rather simple) fix that makes the schema conversion utils succeed.

      Would appreciate further comments and if this can be included upstream.

        Attachments

        1. supportForMapsOfArraysOfRecords.patch
          6 kB
          Michael Prim
        2. PIG-4326-0.patch
          2 kB
          Jianyong Dai
        3. mapsOfArraysOfRecords.patch
          5 kB
          Michael Prim

          Activity

            People

            • Assignee:
              mprim Michael Prim
              Reporter:
              mprim Michael Prim
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: