Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-3584

AvroStorage does not correctly translate arrays of strings

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.12.0
    • Fix Version/s: 0.13.0
    • Component/s: None
    • Labels:
      None
    • Patch Info:
      Patch Available

      Description

      I found an issue this morning with AvroStorage and arrays of strings. Suppose that you have an Avro file with an array of strings like this:

      {"name" : "example",
      "namespace" : "org.apache.pig.avro",
      "type" : "record",
      "fields" : [
      {"name" : "arrayOfStrings",
      "type" : "type" : "array", "items" : "string"}}

      The current version of AvroStorage would translate this schema into a bag of tuples, each containing a single chararray field.

      The current version of AvroStorage will naively place each value in the array into a tuple inside a bag of values. Unfortunately, each value is a Utf8 item (not a Java String type). This can cause problems in later processing steps, because Pig does not know how to deal with Utf8 objects.

      I've written a patch for AvroStorage that will correctly translate objects in arrays into a form that Pig can process. (While I was at it, I made sure that we correctly translated other Avro types, including fixed, enums, and unions.)

      1. PIG-3584.patch
        6 kB
        Joseph Adler

        Issue Links

          Activity

          Hide
          jadler Joseph Adler added a comment -

          Patch for this issue

          Show
          jadler Joseph Adler added a comment - Patch for this issue
          Hide
          cheolsoo Cheolsoo Park added a comment -

          +1. I will commit this.

          Show
          cheolsoo Cheolsoo Park added a comment - +1. I will commit this.
          Hide
          cheolsoo Cheolsoo Park added a comment -

          Committed to trunk. Thanks Joe!

          Show
          cheolsoo Cheolsoo Park added a comment - Committed to trunk. Thanks Joe!

            People

            • Assignee:
              jadler Joseph Adler
              Reporter:
              jadler Joseph Adler
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development