[PARQUET-651] Parquet-avro fails to decode array of record with a single field name "element" correctly

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.7.0, 1.8.0, 1.8.1, 1.9.0
• Fix Version/s: 1.9.0, 1.8.2
• Component/s: parquet-avro
• Assignee: Ryan Blue
• Reporter: Cheng Lian
• Created: 01/Jul/16 13:16
• Updated: 21/Apr/18 12:39
• Resolved: 17/Jul/16 22:00

Description

Found this issue while investigating SPARK-16344.

For the following Parquet schema

message root {
  optional group f (LIST) {
    repeated group list {
      optional group element {
        optional int64 element;
      }
    }
  }
}

parquet-avro decodes it as something like this:

record SingleElement {
  int element;
}

record NestedSingleElement {
  SingleElement element;
}

record Spark16344Wrong {
  array<NestedSingleElement> f;
}

while correct interpretation should be:

record SingleElement {
  int element;
}

record Spark16344 {
  array<SingleElement> f;
}

The reason is that the element syntactic group for LIST in

<list-repetition> group <name> (LIST) {
  repeated group list {
    <element-repetition> <element-type> element;
  }
}

is recognized as a record field named element. The problematic code lies in AvroRecordConverter.isElementType(). We should probably check the standard 3-level layout first before falling back to the legacy 2-level layout.

Issue Links

• is related to: SPARK-16344 Array of struct with a single field name "element" can't be decoded from Parquet files written by Spark 1.6+
• links to: PR #352