[PARQUET-1305] Backward incompatible change introduced in 1.8

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.11.0
• Labels: pull-request-available
• Assignee: Nándor Kollár
• Reporter: Nándor Kollár
• Created: 18/May/18 15:00
• Updated: 07/Nov/18 10:19
• Resolved: 07/Nov/18 10:19

Description

ThriftSchemaConverter#convert(StructType struct) introduced a backward incompatible logic in Parquet 1.8, If StructType's deprecated constructor is used to create StructTypes, and this convert method is used, then the client receives the following exception:

org.apache.parquet.ShouldNeverHappenException: Encountered UNKNOWN StructOrUnionType

	at org.apache.parquet.thrift.ThriftSchemaConvertVisitor.isUnion(ThriftSchemaConvertVisitor.java:342)
	at org.apache.parquet.thrift.ThriftSchemaConvertVisitor.visit(ThriftSchemaConvertVisitor.java:218)
	at org.apache.parquet.thrift.ThriftSchemaConvertVisitor.visit(ThriftSchemaConvertVisitor.java:74)
	at org.apache.parquet.thrift.struct.ThriftType$StructType.accept(ThriftType.java:269)
	at org.apache.parquet.thrift.ThriftSchemaConvertVisitor.convert(ThriftSchemaConvertVisitor.java:93)
	at org.apache.parquet.thrift.ThriftSchemaConverter.convert(ThriftSchemaConverter.java:75)

To illustrate the issue, here's a test case, which passes on older versions, but fails on master:

  @Test
  public void testIncompatibleThriftConverterChange() {
    ThriftSchemaConverter converter = new ThriftSchemaConverter();

    ThriftType.StructType structType = new ThriftType.StructType(
      asList(
        new ThriftField("a", (short)1, REQUIRED, new ThriftType.StringType()),
        new ThriftField("b", (short) 2, REQUIRED, new ThriftType.StringType())
      )
    );
    converter.convert(structType);
  }

Issue Links

• is part of: PARQUET-1295 Parquet libraries do not follow proper semantic versioning
• links to: GitHub Pull Request #483