[PARQUET-2292] Improve default SpecificRecord model selection for Avro{Write,Read}Support

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.14.0, 1.13.1
• Component/s: parquet-avro
• Assignee: Claire McGinty
• Reporter: Claire McGinty
• Created: 21/Apr/23 19:31
• Updated: 23/Jun/24 03:32
• Resolved: 06/May/23 02:13

Description

AvroWriteSupport/AvroReadSupport can improve the precision of their default `model` selection. Currently they default to new SpecificDataSupplier().get()[0]. This means that SpecificRecord classes that contain logical types will fail out-of-the-box unless a specific DATA_SUPPLIER is configured that contains logical type conversions.

I think we can improve this and make logical types work by default by defaulting to the value of the `MODEL$` field that every SpecificRecordBase implementation contains, which already contains all the logical conversions for that Avro type. It would require reflection, but that's what the Avro library is already doing to fetch models for Specific types[1].

 

[0] https://github.com/apache/parquet-mr/blob/d38044f5395494e1543581a4b763f624305d3022/parquet-avro/src/main/java/org/apache/parquet/avro/AvroWriteSupport.java#L403-L407

[1] https://github.com/apache/avro/blob/release-1.11.1/lang/java/avro/src/main/java/org/apache/avro/specific/SpecificData.java#L76-L86