[PARQUET-2037] Write INT96 with parquet-avro

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Component/s: parquet-avro, parquet-mr
• Assignee: Gabor Szadovszky
• Reporter: Gabor Szadovszky
• Created: 27/Apr/21 17:03
• Updated: 12/May/21 08:08
• Resolved: 12/May/21 08:08

Description

This jira is about the write path of PARQUET-1928.

The issue here is how to identify an Avro FIXED type that was an INT96 before. Of course, this feature would be behind a configuration flag similarly to PARQUET-1928. But even with this flag it is not obvious to differentiate a "simple" FIXED[12] byte from one that was an INT96 before.

Two options to solve this issue:

• Write the doc field of the avro schema that the FIXED value was an INT96.
• Instead of implementing a configuration flag let the user specify the names of the columns to be converted to INT96 via the configuration.