[PARQUET-1253] Support for new logical type representation

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.11.0
• Component/s: parquet-mr
• Assignee: Nándor Kollár
• Reporter: Nándor Kollár
• Created: 26/Mar/18 12:47
• Updated: 21/Nov/18 12:46
• Resolved: 24/May/18 11:48

Description

Latest parquet-format introduced a new representation for logical types. As of now this is not yet supported in parquet-mr, thus there's no way to use parametrized UTC normalized timestamp data types. When reading and writing Parquet files, besides 'converted_type' parquet-mr should use the new 'logicalType' field in SchemaElement to tell the current logical type annotation. To maintain backward compatibility, the semantic of converted_type shouldn't change.

Issue Links

• depends upon: PARQUET-1293 Build failure when using Java 8 lambda expressions
• is depended upon by: PARQUET-1284 Incorporate new logical type parameters into schema language
• is duplicated by: PARQUET-905 Add "Floating Timestamp" logical type
• is duplicated by: PARQUET-1284 Incorporate new logical type parameters into schema language
• links to: GitHub Pull Request #463