[PARQUET-1297] [Java] SchemaConverter should not convert from Timestamp(TimeUnit.SECOND) and Timestamp(TimeUnit.NANOSECOND) of Arrow

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 1.11.0
• Component/s: parquet-mr
• Assignee: Masayuki Takahashi
• Reporter: Masayuki Takahashi
• Due: 29/Apr/18
• Created: 12/May/18 12:40
• Updated: 13/May/18 17:31
• Resolved: 13/May/18 17:31

Description

Arrow's 'Timestamp' definition is below:

{
  "name" : "timestamp",
  "unit" : "SECOND|MILLISECOND|MICROSECOND|NANOSECOND"
}

http://arrow.apache.org/docs/metadata.html

But Parquet only supports 'TIMESTAMP_MILLIS' and 'TIMESTAMP_MICROS'.
https://github.com/Apache/parquet-format/blob/master/LogicalTypes.md

Therefore SchemaConverter should not convert from Timestamp(TimeUnit.SECOND) and Timestamp(TimeUnit.NANOSECOND) of Arrow to Parquet.

Related:
https://issues.apache.org/jira/browse/PARQUET-1285
 

Issue Links

• is a clone of: PARQUET-1285 [Java] SchemaConverter should not convert from TimeUnit.SECOND AND TimeUnit.NANOSECOND of Arrow
• links to: GitHub Pull Request #477