[ARROW-17506] [Python][C++] pyarrow parquet writer - missing time logical type

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 8.0.0, 9.0.0
• Component/s: C++, Parquet, Python
• Flags: Important
• External issue URL: https://github.com/apache/arrow/issues/32766

Language

Python

• Assignee: Quang Hoang
• Reporter: Vadim Goy
• Created: 23/Aug/22 17:34
• Updated: 11/Jan/23 14:51

Description

 

pyarrow.parquet.write_table used for write Parquet file
In parquet schema missed logical type for TIME, it’s just long type

PyArrow Schema

NUMBER: int64
DECIMAL: int64
NUMERIC: int64
INT: int64
FLOAT: double
VARCHAR: string
TEXT: string
CHAR: string
BOOLEAN: bool
ARR: string
VAR: string
OBJ: string
TIMESTAMP: timestamp[ns]
DATE: date64[ms]
TIME: time64[ns]
PK: int64
UUID: binary
UUID2: string
UUID3: string 

Parquet schema

{
  "type" : "record",
  "name" : "schema",
  "fields" : [ {
    "name" : "NUMBER",
    "type" : [ "null", "long" ],
    "default" : null
  }, {
    "name" : "DECIMAL",
    "type" : [ "null", "long" ],
    "default" : null
  }, {
    "name" : "NUMERIC",
    "type" : [ "null", "long" ],
    "default" : null
  }, {
    "name" : "INT",
    "type" : [ "null", "long" ],
    "default" : null
  }, {
    "name" : "FLOAT",
    "type" : [ "null", "double" ],
    "default" : null
  }, {
    "name" : "VARCHAR",
    "type" : [ "null", "string" ],
    "default" : null
  }, {
    "name" : "TEXT",
    "type" : [ "null", "string" ],
    "default" : null
  }, {
    "name" : "CHAR",
    "type" : [ "null", "string" ],
    "default" : null
  }, {
    "name" : "BOOLEAN",
    "type" : [ "null", "boolean" ],
    "default" : null
  }, {
    "name" : "ARR",
    "type" : [ "null", "string" ],
    "default" : null
  }, {
    "name" : "VAR",
    "type" : [ "null", "string" ],
    "default" : null
  }, {
    "name" : "OBJ",
    "type" : [ "null", "string" ],
    "default" : null
  }, {
    "name" : "TIMESTAMP",
    "type" : [ "null", {
      "type" : "long",
      "logicalType" : "timestamp-micros"
    } ],
    "default" : null
  }, {
    "name" : "DATE",
    "type" : [ "null", {
      "type" : "int",
      "logicalType" : "date"
    } ],
    "default" : null
  }, {
    "name" : "TIME",
    "type" : [ "null", "long" ],
    "default" : null
  }, {
    "name" : "PK",
    "type" : [ "null", "long" ],
    "default" : null
  }, {
    "name" : "UUID",
    "type" : [ "null", "bytes" ],
    "default" : null
  }, {
    "name" : "UUID2",
    "type" : [ "null", "string" ],
    "default" : null
  }, {
    "name" : "UUID3",
    "type" : [ "null", "string" ],
    "default" : null
  } ]
}