I am trying to use pyarrow to convert CSV files to Parquet for use with Redshift Spectrum. I've got everything sorted... almost
Unfortunately, the only format they accept for timestamp columns is int96. I understand that int96 timestamps are unofficial/deprecated, but unfortunately it's what I am stuck with for integrating with, at least for the moment. I contacted Amazon support and 64-bit timestamp support has been added to the list of feature requests, but it's unclear when it will be prioritized/added/released.
In the meantime, I am thinking of adding write support for int96 columns to arrow. Would that be a welcome addition?