[IMPALA-9074] Add support for zstd in ORC

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Component/s: Backend
• Epic Link: Basic ORC Support
• Target Version: Impala 4.0.0
• Epic Color: ghx-label-10
• Assignee: Quanlong Huang
• Reporter: Quanlong Huang
• Created: 21/Oct/19 13:27
• Updated: 03/Dec/21 06:16
• Resolved: 03/Dec/21 06:16

Description

The ORC lib already supports reading/writing to zstd compressed ORC files. However, I failed in a quick try in Impala:

hive> create table orc_zstd (id int, name string) stored as orc;
$ hdfs dfs -put id_name_zstd.orc hdfs://localhost:20500/test-warehouse/orc_zstd
impala-shell> invalidate metadata orc_zstd;
impala-shell> select * from orc_zstd;
ERROR: Encountered parse error in tail of ORC file hdfs://localhost:20500/test-warehouse/orc_zstd/id_name_zstd.orc: Unknown compression codec 5

The ORC file is generated by the csv-import tool: https://github.com/apache/orc/blob/rel/release-1.6.0/tools/src/CSVFileImport.cc
(Manually changing the compression from ZLIB to ZSTD in it)