[ORC-1376] [C++] Support schema evolution

• Type: New Feature
• Status: Reopened
• Priority: Major
• Resolution: Unresolved
• Component/s: C++
• Assignee: Yuanping Wu
• Reporter: Gang Wu
• Created: 21/Feb/23 03:46
• Updated: 29/Jun/23 01:47

Description

The Java ORC reader supports schema evolution by comparing readSchema with fileSchema [1] and creates a set of ConvertTreeReaders [2] to do the conversion. The conversion happens when reading columnar data into vector and evaluating expressions for predicate push down. The C++ reader does not support it yet.

The type conversion on the Java side originates from Apache Hive. We should define what conversions are supposed to implement on the C++ side.

[1] https://github.com/apache/orc/blob/main/java/core/src/java/org/apache/orc/impl/SchemaEvolution.java
[2] https://github.com/apache/orc/blob/main/java/core/src/java/org/apache/orc/impl/ConvertTreeReaderFactory.java#L60:14

Issue Links

• contains: ORC-1391 [C++] Support schema evolution from timestamp to numeric/decimal/string group
• contains: ORC-1392 [C++] Support schema evolution with date and binary
• contains: ORC-1385 [C++] Support schema evolution from numeric to numeric
• contains: ORC-1386 [C++] Support schema evolution from primtive to string group/decimal/timestamp
• contains: ORC-1388 [C++] Support schema evolution from decimal to timestamp/string group
• contains: ORC-1389 [C++] Support schema evolution from string group to numeric/string group
• contains: ORC-1390 [C++] Support schema evolution from string group to decimal/timestamp
• contains: ORC-1387 [C++] Support schema evolution from decimal to numeric/decimal
• links to: GitHub Pull Request #1427
• links to: GitHub Pull Request #1441