[ORC-397] ORC should allow selectively disabling dictionary-encoding on specified columns

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.5.3, 1.6.0
• Assignee: Mithun Radhakrishnan
• Reporter: Mithun Radhakrishnan
• Created: 02/Feb/18 01:55
• Updated: 19/Nov/20 21:50
• Resolved: 04/Sep/18 18:03

Description

Just as ORC allows the choice of columns to enable bloom-filters on, it would be nice to have a way to specify which columns DICTIONARY_V2 encoding should be disabled on.

Currently, the choice of dictionary-encoding depends on the results of sampling the first row-stride within a stripe. If the user knows that a column's cardinality is bound to prevent an effective dictionary, she might choose to simply disable it on just that column, and avoid the cost of sampling in the first row-stride.

Issue Links

• is related to: SPARK-25635 Support selective direct encoding in native ORC write
• is related to: ORC-308 Add function to get subtypes by name
• is related to: ORC-299 Improve heuristics for bailing on dictionary encoding
• links to: GitHub Pull Request #304