Details
-
Improvement
-
Status: Closed
-
Major
-
Resolution: Fixed
-
4.0.0
-
None
Description
Using st(reaming)ORC (hive.exec.orc.delta.streaming.optimizations.enabled=true) format has massive performance advantages when creating data-sets which will not be stored for long-term.
The format is compatible with ORC for vectorization and other features, while being cheaper to write out to filesystem.