omalley has commented on the revision "
HIVE-3874 [jira] Create a new Optimized Row Columnar file format for Hive".
| Do you want to have a simple test for
|HIVE-4015 as part of this patch ?
Since the test will fail until I change the grammer, I think it would be better to wait until they will pass.
| 1. Can you add more comments - specially, in the class/interface definitions Writer/TreeWriter/StreamFactory to name a few.
| 2. Can column statistics be made optional ? (can be a follow-up)
They are very cheap in practice, but it wouldn't be hard to disable them.
| 3. This has a lot of new code - I mean, is it possible to use some of the constructs which are already there - for eg. RedBlackTrees, RLE etc. Can you use some existing implementations instead of writing these from scratch ?
I'm a big fan of not writing new code when I can just use someone else's. That said, it is always a trade off when evaluating when to re-use versus write new that involves comparing the requirements to what the other code provides.
I'm not aware of any open source java red-black trees that work on primitives without allocating multiple objects per an entry. Do you have a suggestion?
The RLE is very specific to ORC format and it didn't seem like any of the implementations available were a good match. I'm also considering how to do a better delta and small integer encoding, but I'll do that in a follow up jira.
| Right now, the RLE is fixed. Should it be pluggable ? I mean - we can have a different scheme to store deltas.
I think that pluggable will create compatibility problems, since you won't be able to read an ORC file that was written by a different plugin.
My preferred direction is to use the ColumnEncoding to allow the Writer to pick a different encoding based on the observed data. For example, by looking at the first 100,000 values the writer should be able to decide if a dictionary or direct encoding is better. We could use the same mechanism to add additional encodings.
ql/src/java/org/apache/hadoop/hive/ql/orc/OutStream.java:136-140 There is an requirement that the codec's compress method will return false rather than take more space than the input. Given that, if the compressed is empty, we can't need the overflow.
ql/src/java/org/apache/hadoop/hive/ql/orc/OrcInputFormat.java:149-151 I've removed it.
ql/src/java/org/apache/hadoop/hive/ql/orc/WriterImpl.java:561-562 I've added the size of the dictionary to the estimate of the memory size, which should be better.
ql/src/java/org/apache/hadoop/hive/ql/orc/BitFieldReader.java:18 i managed to move the directory in the wrong place. fixed.
To: JIRA, omalley
Cc: kevinwilfong, njain