Mike thanks for the review!!!!!
Phew been a long time since I looked at this branch!
its been changing
We have some stale jdocs that reference .setIntValue methods (they
are now .setInt)
True - thanks I will fix.
Hmm do we have byte ordering problems? Ie, if I write index on
machine with little-endian but then try to load values on
big-endian...? I think we're OK (we seem to always use
IndexOutput.writeInt, and we convert float-to-raw-int-bits using
We are ok here since we write big-endian (enforced by DataOutput) and read it back in as plain bytes. The created ByteBuffer will always use BIG_ENDIAN as the default order. I added a comment for this.
How come codecID changed from String to int on the branch?
due to DocValues I need to compare the ID to certain fields to see for what field I stored and need to open docValues. I always had to parse the given string which is kind of odd. I think its more natural to have the same datatype on FieldInfo, SegmentCodecs and eventually in the Codec#files() method. Making a string out of it is way simpler / less risky than parsing IMO.
What are oal.util.Pair and ParallelArray for?
legacy I will remove
FloatsRef should state in the jdocs that it's really slicing a
Can SortField somehow detect whether the needed field was stored
in FC vs DV and pick the right comparator accordingly...? Kind of
like how NumericField can detect whether the ints are encoded as
"plain text" or as NF? We can open a new issue for this,
This is tricky though. You can have a DV field that is indexed too so its hard to tell if we can reliably do it. If we can't make it reliable I think we should not do it at all.
It looks like we can sort by int/long/float/double pulled from DV,
but not by terms? This is fine for landing... but I think we
should open a post-landing issue to also make FieldComparators for
the Terms cases?
Yeah true. I didn't add a FieldComparator for bytes yet. I think this is post landing!
Should we rename oal.index.values.Type -> .ValueType? Just
because... it looks so generic when its imported & used as "Type"
agreed. I also think we should rename Source but I don't have a good name yet. Any idea?
Since we dynamically reserve a value to mean "unset", does that
mean there are some datasets we cannot index? Or... do we tap
into the unused bit of a long, ie the sentinel value can be
negative? But if the data set spans Long.MIN_VALUE to
Long.MAX_VALUE, what do we do...?
Again, tricky! The quick answer is yes, but we can't do that anyway since I have not normalize the range to be 0 based since PackedInts doesn't allow negative values. so the range we can store is (2^63) -1. So essentially with the current impl we can store (2^63)-2 and the max value is Long#MAX_VALUE-1. Currently there is no assert for this which is needed I think but to get around this we need to have a different impl I think or do I miss something?
I will make the changes once SVN is writeable again.