I think we need to make this (RAM vs CPU tradeoff) settable by the app, at indexing time?
I think we should do that. Maybe via IWC?
I noticed that with BYTES_VAR_SORTED the memory usage varied a lot during the test execution from around 10MB to 320MB. The memory usage for BYTES_FIXED_SORTED is stable around 10MB.
yeah, the VAR_SORTED variant loads the ords in memory which is wrong though. However,
LUCENE-3507 is adressing this issue.
one thing I noticed about the patch is that you consider 0 ords as null values which is not the case in IDV. All documents do have values, there is nothing like missing values. VAR_BYTES will return a 0-length byte and FIXED_BYTES will return a fixed-length 0-byte array. Numeric variants default to 0 and 0.0 receptively.