Details
-
Improvement
-
Status: Closed
-
Minor
-
Resolution: Won't Fix
-
None
-
None
-
None
-
None
Description
This is probably a special case, but we're considering serving data from the generated sequence files to avoid having to convert to other file format.
However, using block compression means we'd have to read up to almost one mb (default) of data to find the data. Our records are so small that compressing
them using records compression increases the size of the file compared to no compression.
I'd like to make a modified version of the BlockCompressWriter that ends a block depending on features of the key appended.
There's currently no easy way of adding this without modifying SequenceFile directly.