[KUDU-2243] CFile Reader improvements

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.6.0
• Component/s: cfile
• Assignee: Unassigned
• Reporter: Dan Burkert
• Created: 19/Dec/17 00:12
• Updated: 25/Jan/18 19:20

Description

I've done a pretty thorough review of all the CFile reader code over the last few days in order to make a targeted bug fix, and I've got some ideas for how we can simplify it. I'd like to get others thoughts.

• To reduce confusion between CFile data blocks and FS manager blocks, I think we should change all references in code and docs of CFile data blocks to 'cblock'.

• Much of the complexity of the CFileIterator is due to it's complex public API, which requires separate Seek(idx) -> Prepare(nrows) -> Scan(output buf, predicates) calls. Additionally, the Prepare step can materialize many blocks, which then need to be put in a queue. I think all of this could be simplified by changing the API to be Seek(idx) -> Scan(nrows, output buf, predicates), and have the CFile iterator only cache the most-recently-materialized block (instead of the queue). For really big scan batches, this will change the internal scan/materialize pattern from materializing all cblocks up front then copying, to materializing and copying of cblocks being interleaved. Since in most cases cblocks are usually much bigger (256kib) than scan batches (100 cells), I think it won't actually lead to measurably different behavior.

• QueueCurrentDataBlock and ReadCurrentDataBlock should drop Current.