[IMPALA-9874] Reduce or avoid I/O for pruned columns

• Type: Sub-task
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Component/s: Backend
• Labels: parquet
• Epic Color: ghx-label-11
• Assignee: Abhishek Rawat
• Reporter: Tim Armstrong
• Created: 19/Jun/20 15:54
• Updated: 06/Apr/23 07:51

Description

Skipping decoding of values may not be effective at reducing I/O in some cases, because we start the I/O in StartScans(). We don't wait for the I/O until we actually read the first data page from the column reader. So there is a race to determine whether the I/O happens in some cases.

There are a couple of things we can do here.

• The basic thing is to issue reads for the column readers in the order in which they are needed. We may be able to get this for free by ordering the column readers based on materialisation order.
• We also want to avoid issuing I/O for columns that are not needed, if predicates are highly selective. This is maybe a bit harder and avoids more trade-offs, since delaying issuing of the reads may impact scan latency.