[IMPALA-11810] The mem estimate is incorrect in HdfsScanNode

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: Impala 4.2.0, Impala 4.1.1
• Component/s: fe
• Epic Color: ghx-label-12
• Assignee: Unassigned
• Reporter: jhkcool
• Created: 23/Dec/22 02:09
• Updated: 23/Dec/22 02:29

Description

About perInstanceMemEstimate calculate in the below method of the HdfsScanNode class:

@Override  
public void computeNodeResourceProfile(TQueryOptions queryOptions) {
...
    long avgScanRangeBytes = (long) Math.ceil(sumValues(totalBytesPerFs_) / (double) scanRangeSize);
...
}

All table data file sizes are used in the calculation of scan hdfs memory consumption, it is unreasonable. Because not all data in the table needs to be read into the memory, but only the materialized fields are read, so only the data file size occupied by the materialized field needs to be calculated.