Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
Impala 4.2.0, Impala 4.1.1
-
None
-
None
-
ghx-label-12
Description
About perInstanceMemEstimate calculate in the below method of the HdfsScanNode class:
@Override public void computeNodeResourceProfile(TQueryOptions queryOptions) { ... long avgScanRangeBytes = (long) Math.ceil(sumValues(totalBytesPerFs_) / (double) scanRangeSize); ... }
All table data file sizes are used in the calculation of scan hdfs memory consumption, it is unreasonable. Because not all data in the table needs to be read into the memory, but only the materialized fields are read, so only the data file size occupied by the materialized field needs to be calculated.