Details
-
Bug
-
Status: Resolved
-
Minor
-
Resolution: Fixed
-
Impala 3.0
-
None
-
None
-
ghx-label-4
Description
The HdfsTable.getFilesSample function takes a list of input partitions to sample files from, but then, when allocating an array to sample into, sizes that array based on the total file count across all partitions. This is an unnecessarily large array, which is expensive to allocate (may cause full GC when the heap is fragmented). The code claims this to be an optimization:
// Use max size to avoid looping over inputParts for the exact size.
...but I think the loop over inputParts is likely to be trivial here since we'll loop over them anyway later in the function and thus will already be pulled into CPU cache, etc. This is also necessary for fine-grained metadata loading in the impalad – for a large table with many partitions, we don't want to load the file lists of all partitions just to tablesample from one partition.