[IMPALA-8237] Enabling preads always fetches hedged reads metrics

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Backend
• Epic Color: ghx-label-4
• Assignee: Sahil Takiar
• Reporter: Sahil Takiar
• Created: 21/Feb/19 16:35
• Updated: 21/Feb/19 16:38

Description

In HdfsFileReader if preads are enabled, we assume that hedged reads are enabled as well, so whenever we close a file we make a libhdfs call to collect a few hedged read metrics from the underlying FileSystem object. However, as part of IMPALA-5212 we may want to enable preads even when hedged reads are disabled, so making the call to libhdfs to fetch hedged read metrics will be a waste.

Digging through the HDFS code, it seems the HDFS client triggers hedged reads only if dfs.client.hedged.read.threadpool.size is greater than 0. We can use the same check in HdfsFileReader to trigger the fetch of hedged read metrics. The issue is that currently libhdfs does not provide a good way of getting the value of dfs.client.hedged.read.threadpool.size, it provides a method called hdfsConfGetInt, but that method simply calls new Configuration() and fetches the value of  dfs.client.hedged.read.threadpool.size from it. The issue is that calling new Configuration simply loads the current hdfs-site.xml, core-site.xml, etc. which does not take into account the scenario where the default configuration has been modified for specific filesystem objects - e.g. using hdfsBuilder to set non-default configuration parameters (see HDFS-14301 for more details).

Issue Links

• is related to: HDFS-14301 libhdfs should expose a method to get conf parameters from a hdfsFS instance
• is related to: IMPALA-5212 consider switching to pread by default