[HDFS-14301] libhdfs should expose a method to get conf parameters from a hdfsFS instance

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: hdfs-client, libhdfs, native
• Assignee: Sahil Takiar
• Reporter: Sahil Takiar
• Created: 20/Feb/19 15:06
• Updated: 21/Feb/19 16:36

Description

libhdfs currently exposes a few methods for getting the values of a configuration parameters: hdfsConfGetStr and hdfsConfGetInt. The issue is that in hdfs.c the implementation of these methods simply calls new Configuration() and fetches values using get. The issue is that calling new Configuration simply loads the current hdfs-site.xml, core-site.xml, etc. which does not take into account the scenario where the default configuration has been modified for specific filesystem instances. For example, the hdfsBuilder exposes a hdfsBuilderConfSetStr method that allows setting non-default configuration parameters.

This could lead to issues such as:

struct hdfsBuilder *bld;
bld = hdfsNewBuilder();
hdfsBuilderSetForceNewInstance(bld);
hdfsBuilderConfSetStr(bld, "hello", "world");
hdfs = hdfsBuilderConnect(bld);

char* value = NULL;
hdfsConfGetStr("hello", &value);
// Value is NULL!

This JIRA proposes adding a new set of methods to libhdfs that take in a hdfsFS object and get the value of the key from the hdfsFS object rather than using new Configuration.

Issue Links

• relates to: IMPALA-8237 Enabling preads always fetches hedged reads metrics