[DRILL-6521] When using Hadoop configs, allow dfs connection to be unset

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 1.14.0
• Assignee: Unassigned
• Reporter: Paul Rogers
• Created: 21/Jun/18 04:07
• Updated: 22/Feb/19 07:00

Description

Drill usually works stand-alone. In a Hadoop HDFS environment, the docs. say to set the HDFS configuration in the storage plugin config.

In a MapR installation, MaprFS settings are applied automatically.

However, when Drill runs on an existing HDFS cluster, one must often provide more than the simple HDFS URL. Particularly in a secure cluster, other configuration settings are also needed. At present, these must be copied out of the HDFS config files into the Drill storage plugin config, and the two must be updated in tandem. Clearly less than ideal.

Drill does allow the user to add Hadoop configs to the class path. (Though, it looks like in recent releases the previous HADOOP_CONF setting has been removed.) The user can edit drill-env.sh to add the Hadoop class path to EXTN_CLASSPATH. (But see DRILL-6520.)

This is all good, but Drill still requires that the "dfs" storage plugin config contain a connection. Omit the connection and we get:

Please retry: Error while creating/ updating storage : The value of property fs.defaultFS must not be null

Would expect to be able to omit this property if the value is provided by the standard Hadoop core-site.xml file (and to do so without Drill crashing, per DRILL-6520).

Issue Links

• relates to: DRILL-6540 Upgrade to HADOOP-3.0 libraries