[SQOOP-2201] Sqoop2: Add possibility to read Hadoop configuration files to HFDS connector

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.99.5
• Fix Version/s: 1.99.6
• Assignee: Jarek Jarcec Cecho
• Reporter: Jarek Jarcec Cecho
• Created: 16/Mar/15 15:33
• Updated: 17/Mar/15 06:47
• Resolved: 17/Mar/15 05:45

Description

Currently the HDFS connector is not explicitly reading Hadoop configuration files. During Initialization phase it doesn't do anything, so the configuration files are not needed. During other parts of the workflow, we're explicitly casting the general Context object to Hadoop Configuration.

This is unfortunate because:

• It couples HDFS connector to Mapreduce execution engine. It will break with adding non mapreduce based execution engine.
• We can't do any HDFS specific checks in Initializer as the Hadoop Configuration object is not available there.

As a result I would like to propose breaking this coupling between HDFS connector and Mapreduce execution engine and add configuration option to HDFS Link to specify directory from which we should read the appropriate Hadoop configuration files (with reasonable defaults such as /etc/conf/hadoop).

Issue Links

• requires: SQOOP-2202 Sqoop2: Add validator to check existence of local directory on Sqoop 2 server
• requires: SQOOP-2214 Sqoop2: Add ability to easily iterate over entries stored in Context
• links to: Review board