[HUDI-1585] Reduce the coupling of hadoop.

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Component/s: Common Core, flink
• Assignee: Unassigned
• Reporter: Moses
• Created: 05/Feb/21 09:57
• Updated: 12/Nov/21 03:16
• Resolved: 12/Nov/21 03:16

Description

In my oppions, the hadoop configuration totally read from the actual environment. For example, the fs.hdfs.impl may be org.apache.hadoop.fs.viewfs.ViewFileSyste but not org.apache.hadoop.hdfs.DistributedFileSystem. Now, the implemtation was forced to be set with org.apache.hadoop.hdfs.DistributedFileSystem, which is not a graceful solution.
Also, the hadoop dependency scope level should be 'test'.