[FLINK-10601] Make user home dir consistent with Flink default filesystem

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 1.6.1
• Component/s: Deployment / YARN
• Labels: pull-request-available
• Assignee: Paul Lin
• Reporter: Paul Lin
• Created: 18/Oct/18 12:07
• Updated: 17/Dec/18 06:12
• Resolved: 14/Dec/18 03:16

Description

Currently, Flink client uses the raw default Hadoop filesystem to upload local files, and this could be problematic when using a non-default filesystem for HA or checkpointing in delegation tokens scenarios. The jobmanager only has the delegation tokens for the default FS, so it gets authentication errors when trying to connect a non-default filesystem.

So I propose to replace the default FS property in yarn configuration with the Flink filesystem property `fs.default-scheme` on the client side (AbstractYarnClusterDescriptor), to avoid this problem and also make the client behavior more configurable.

Issue Links

• relates to: FLINK-6376 when deploy flink cluster on the yarn, it is lack of hdfs delegation token.
• links to: GitHub Pull Request #6887