[BEAM-8399] Python HDFS implementation should support filenames of the format "hdfs://namenodehost/parent/child"

• Type: Improvement
• Status: Triage Needed
• Priority: P2
• Resolution: Fixed
• Fix Version/s: 2.20.0
• Component/s: sdk-py-core
• Assignee: Udi Meiri
• Reporter: Chamikara Madhusanka Jayalath
• Created: 14/Oct/19 22:00
• Updated: 13/Apr/23 10:59
• Resolved: 14/Feb/20 23:53

Description

"hdfs://namenodehost/parent/child" and "/parent/child" seems to be the correct filename formats for HDFS based on [1] but we currently support format "hdfs://parent/child".

To not break existing users, we have to either (1) somehow support both versions by default (based on [2] seems like HDFS does not allow colons in file path so this might be possible) (2) make  "hdfs://namenodehost/parent/child" optional for now and change it to default after few versions.

We should also make sure that Beam Java and Python HDFS file-system implementations are consistent in this regard.

 

[1]https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/FileSystemShell.html

[2] https://issues.apache.org/jira/browse/HDFS-13

 

cc: udim

Issue Links

• links to: GitHub Pull Request #10223