[HADOOP-15070] add test to verify FileSystem and paths differentiate on user info

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.8.2
• Fix Version/s: 3.1.0
• Component/s: fs, test
• Target Version/s: 3.1.0
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 27/Nov/17 11:14
• Updated: 19/Feb/18 21:04
• Resolved: 19/Feb/18 20:46

Description

Add a test to verify that userinfo data is (correctly) used to differentiate the entries in the FS cache, so are treated as different filesystems.

• This is criticalk for wasb, which uses the username to identify the container, in a path like wasb:[e-mail]. This works in Hadoop, but SPARK-22587 shows that it may not be followed everywhere (and given there's no documentation, who can fault them?)
• AbstractFileSystem.checkPath looks suspiciously like it's path validation just checks host, not authority. That needs a test too.
• And we should cut the @LimitedPrivate(HDFS, Mapreduce) from Path.makeQualified. If MR needs it, it should be considered open to all apps using the Hadoop APIs. Until I looked at the code I thought it was...

Issue Links

• relates to: SPARK-22587 Spark job fails if fs.defaultFS and application jar are different url