[HADOOP-13186] GenericOptionParser -libjars option validity check not always working because of bad local FS equality check

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Cannot Reproduce
• Affects Version/s: 2.6.0, 2.7.1
• Component/s: util

Environment

Occured on: CentOS 6.3 or Ubuntu 14.04, Oracle JDK 1.7.0_45 or 1.8.0_66

• Tags: Job, Command Line, Local File System, libjars
• Assignee: Unassigned
• Reporter: Arnaud Linz
• Created: 20/May/16 08:43
• Updated: 17/Jun/18 19:18
• Resolved: 11/Aug/17 16:20

Description

Class concerned :
org.apache.hadoop.util.GenericOptionParser
Method :
public static URL[] getLibJars(final Configuration conf)
Line :
if (tmp.getFileSystem(conf).equals(FileSystem.getLocal(conf)))

In this method we check if the provided jar on the command line are on a local file system, else we emit a warning log and ignore it.

I've got the case where the two file systems retrieved (the one from the Path.getFileSystem(conf) and the one returned by FileSystem.getLocal(conf)) where two local file systems but different objects, and the equals() method, not having been implemented, defaulted to the object pointer equality, leading to my jar files not being taken into account.

I've quickly patched it to
tmp.getFileSystem(conf).getUri().equals(FileSystem.getLocal(conf).getUri()
to have my application work.
I did not dig further into determining whether the objects should have been the same, or whether the equals() method should have been implemented, but it has to be done.