Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13186

GenericOptionParser -libjars option validity check not always working because of bad local FS equality check

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Cannot Reproduce
    • Affects Version/s: 2.6.0, 2.7.1
    • Fix Version/s: None
    • Component/s: util
    • Labels:
      None
    • Environment:

      Occured on: CentOS 6.3 or Ubuntu 14.04, Oracle JDK 1.7.0_45 or 1.8.0_66

    • Tags:
      Job, Command Line, Local File System, libjars

      Description

      Class concerned :
      org.apache.hadoop.util.GenericOptionParser
      Method :
      public static URL[] getLibJars(final Configuration conf)
      Line :
      if (tmp.getFileSystem(conf).equals(FileSystem.getLocal(conf)))

      In this method we check if the provided jar on the command line are on a local file system, else we emit a warning log and ignore it.

      I've got the case where the two file systems retrieved (the one from the Path.getFileSystem(conf) and the one returned by FileSystem.getLocal(conf)) where two local file systems but different objects, and the equals() method, not having been implemented, defaulted to the object pointer equality, leading to my jar files not being taken into account.

      I've quickly patched it to
      tmp.getFileSystem(conf).getUri().equals(FileSystem.getLocal(conf).getUri()
      to have my application work.
      I did not dig further into determining whether the objects should have been the same, or whether the equals() method should have been implemented, but it has to be done.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              ArnaudL Arnaud Linz
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 24h
                24h
                Remaining:
                Remaining Estimate - 24h
                24h
                Logged:
                Time Spent - Not Specified
                Not Specified