Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-4864

-libjars with multiple jars broken when client and cluster reside on different OSs

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Minor
    • Resolution: Fixed
    • 0.19.0
    • 0.21.0
    • filecache
    • None
    • When your hadoop job spans OSs.

    • Reviewed

    Description

      When submitting a hadoop job from Windows (Cygwin) to a Linux hadoop cluster (or vice versa), and when you specify multiple additional jar files via the -libjars flag, hadoop throws a ClassNotFoundException for any classes located in the additional jars specified via the -libjars flag.

      This is caused by the fact that hadoop uses system.getProperty("path.separator") as the delimiter in the list of jar files passed via -libjars.

      My suggested solution is to use a comma as the delimiter, rather than the path.separator.

      I realize comma is, perhaps, a poor choice for a delimiter because it is valid in filenames on both Windows and Linux, but the -libjars flag uses it as the delimiter when listing the additional required jars. So, I figured if it's already being used as a delimiter, then it's reasonable to use it internally as well.

      Attachments

        1. patch.txt
          1 kB
          Stuart White
        2. patch-4864.txt
          2 kB
          Amareshwari Sriramadasu
        3. patch-4864-1.txt
          3 kB
          Amareshwari Sriramadasu

        Activity

          People

            amareshwari Amareshwari Sriramadasu
            stwhit Stuart White
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - 1h
                1h
                Remaining:
                Remaining Estimate - 1h
                1h
                Logged:
                Time Spent - Not Specified
                Not Specified