Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-10820

Throw an exception in GenericOptionsParser when passed an empty Path

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Minor
    • Resolution: Fixed
    • 2.2.0
    • 2.6.0
    • None
    • None

    Description

      An empty token (e.g. "a.jar,,b.jar") in the -libjars option causes the current working directory to be recursively localized.

      Here's an example of this in action (using Hadoop 2.2.0):

      # create a temp directory and touch three JAR files
      mkdir -p tmp/path && cd tmp && touch a.jar b.jar c.jar path/d.jar
      
      # Run an example job only specifying two of the JARs.
      # Include an empty entry in libjars.
      hadoop jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar pi -libjars a.jar,,c.jar 2 1000000000
      
      # As the job is running examine the localized directory in HDFS.
      # Notice that not only are the two JAR's specified in libjars copied,
      # but in addition the contents of the working directory are also recursively copied.
      $ hadoop fs -lsr /tmp/hadoop-yarn/staging/aholmes/.staging/job_1404752711144_0018/libjars
      /tmp/hadoop-yarn/staging/aholmes/.staging/job_1404752711144_0018/libjars/a.jar
      /tmp/hadoop-yarn/staging/aholmes/.staging/job_1404752711144_0018/libjars/c.jar
      /tmp/hadoop-yarn/staging/aholmes/.staging/job_1404752711144_0018/libjars/tmp
      /tmp/hadoop-yarn/staging/aholmes/.staging/job_1404752711144_0018/libjars/tmp/a.jar
      /tmp/hadoop-yarn/staging/aholmes/.staging/job_1404752711144_0018/libjars/tmp/b.jar
      /tmp/hadoop-yarn/staging/aholmes/.staging/job_1404752711144_0018/libjars/tmp/c.jar
      /tmp/hadoop-yarn/staging/aholmes/.staging/job_1404752711144_0018/libjars/tmp/path
      /tmp/hadoop-yarn/staging/aholmes/.staging/job_1404752711144_0018/libjars/tmp/path/d.jar
      

      Attachments

        1. HADOOP-10820-3.patch
          5 kB
          Zhihai Xu
        2. HADOOP-10820-2.patch
          5 kB
          Zhihai Xu
        3. HADOOP-10820-1.patch
          7 kB
          Alex Holmes
        4. HADOOP-10820.patch
          3 kB
          Alex Holmes

        Activity

          People

            zxu Zhihai Xu
            alex.holmes Alex Holmes
            Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: