Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-alpha1
    • Component/s: util
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      It is now possible to specify multiple jar files for the libjars argument using a wildcard. For example, you can specify "-libjars 'libs/*'" as a shorthand for all jars in the libs directory.

      Description

      There is a problem when a user job adds too many dependency jars in their command line. The HADOOP_CLASSPATH part can be addressed, including using wildcards (*). But the same cannot be done with the -libjars argument. Today it takes only fully specified file paths.

      We may want to consider supporting wildcards as a way to help users in this situation. The idea is to handle it the same way the JVM does it: * expands to the list of jars in that directory. It does not traverse into any child directory.

      Also, it probably would be a good idea to do it only for libjars (i.e. don't do it for -files and -archives).

      1. HADOOP-12747.01.patch
        21 kB
        Sangjin Lee
      2. HADOOP-12747.02.patch
        24 kB
        Sangjin Lee
      3. HADOOP-12747.03.patch
        24 kB
        Sangjin Lee
      4. HADOOP-12747.04.patch
        25 kB
        Sangjin Lee
      5. HADOOP-12747.05.patch
        25 kB
        Sangjin Lee
      6. HADOOP-12747.06.patch
        26 kB
        Sangjin Lee
      7. HADOOP-12747.07.patch
        27 kB
        Sangjin Lee

        Issue Links

          Activity

          Hide
          cnauroth Chris Nauroth added a comment -

          There is some existing wildcard matching code in FileUtil#createJarWithClassPath that might be useful for refactoring or just inspiration.

          Show
          cnauroth Chris Nauroth added a comment - There is some existing wildcard matching code in FileUtil#createJarWithClassPath that might be useful for refactoring or just inspiration.
          Hide
          sjlee0 Sangjin Lee added a comment -

          Thanks for that Chris Nauroth! There is also ApplicationClassLoader#constructUrlsFromClasspath that does a similar thing. Perhaps we can refactor this logic into a common place.

          One interesting thing to consider is whether we want to limit this wildcard expansion to local paths or remote filesystems as well. I think both FileUtil and ApplicationClassLoader consider only local paths. Although supporting expanding for remote paths might make this enhancement more consistent, it might increase the scope somewhat. Thoughts?

          Show
          sjlee0 Sangjin Lee added a comment - Thanks for that Chris Nauroth ! There is also ApplicationClassLoader#constructUrlsFromClasspath that does a similar thing. Perhaps we can refactor this logic into a common place. One interesting thing to consider is whether we want to limit this wildcard expansion to local paths or remote filesystems as well. I think both FileUtil and ApplicationClassLoader consider only local paths. Although supporting expanding for remote paths might make this enhancement more consistent, it might increase the scope somewhat. Thoughts?
          Hide
          cnauroth Chris Nauroth added a comment -

          I'm pretty sure -libjars only accepts paths on the local file system. (i.e. HADOOP-7112) Since that's the case, I'd suggest keeping the scope of this issue limited to wildcard support on the local file system.

          Show
          cnauroth Chris Nauroth added a comment - I'm pretty sure -libjars only accepts paths on the local file system. (i.e. HADOOP-7112 ) Since that's the case, I'd suggest keeping the scope of this issue limited to wildcard support on the local file system.
          Hide
          sjlee0 Sangjin Lee added a comment -

          Posted patch v.1. I tested it with a pseudo-distributed cluster.

          It takes a pretty minimal approach. When it sees a wildcard in the libjars option value, it replaces it with jars in that directory and sets it onto tmpjars.

          I refactored FileUtil, ApplicationClassLoader, and GenericOptionsParser to use the common implementation (the one that was in FileUtil).

          I also updated TestGenericOptionsParser to use JUnit 4.

          I would greatly appreciate your review. Thanks!

          Show
          sjlee0 Sangjin Lee added a comment - Posted patch v.1. I tested it with a pseudo-distributed cluster. It takes a pretty minimal approach. When it sees a wildcard in the libjars option value, it replaces it with jars in that directory and sets it onto tmpjars. I refactored FileUtil , ApplicationClassLoader , and GenericOptionsParser to use the common implementation (the one that was in FileUtil ). I also updated TestGenericOptionsParser to use JUnit 4. I would greatly appreciate your review. Thanks!
          Hide
          jira.shegalov Gera Shegalov added a comment -

          FYI, I've been working around this problem by using GOP options

           
           -Dmapreduce.application.classpath='libs/*' -files libs
          

          for the case where all the dependencies are in the directory ./libs . You can consider making this official behavior of libjars by allowing directories in libjars.

          Show
          jira.shegalov Gera Shegalov added a comment - FYI, I've been working around this problem by using GOP options -Dmapreduce.application.classpath='libs/*' -files libs for the case where all the dependencies are in the directory ./libs . You can consider making this official behavior of libjars by allowing directories in libjars.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 10m 8s trunk passed
          +1 compile 13m 39s trunk passed with JDK v1.8.0_66
          +1 compile 12m 42s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 35s trunk passed
          +1 mvnsite 1m 44s trunk passed
          +1 mvneclipse 0m 22s trunk passed
          +1 findbugs 3m 10s trunk passed
          +1 javadoc 1m 42s trunk passed with JDK v1.8.0_66
          +1 javadoc 1m 55s trunk passed with JDK v1.7.0_91
          +1 mvninstall 1m 7s the patch passed
          +1 compile 17m 48s the patch passed with JDK v1.8.0_66
          +1 javac 17m 48s the patch passed
          +1 compile 13m 12s the patch passed with JDK v1.7.0_91
          +1 javac 13m 12s the patch passed
          -1 checkstyle 0m 33s hadoop-common-project/hadoop-common: patch generated 2 new + 105 unchanged - 5 fixed = 107 total (was 110)
          +1 mvnsite 1m 43s the patch passed
          +1 mvneclipse 0m 21s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 3m 42s the patch passed
          +1 javadoc 1m 53s the patch passed with JDK v1.8.0_66
          +1 javadoc 1m 51s the patch passed with JDK v1.7.0_91
          -1 unit 14m 36s hadoop-common in the patch failed with JDK v1.8.0_66.
          -1 unit 12m 4s hadoop-common in the patch failed with JDK v1.7.0_91.
          +1 asflicense 0m 35s Patch does not generate ASF License warnings.
          117m 22s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.ipc.TestRPCWaitForProxy
            hadoop.fs.shell.find.TestIname
            hadoop.ha.TestZKFailoverController
            hadoop.fs.shell.find.TestPrint0
            hadoop.security.ssl.TestReloadingX509TrustManager
            hadoop.fs.shell.find.TestPrint
            hadoop.fs.shell.find.TestName
            hadoop.io.compress.TestCodecPool
          JDK v1.7.0_91 Failed junit tests hadoop.fs.shell.find.TestPrint0
            hadoop.fs.shell.find.TestPrint
            hadoop.fs.shell.find.TestName
            hadoop.ipc.TestProtoBufRpc



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12785093/HADOOP-12747.01.patch
          JIRA Issue HADOOP-12747
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 8a163b779d84 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / f67149a
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8492/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8492/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8492/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8492/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8492/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8492/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8492/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 10m 8s trunk passed +1 compile 13m 39s trunk passed with JDK v1.8.0_66 +1 compile 12m 42s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 35s trunk passed +1 mvnsite 1m 44s trunk passed +1 mvneclipse 0m 22s trunk passed +1 findbugs 3m 10s trunk passed +1 javadoc 1m 42s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 55s trunk passed with JDK v1.7.0_91 +1 mvninstall 1m 7s the patch passed +1 compile 17m 48s the patch passed with JDK v1.8.0_66 +1 javac 17m 48s the patch passed +1 compile 13m 12s the patch passed with JDK v1.7.0_91 +1 javac 13m 12s the patch passed -1 checkstyle 0m 33s hadoop-common-project/hadoop-common: patch generated 2 new + 105 unchanged - 5 fixed = 107 total (was 110) +1 mvnsite 1m 43s the patch passed +1 mvneclipse 0m 21s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 3m 42s the patch passed +1 javadoc 1m 53s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 51s the patch passed with JDK v1.7.0_91 -1 unit 14m 36s hadoop-common in the patch failed with JDK v1.8.0_66. -1 unit 12m 4s hadoop-common in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 35s Patch does not generate ASF License warnings. 117m 22s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.ipc.TestRPCWaitForProxy   hadoop.fs.shell.find.TestIname   hadoop.ha.TestZKFailoverController   hadoop.fs.shell.find.TestPrint0   hadoop.security.ssl.TestReloadingX509TrustManager   hadoop.fs.shell.find.TestPrint   hadoop.fs.shell.find.TestName   hadoop.io.compress.TestCodecPool JDK v1.7.0_91 Failed junit tests hadoop.fs.shell.find.TestPrint0   hadoop.fs.shell.find.TestPrint   hadoop.fs.shell.find.TestName   hadoop.ipc.TestProtoBufRpc Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12785093/HADOOP-12747.01.patch JIRA Issue HADOOP-12747 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 8a163b779d84 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f67149a Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8492/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8492/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8492/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8492/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8492/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8492/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8492/console This message was automatically generated.
          Hide
          sjlee0 Sangjin Lee added a comment -

          Posted patch v.2.

          Fixed the checkstyle issues. I believe the unit test failures are unrelated. I am unable to reproduce them locally, and other pre-commit runs seem to see them sporadically.

          Show
          sjlee0 Sangjin Lee added a comment - Posted patch v.2. Fixed the checkstyle issues. I believe the unit test failures are unrelated. I am unable to reproduce them locally, and other pre-commit runs seem to see them sporadically.
          Hide
          sjlee0 Sangjin Lee added a comment -

          Yes, that's what the patch accomplishes. You can now do

          hadoop jar ... -libjars 'libs/*' ...
          

          What is "GOP options" by the way?

          Show
          sjlee0 Sangjin Lee added a comment - Yes, that's what the patch accomplishes. You can now do hadoop jar ... -libjars 'libs/*' ... What is "GOP options" by the way?
          Hide
          cnauroth Chris Nauroth added a comment -

          What is "GOP options" by the way?

          hadoop jar loud-angry-debate.jar --participants=donald-trump,ted-cruz,chris-christie...

          Show
          cnauroth Chris Nauroth added a comment - What is "GOP options" by the way? hadoop jar loud-angry-debate.jar --participants=donald-trump,ted-cruz,chris-christie...
          Hide
          jira.shegalov Gera Shegalov added a comment -

          GOP=Generic Options Parser but there is room for Chris Nauroth's interpretation.

          Your approach touches more code I would think, and keeps the annoying problem that there are tons of entries under cache.files from the same dir. Thus going from fat jar to bundle results in fat conf so to speak.

          Show
          jira.shegalov Gera Shegalov added a comment - GOP=Generic Options Parser but there is room for Chris Nauroth 's interpretation. Your approach touches more code I would think, and keeps the annoying problem that there are tons of entries under cache.files from the same dir. Thus going from fat jar to bundle results in fat conf so to speak.
          Hide
          jira.shegalov Gera Shegalov added a comment -

          What I suggest already works now, without bloating configuration. So we could limit the change to make it the official behavior if an entry libjars is a dir.

          Show
          jira.shegalov Gera Shegalov added a comment - What I suggest already works now, without bloating configuration. So we could limit the change to make it the official behavior if an entry libjars is a dir.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 6m 42s trunk passed
          +1 compile 6m 11s trunk passed with JDK v1.8.0_66
          +1 compile 6m 54s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 21s trunk passed
          +1 mvnsite 1m 3s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 46s trunk passed
          +1 javadoc 0m 50s trunk passed with JDK v1.8.0_66
          +1 javadoc 1m 3s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 41s the patch passed
          +1 compile 6m 25s the patch passed with JDK v1.8.0_66
          +1 javac 6m 25s the patch passed
          +1 compile 7m 21s the patch passed with JDK v1.7.0_91
          +1 javac 7m 21s the patch passed
          +1 checkstyle 0m 20s hadoop-common-project/hadoop-common: patch generated 0 new + 103 unchanged - 7 fixed = 103 total (was 110)
          +1 mvnsite 1m 2s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 7s the patch passed
          +1 javadoc 0m 54s the patch passed with JDK v1.8.0_66
          +1 javadoc 1m 10s the patch passed with JDK v1.7.0_91
          +1 unit 7m 23s hadoop-common in the patch passed with JDK v1.8.0_66.
          +1 unit 7m 15s hadoop-common in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 21s Patch does not generate ASF License warnings.
          61m 32s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12785204/HADOOP-12747.02.patch
          JIRA Issue HADOOP-12747
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 7074e5ba85b0 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / c9a09d6
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8494/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Max memory used 77MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8494/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 6m 42s trunk passed +1 compile 6m 11s trunk passed with JDK v1.8.0_66 +1 compile 6m 54s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 21s trunk passed +1 mvnsite 1m 3s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 46s trunk passed +1 javadoc 0m 50s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 3s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 41s the patch passed +1 compile 6m 25s the patch passed with JDK v1.8.0_66 +1 javac 6m 25s the patch passed +1 compile 7m 21s the patch passed with JDK v1.7.0_91 +1 javac 7m 21s the patch passed +1 checkstyle 0m 20s hadoop-common-project/hadoop-common: patch generated 0 new + 103 unchanged - 7 fixed = 103 total (was 110) +1 mvnsite 1m 2s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 7s the patch passed +1 javadoc 0m 54s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 10s the patch passed with JDK v1.7.0_91 +1 unit 7m 23s hadoop-common in the patch passed with JDK v1.8.0_66. +1 unit 7m 15s hadoop-common in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 21s Patch does not generate ASF License warnings. 61m 32s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12785204/HADOOP-12747.02.patch JIRA Issue HADOOP-12747 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 7074e5ba85b0 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c9a09d6 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8494/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Max memory used 77MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8494/console This message was automatically generated.
          Hide
          sjlee0 Sangjin Lee added a comment -

          Thanks Gera Shegalov for your suggestion. You brought up some important points and suggestions we need to reconcile.

          It is true that today one can pass in a directory for -files and also for -libjars. In case of MR, the entire directory (including all files and directories recursively) does get copied over and localized to nodes. For libjars, however, as you observed, the classpath basically doesn't work if you meant it as a list of jars as it simply references the directory. On the other hand, if you meant it as a real directory root (consisting of class files), it still works correctly.

          When it comes to classpaths (after which libjars is modeled), directory and directory/* are different as you're undoubtedly aware. directory/* is specifically interpreted as the list of jars in that directory by the JVM. IMO it would be good to maintain that definition for libjars. That would lead to a consistent expectation.

          Also, I learned of this interesting nugget while looking at GenericOptionsParser: the value of libjars is added to the client classpath:

                //setting libjars in client classpath
                URL[] libjars = getLibJars(conf);
                if(libjars!=null && libjars.length>0) {
                  conf.setClassLoader(new URLClassLoader(libjars, conf.getClassLoader()));
                  Thread.currentThread().setContextClassLoader(
                      new URLClassLoader(libjars, 
                          Thread.currentThread().getContextClassLoader()));
                }
          

          Thus, if we allow the wildcard, it will need to be expanded in GenericOptionsParser before this point.

          On a related note, there is the matter of the shared cache (YARN-1492). For the shared cache to work correctly for a directory, the shared cache client needs to negotiate with the shared cache manager on an individual file basis (some files in the directory may be present in the shared cache; some may not). So a client-side expansion (at some point) is likely needed for the shared cache. We'll need to ensure the mapreduce portion of the shared cache work (MAPREDUCE-5951) handles directories correctly (cc Chris Trezzo).

          If needed, I can spell out a little more how directory and directory/* should be interpreted and used for libjars in comments/javadoc/documentation. Let me know. Thanks!

          Show
          sjlee0 Sangjin Lee added a comment - Thanks Gera Shegalov for your suggestion. You brought up some important points and suggestions we need to reconcile. It is true that today one can pass in a directory for -files and also for -libjars. In case of MR, the entire directory (including all files and directories recursively) does get copied over and localized to nodes. For libjars, however, as you observed, the classpath basically doesn't work if you meant it as a list of jars as it simply references the directory. On the other hand, if you meant it as a real directory root (consisting of class files), it still works correctly. When it comes to classpaths (after which libjars is modeled), directory and directory/* are different as you're undoubtedly aware. directory/* is specifically interpreted as the list of jars in that directory by the JVM. IMO it would be good to maintain that definition for libjars. That would lead to a consistent expectation. Also, I learned of this interesting nugget while looking at GenericOptionsParser : the value of libjars is added to the client classpath: //setting libjars in client classpath URL[] libjars = getLibJars(conf); if (libjars!= null && libjars.length>0) { conf.setClassLoader( new URLClassLoader(libjars, conf.getClassLoader())); Thread .currentThread().setContextClassLoader( new URLClassLoader(libjars, Thread .currentThread().getContextClassLoader())); } Thus, if we allow the wildcard, it will need to be expanded in GenericOptionsParser before this point. On a related note, there is the matter of the shared cache ( YARN-1492 ). For the shared cache to work correctly for a directory, the shared cache client needs to negotiate with the shared cache manager on an individual file basis (some files in the directory may be present in the shared cache; some may not). So a client-side expansion (at some point) is likely needed for the shared cache. We'll need to ensure the mapreduce portion of the shared cache work ( MAPREDUCE-5951 ) handles directories correctly (cc Chris Trezzo ). If needed, I can spell out a little more how directory and directory/* should be interpreted and used for libjars in comments/javadoc/documentation. Let me know. Thanks!
          Hide
          jira.shegalov Gera Shegalov added a comment -

          It is true that today one can pass in a directory for -files and also for -libjars. In case of MR, the entire directory (including all files and directories recursively) does get copied over and localized to nodes. For libjars, however, as you observed, the classpath basically doesn't work if you meant it as a list of jars as it simply references the directory. On the other hand, if you meant it as a real directory root (consisting of class files), it still works correctly.

          My solution (tested many times with production pipelines I own) using dir uploads is obviously geared to the former case and not the case of exploded archive as you can see from the value of application classpath. was just suggesting to automate it via this property or mapreduce.job.classpath.files.

          When it comes to classpaths (after which libjars is modeled), directory and directory/* are different as you're undoubtedly aware. directory/* is specifically interpreted as the list of jars in that directory by the JVM. IMO it would be good to maintain that definition for libjars. That would lead to a consistent expectation.

          I don't know whether libjars (<comma separated list of jars>) is modeled after CLASSPATH. But I think there should be a separation of concerns: syntax vs how it's implemented. In the end, I am saying let us not bloat configuration regardless whether it's 'libs/*' or 'libs/'.

          Also, I learned of this interesting nugget while looking at GenericOptionsParser: the value of libjars is added to the client classpath:

          This is the code we had discussed with Ian O Connell. This a very brittle code because you have no control when frameworks start using GOP. E.g., Scalding needs scala before it comes to GOP. Once you start asking people to put more stuff on HADOOP_CLASSPATH to bootstrap anyways why do this in libjars, as well? With Pants we don't need it at all becuase the jar manifest already includes all jars on the classpath see MAPREDUCE-6128. Maybe we should deprecate the client-side effect of libjars and not try to add more in this JIRA.

          Regarding YARN-1492, since it's a recent feature, it should accommodate for directory uploads anyways. It can negotiate at the directory level (e.g., recursive checksum of files/dirs). But that's not a subject for this JIRA.

          Show
          jira.shegalov Gera Shegalov added a comment - It is true that today one can pass in a directory for -files and also for -libjars. In case of MR, the entire directory (including all files and directories recursively) does get copied over and localized to nodes. For libjars, however, as you observed, the classpath basically doesn't work if you meant it as a list of jars as it simply references the directory. On the other hand, if you meant it as a real directory root (consisting of class files), it still works correctly. My solution (tested many times with production pipelines I own) using dir uploads is obviously geared to the former case and not the case of exploded archive as you can see from the value of application classpath. was just suggesting to automate it via this property or mapreduce.job.classpath.files. When it comes to classpaths (after which libjars is modeled), directory and directory/* are different as you're undoubtedly aware. directory/* is specifically interpreted as the list of jars in that directory by the JVM. IMO it would be good to maintain that definition for libjars. That would lead to a consistent expectation. I don't know whether libjars (<comma separated list of jars>) is modeled after CLASSPATH. But I think there should be a separation of concerns: syntax vs how it's implemented. In the end, I am saying let us not bloat configuration regardless whether it's 'libs/*' or 'libs/'. Also, I learned of this interesting nugget while looking at GenericOptionsParser: the value of libjars is added to the client classpath: This is the code we had discussed with Ian O Connell . This a very brittle code because you have no control when frameworks start using GOP. E.g., Scalding needs scala before it comes to GOP. Once you start asking people to put more stuff on HADOOP_CLASSPATH to bootstrap anyways why do this in libjars, as well? With Pants we don't need it at all becuase the jar manifest already includes all jars on the classpath see MAPREDUCE-6128 . Maybe we should deprecate the client-side effect of libjars and not try to add more in this JIRA. Regarding YARN-1492 , since it's a recent feature, it should accommodate for directory uploads anyways. It can negotiate at the directory level (e.g., recursive checksum of files/dirs). But that's not a subject for this JIRA.
          Hide
          sjlee0 Sangjin Lee added a comment -

          My solution (tested many times with production pipelines I own) using dir uploads is obviously geared to the former case and not the case of exploded archive as you can see from the value of application classpath. was just suggesting to automate it via this property or mapreduce.job.classpath.files.

          OK, could you elaborate a little on your proposal? Currently directories are already allowed in libjars. It's that the classpath on the task side is added as the directory name (i.e. directory) and not as a set of jars (i.e. directory/*). Therefore jars are not recognized as part of the classpath. Does the proposal then entail adding the classpath as directory/* instead of directory? Strictly speaking, it would be a re-interpretation (or a change of behavior) for a directory entry for libjars. Today, if you add a directory of classfiles to libjars, that works correctly. I can imagine a scenario where someone submits a job out of a dev environment and includes a source build directory as a libjars argument. Again, I don't know whether it is an advertised feature/behavior, but at least I can see that working today, and if we change it this way we would re-interpret that to mean something else.

          Also, what should be uploaded and localized? Should everything be uploaded or only jars be uploaded? Note that anything other than jars would not be used for the classpath.

          I don't know whether libjars (<comma separated list of jars>) is modeled after CLASSPATH. But I think there should be a separation of concerns: syntax vs how it's implemented.

          I agree completely. What I was arguing for is that it would be helpful for users if we adopt a consistent syntax between classpath and libjars. This patch is just one way of implementing that syntax. It could also be implemented by accepting directory/* as a directory, uploading only jar files to hdfs, and adding directory/* to the classpath. Again, in that case, IMO we still need to leave directory as libjars argument alone as it works today.

          Show
          sjlee0 Sangjin Lee added a comment - My solution (tested many times with production pipelines I own) using dir uploads is obviously geared to the former case and not the case of exploded archive as you can see from the value of application classpath. was just suggesting to automate it via this property or mapreduce.job.classpath.files. OK, could you elaborate a little on your proposal? Currently directories are already allowed in libjars. It's that the classpath on the task side is added as the directory name (i.e. directory ) and not as a set of jars (i.e. directory/* ). Therefore jars are not recognized as part of the classpath. Does the proposal then entail adding the classpath as directory/* instead of directory ? Strictly speaking, it would be a re-interpretation (or a change of behavior) for a directory entry for libjars. Today, if you add a directory of classfiles to libjars, that works correctly. I can imagine a scenario where someone submits a job out of a dev environment and includes a source build directory as a libjars argument. Again, I don't know whether it is an advertised feature/behavior, but at least I can see that working today, and if we change it this way we would re-interpret that to mean something else. Also, what should be uploaded and localized? Should everything be uploaded or only jars be uploaded? Note that anything other than jars would not be used for the classpath. I don't know whether libjars (<comma separated list of jars>) is modeled after CLASSPATH. But I think there should be a separation of concerns: syntax vs how it's implemented. I agree completely. What I was arguing for is that it would be helpful for users if we adopt a consistent syntax between classpath and libjars. This patch is just one way of implementing that syntax. It could also be implemented by accepting directory/* as a directory, uploading only jar files to hdfs, and adding directory/* to the classpath. Again, in that case, IMO we still need to leave directory as libjars argument alone as it works today.
          Hide
          jira.shegalov Gera Shegalov added a comment -

          My proposal: for each directory dir encountered in the comma-separated libjars keep the existing behavior but also add support for jars in terms of task classpath:

          • upload/localize dir as today
          • add dir to the task classpath as today especially since you want to keep supporting uncompressed classes.
          • and dir/* to the task container classpath.
          Show
          jira.shegalov Gera Shegalov added a comment - My proposal: for each directory dir encountered in the comma-separated libjars keep the existing behavior but also add support for jars in terms of task classpath: upload/localize dir as today add dir to the task classpath as today especially since you want to keep supporting uncompressed classes. and dir/* to the task container classpath.
          Hide
          sjlee0 Sangjin Lee added a comment -

          Thanks for the clarification Gera Shegalov.

          I can summarize the pros and cons of both approaches as follows.
          (1) expand wildcard at GenericOptionsParser

          • pro: provide a consistent libjars syntax with classpaths
          • pro: no impact on the existing behavior
          • pro: only needed files (i.e. jars) are uploaded to hdfs and localized
          • con: increases the size of the configuration

          (2) add the wildcard notation on the task classpath

          • pro: smallest amount of changes; leverages existing functionality of accepting and uploading a directory
          • pro: configuration stays slim
          • con: would upload and localize files unnecessarily if only the jars were supposed to be used
          • con: need to re-interpret or deprecate (minor) behavior, such as adding libjar entries to the client classpath and allowing directories as a set of classfiles

          In terms of the amount of changes, I believe the changes for approach (1) are still quite small. The only impactful change is expanding the wildcard in GenericOptionsParser. All other changes are refactoring to use a common implementation for that expansion and unit test changes.

          My proposal would be to proceed with approach (1). IMO there is a lot of value in having libjars consistent with the classpath syntax. And if we can provide that support with an equally small change, I don't see a reason not to do it.

          I'd like to hear your thoughts and others too!

          Show
          sjlee0 Sangjin Lee added a comment - Thanks for the clarification Gera Shegalov . I can summarize the pros and cons of both approaches as follows. (1) expand wildcard at GenericOptionsParser pro: provide a consistent libjars syntax with classpaths pro: no impact on the existing behavior pro: only needed files (i.e. jars) are uploaded to hdfs and localized con: increases the size of the configuration (2) add the wildcard notation on the task classpath pro: smallest amount of changes; leverages existing functionality of accepting and uploading a directory pro: configuration stays slim con: would upload and localize files unnecessarily if only the jars were supposed to be used con: need to re-interpret or deprecate (minor) behavior, such as adding libjar entries to the client classpath and allowing directories as a set of classfiles In terms of the amount of changes, I believe the changes for approach (1) are still quite small. The only impactful change is expanding the wildcard in GenericOptionsParser. All other changes are refactoring to use a common implementation for that expansion and unit test changes. My proposal would be to proceed with approach (1). IMO there is a lot of value in having libjars consistent with the classpath syntax. And if we can provide that support with an equally small change, I don't see a reason not to do it. I'd like to hear your thoughts and others too!
          Hide
          ctrezzo Chris Trezzo added a comment -

          Thanks Sangjin Lee for breaking down the two options. Currently I am in favor of option 1. I think it is valuable to have the libjars syntax similar to the classpath syntax as a large number of users are familiar with that syntax. Another plus to expanding the wildcard pre-conf is that we get the per-path file size information in the config. Additionally, the conf is more explicit about what is being depended on by the task, which hopefully makes it harder to miss a task that draws in a large number of dependencies (unintentionally or intentionally).

          The v2 patch looks good to me. Do we want to separate out the changes to ApplicationClassLoader into another patch? I could go either way. The change is small, but orthogonal to this jira.

          Show
          ctrezzo Chris Trezzo added a comment - Thanks Sangjin Lee for breaking down the two options. Currently I am in favor of option 1. I think it is valuable to have the libjars syntax similar to the classpath syntax as a large number of users are familiar with that syntax. Another plus to expanding the wildcard pre-conf is that we get the per-path file size information in the config. Additionally, the conf is more explicit about what is being depended on by the task, which hopefully makes it harder to miss a task that draws in a large number of dependencies (unintentionally or intentionally). The v2 patch looks good to me. Do we want to separate out the changes to ApplicationClassLoader into another patch? I could go either way. The change is small, but orthogonal to this jira.
          Hide
          ctrezzo Chris Trezzo added a comment -

          I also like that option 1 does not change existing behavior, but simply adds additional functionality.

          Show
          ctrezzo Chris Trezzo added a comment - I also like that option 1 does not change existing behavior, but simply adds additional functionality.
          Hide
          sjlee0 Sangjin Lee added a comment -

          Posted patch v.3.

          Added a check for a non-local wildcard path, and a warning logging for an empty directory.

          Show
          sjlee0 Sangjin Lee added a comment - Posted patch v.3. Added a check for a non-local wildcard path, and a warning logging for an empty directory.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 10s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 6m 30s trunk passed
          +1 compile 5m 50s trunk passed with JDK v1.8.0_72
          +1 compile 6m 38s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 21s trunk passed
          +1 mvnsite 1m 1s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 32s trunk passed
          +1 javadoc 0m 51s trunk passed with JDK v1.8.0_72
          +1 javadoc 1m 4s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 38s the patch passed
          +1 compile 5m 49s the patch passed with JDK v1.8.0_72
          +1 javac 5m 49s the patch passed
          +1 compile 6m 38s the patch passed with JDK v1.7.0_95
          +1 javac 6m 38s the patch passed
          +1 checkstyle 0m 22s hadoop-common-project/hadoop-common: patch generated 0 new + 103 unchanged - 7 fixed = 103 total (was 110)
          +1 mvnsite 1m 0s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 1m 48s the patch passed
          +1 javadoc 0m 51s the patch passed with JDK v1.8.0_72
          +1 javadoc 1m 2s the patch passed with JDK v1.7.0_95
          +1 unit 6m 32s hadoop-common in the patch passed with JDK v1.8.0_72.
          -1 unit 6m 17s hadoop-common in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 21s Patch does not generate ASF License warnings.
          57m 0s



          Reason Tests
          JDK v1.7.0_95 Failed junit tests hadoop.security.ssl.TestReloadingX509TrustManager



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12787520/HADOOP-12747.03.patch
          JIRA Issue HADOOP-12747
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 1ba7dbcb723e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 23f937e
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8602/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8602/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8602/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Max memory used 77MB
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8602/console
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 6m 30s trunk passed +1 compile 5m 50s trunk passed with JDK v1.8.0_72 +1 compile 6m 38s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 21s trunk passed +1 mvnsite 1m 1s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 32s trunk passed +1 javadoc 0m 51s trunk passed with JDK v1.8.0_72 +1 javadoc 1m 4s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 38s the patch passed +1 compile 5m 49s the patch passed with JDK v1.8.0_72 +1 javac 5m 49s the patch passed +1 compile 6m 38s the patch passed with JDK v1.7.0_95 +1 javac 6m 38s the patch passed +1 checkstyle 0m 22s hadoop-common-project/hadoop-common: patch generated 0 new + 103 unchanged - 7 fixed = 103 total (was 110) +1 mvnsite 1m 0s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 48s the patch passed +1 javadoc 0m 51s the patch passed with JDK v1.8.0_72 +1 javadoc 1m 2s the patch passed with JDK v1.7.0_95 +1 unit 6m 32s hadoop-common in the patch passed with JDK v1.8.0_72. -1 unit 6m 17s hadoop-common in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 21s Patch does not generate ASF License warnings. 57m 0s Reason Tests JDK v1.7.0_95 Failed junit tests hadoop.security.ssl.TestReloadingX509TrustManager Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12787520/HADOOP-12747.03.patch JIRA Issue HADOOP-12747 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 1ba7dbcb723e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 23f937e Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8602/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8602/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8602/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Max memory used 77MB Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8602/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          sjlee0 Sangjin Lee added a comment -

          This is a question for Chris Nauroth (or anyone who's intimately familiar with how we support the libjars argument). You mentioned earlier that libjars don't support non-local paths, but strictly speaking HADOOP-7112 addresses only the aspect of adding libjars back to the client classpath. And I know for a fact today one can use hdfs URLs in libjars successfully (minus putting them in the client classpath). Is it an accidental behavior while the official position is that we don't support them? If we do support them in libjars, then we probably need to support wildcards for them as well. Any feedback is greatly appreciated.

          Show
          sjlee0 Sangjin Lee added a comment - This is a question for Chris Nauroth (or anyone who's intimately familiar with how we support the libjars argument). You mentioned earlier that libjars don't support non-local paths, but strictly speaking HADOOP-7112 addresses only the aspect of adding libjars back to the client classpath. And I know for a fact today one can use hdfs URLs in libjars successfully (minus putting them in the client classpath). Is it an accidental behavior while the official position is that we don't support them? If we do support them in libjars, then we probably need to support wildcards for them as well. Any feedback is greatly appreciated.
          Hide
          cnauroth Chris Nauroth added a comment -

          You mentioned earlier that libjars don't support non-local paths, but strictly speaking HADOOP-7112 addresses only the aspect of adding libjars back to the client classpath.

          That's very interesting. I missed the point that non-local jars are skipped only for adding to the client's own classpath. JobResourceUploader separately parses libjars and does not do the same filtering. Certainly since non-local libjars for the task is already supported, we'd have to maintain that behavior for reasons of backwards compatibility.

          I find the lack of consistency quite confusing. It's unclear to me how much of this behavior is by design and how much is accidental. I assume the filtering away from the client's classpath was done to avoid the complexity of needing to run some kind of "mini-localization" on the client side to support non-local files.

          Regarding the proposed options, I have a question on this con for option 2:

          con: need to re-interpret or deprecate (minor) behavior, such as adding libjar entries to the client classpath and allowing directories as a set of classfiles

          This sounds backwards-incompatible, right? If so, then that would tip my opinion towards option 1.

          Also, if wildcard expansion is delayed, then it seems there could be a risk of unexpected behavior if the contents of the directory change after job submission but before launch of the container. Maybe rolling upgrade scenarios would get weird. (Maybe not if the directories themselves are version-stamped properly.)

          Show
          cnauroth Chris Nauroth added a comment - You mentioned earlier that libjars don't support non-local paths, but strictly speaking HADOOP-7112 addresses only the aspect of adding libjars back to the client classpath. That's very interesting. I missed the point that non-local jars are skipped only for adding to the client's own classpath. JobResourceUploader separately parses libjars and does not do the same filtering. Certainly since non-local libjars for the task is already supported, we'd have to maintain that behavior for reasons of backwards compatibility. I find the lack of consistency quite confusing. It's unclear to me how much of this behavior is by design and how much is accidental. I assume the filtering away from the client's classpath was done to avoid the complexity of needing to run some kind of "mini-localization" on the client side to support non-local files. Regarding the proposed options, I have a question on this con for option 2: con: need to re-interpret or deprecate (minor) behavior, such as adding libjar entries to the client classpath and allowing directories as a set of classfiles This sounds backwards-incompatible, right? If so, then that would tip my opinion towards option 1. Also, if wildcard expansion is delayed, then it seems there could be a risk of unexpected behavior if the contents of the directory change after job submission but before launch of the container. Maybe rolling upgrade scenarios would get weird. (Maybe not if the directories themselves are version-stamped properly.)
          Hide
          sjlee0 Sangjin Lee added a comment -

          That's very interesting. I missed the point that non-local jars are skipped only for adding to the client's own classpath. JobResourceUploader separately parses libjars and does not do the same filtering. Certainly since non-local libjars for the task is already supported, we'd have to maintain that behavior for reasons of backwards compatibility.

          I find the lack of consistency quite confusing. It's unclear to me how much of this behavior is by design and how much is accidental. I assume the filtering away from the client's classpath was done to avoid the complexity of needing to run some kind of "mini-localization" on the client side to support non-local files.

          Yes, that's what I thought as well. The inconsistency may be that URLClassLoader does not support non-local paths by default, and we did not want the hassle of supporting them on the client-side classpath.

          Back to the original point, are you suggesting that we do allow wildcards for non-local paths and do similar expansion? I can update the patch to do that. Let me know. Thanks!

          Show
          sjlee0 Sangjin Lee added a comment - That's very interesting. I missed the point that non-local jars are skipped only for adding to the client's own classpath. JobResourceUploader separately parses libjars and does not do the same filtering. Certainly since non-local libjars for the task is already supported, we'd have to maintain that behavior for reasons of backwards compatibility. I find the lack of consistency quite confusing. It's unclear to me how much of this behavior is by design and how much is accidental. I assume the filtering away from the client's classpath was done to avoid the complexity of needing to run some kind of "mini-localization" on the client side to support non-local files. Yes, that's what I thought as well. The inconsistency may be that URLClassLoader does not support non-local paths by default, and we did not want the hassle of supporting them on the client-side classpath. Back to the original point, are you suggesting that we do allow wildcards for non-local paths and do similar expansion? I can update the patch to do that. Let me know. Thanks!
          Hide
          cnauroth Chris Nauroth added a comment -

          Back to the original point, are you suggesting that we do allow wildcards for non-local paths and do similar expansion?

          Yes, after noticing that the "local only" restriction applies only to the client classpath, I now think it's better to make this consistent and do wildcard expansion for non-local paths too.

          Do you mind targeting this change to 2.9.0? The wildcard matching logic in FileUtil has been finicky in the past, so I'm reluctant to change it now while 2.8.0 is closing down.

          Show
          cnauroth Chris Nauroth added a comment - Back to the original point, are you suggesting that we do allow wildcards for non-local paths and do similar expansion? Yes, after noticing that the "local only" restriction applies only to the client classpath, I now think it's better to make this consistent and do wildcard expansion for non-local paths too. Do you mind targeting this change to 2.9.0? The wildcard matching logic in FileUtil has been finicky in the past, so I'm reluctant to change it now while 2.8.0 is closing down.
          Hide
          sjlee0 Sangjin Lee added a comment -

          No problem. I'll update the patch soon.

          Show
          sjlee0 Sangjin Lee added a comment - No problem. I'll update the patch soon.
          Hide
          sjlee0 Sangjin Lee added a comment -

          Posted patch v.4.

          Added support for non-local path wildcards. Also, handled an edge case where all wildcard directories expanded to no entries. Touched up a warning logging message.

          Show
          sjlee0 Sangjin Lee added a comment - Posted patch v.4. Added support for non-local path wildcards. Also, handled an edge case where all wildcard directories expanded to no entries. Touched up a warning logging message.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 12s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 6m 45s trunk passed
          +1 compile 5m 51s trunk passed with JDK v1.8.0_72
          +1 compile 6m 29s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 21s trunk passed
          +1 mvnsite 1m 2s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 33s trunk passed
          +1 javadoc 0m 49s trunk passed with JDK v1.8.0_72
          +1 javadoc 1m 2s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 40s the patch passed
          +1 compile 5m 29s the patch passed with JDK v1.8.0_72
          +1 javac 5m 29s the patch passed
          +1 compile 6m 35s the patch passed with JDK v1.7.0_95
          +1 javac 6m 35s the patch passed
          -1 checkstyle 0m 21s hadoop-common-project/hadoop-common: patch generated 1 new + 102 unchanged - 8 fixed = 103 total (was 110)
          +1 mvnsite 1m 1s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 1m 47s the patch passed
          +1 javadoc 0m 52s the patch passed with JDK v1.8.0_72
          +1 javadoc 1m 2s the patch passed with JDK v1.7.0_95
          +1 unit 6m 41s hadoop-common in the patch passed with JDK v1.8.0_72.
          +1 unit 7m 9s hadoop-common in the patch passed with JDK v1.7.0_95.
          +1 asflicense 0m 23s Patch does not generate ASF License warnings.
          57m 36s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12791209/HADOOP-12747.04.patch
          JIRA Issue HADOOP-12747
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux c5d9fe4e9afd 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 0a9f00a
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8777/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8777/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8777/console
          Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 6m 45s trunk passed +1 compile 5m 51s trunk passed with JDK v1.8.0_72 +1 compile 6m 29s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 21s trunk passed +1 mvnsite 1m 2s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 33s trunk passed +1 javadoc 0m 49s trunk passed with JDK v1.8.0_72 +1 javadoc 1m 2s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 40s the patch passed +1 compile 5m 29s the patch passed with JDK v1.8.0_72 +1 javac 5m 29s the patch passed +1 compile 6m 35s the patch passed with JDK v1.7.0_95 +1 javac 6m 35s the patch passed -1 checkstyle 0m 21s hadoop-common-project/hadoop-common: patch generated 1 new + 102 unchanged - 8 fixed = 103 total (was 110) +1 mvnsite 1m 1s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 47s the patch passed +1 javadoc 0m 52s the patch passed with JDK v1.8.0_72 +1 javadoc 1m 2s the patch passed with JDK v1.7.0_95 +1 unit 6m 41s hadoop-common in the patch passed with JDK v1.8.0_72. +1 unit 7m 9s hadoop-common in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 57m 36s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12791209/HADOOP-12747.04.patch JIRA Issue HADOOP-12747 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux c5d9fe4e9afd 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0a9f00a Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8777/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8777/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8777/console Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          sjlee0 Sangjin Lee added a comment -

          V.5: addressed the checkstyle issue.

          Show
          sjlee0 Sangjin Lee added a comment - V.5: addressed the checkstyle issue.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 6m 52s trunk passed
          +1 compile 5m 53s trunk passed with JDK v1.8.0_74
          +1 compile 6m 51s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 23s trunk passed
          +1 mvnsite 1m 6s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 1m 43s trunk passed
          +1 javadoc 0m 54s trunk passed with JDK v1.8.0_74
          +1 javadoc 1m 3s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 42s the patch passed
          +1 compile 5m 35s the patch passed with JDK v1.8.0_74
          +1 javac 5m 35s the patch passed
          +1 compile 6m 36s the patch passed with JDK v1.7.0_95
          +1 javac 6m 36s the patch passed
          +1 checkstyle 0m 21s hadoop-common-project/hadoop-common: patch generated 0 new + 102 unchanged - 8 fixed = 102 total (was 110)
          +1 mvnsite 1m 1s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 1m 47s the patch passed
          +1 javadoc 0m 52s the patch passed with JDK v1.8.0_74
          +1 javadoc 1m 2s the patch passed with JDK v1.7.0_95
          +1 unit 6m 58s hadoop-common in the patch passed with JDK v1.8.0_74.
          +1 unit 7m 35s hadoop-common in the patch passed with JDK v1.7.0_95.
          +1 asflicense 0m 25s Patch does not generate ASF License warnings.
          59m 31s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12791395/HADOOP-12747.05.patch
          JIRA Issue HADOOP-12747
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 3cb7bac5aa62 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / ff0ee84
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8784/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8784/console
          Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 6m 52s trunk passed +1 compile 5m 53s trunk passed with JDK v1.8.0_74 +1 compile 6m 51s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 23s trunk passed +1 mvnsite 1m 6s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 43s trunk passed +1 javadoc 0m 54s trunk passed with JDK v1.8.0_74 +1 javadoc 1m 3s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 42s the patch passed +1 compile 5m 35s the patch passed with JDK v1.8.0_74 +1 javac 5m 35s the patch passed +1 compile 6m 36s the patch passed with JDK v1.7.0_95 +1 javac 6m 36s the patch passed +1 checkstyle 0m 21s hadoop-common-project/hadoop-common: patch generated 0 new + 102 unchanged - 8 fixed = 102 total (was 110) +1 mvnsite 1m 1s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 47s the patch passed +1 javadoc 0m 52s the patch passed with JDK v1.8.0_74 +1 javadoc 1m 2s the patch passed with JDK v1.7.0_95 +1 unit 6m 58s hadoop-common in the patch passed with JDK v1.8.0_74. +1 unit 7m 35s hadoop-common in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 25s Patch does not generate ASF License warnings. 59m 31s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12791395/HADOOP-12747.05.patch JIRA Issue HADOOP-12747 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 3cb7bac5aa62 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ff0ee84 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8784/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8784/console Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          sjlee0 Sangjin Lee added a comment -

          I'd appreciate your review. Thanks much!

          Show
          sjlee0 Sangjin Lee added a comment - I'd appreciate your review. Thanks much!
          Hide
          sjlee0 Sangjin Lee added a comment -

          Ping?

          Show
          sjlee0 Sangjin Lee added a comment - Ping?
          Hide
          sjlee0 Sangjin Lee added a comment -

          Posted patch v.6.

          Minor update to handle the case where "*" or "./*" is passed into libjars. Rebased the patch to trunk.

          Show
          sjlee0 Sangjin Lee added a comment - Posted patch v.6. Minor update to handle the case where "*" or "./*" is passed into libjars. Rebased the patch to trunk.
          Hide
          templedf Daniel Templeton added a comment -

          Just a quick comment from a first parse: both validateFiles() methods are missing the param tag for files and a return tag in the javadoc.

          Show
          templedf Daniel Templeton added a comment - Just a quick comment from a first parse: both validateFiles() methods are missing the param tag for files and a return tag in the javadoc.
          Hide
          sjlee0 Sangjin Lee added a comment -

          Ugh, thanks. I'll wait for the jenkins result to collect all issues if there are more and fix them.

          Show
          sjlee0 Sangjin Lee added a comment - Ugh, thanks. I'll wait for the jenkins result to collect all issues if there are more and fix them.
          Hide
          sjlee0 Sangjin Lee added a comment -

          For some reason, jenkins didn't post the results. It's here: https://builds.apache.org/job/PreCommit-HADOOP-Build/9917/console

          The unit test failure is the only -1 on the result, and it is unrelated to this patch. Checkstyle is -0 because a unit test class that I touched is lacking javadoc at the class level. I think it is OK to not worry about that one.

          Regarding the javadoc issue you mentioned Daniel, jenkins returned +1 because javadoc for private methods/fields is not required and they don't generate warnings. It is more for information. Would you prefer that I fix them still? Let me know... Thanks!

          Show
          sjlee0 Sangjin Lee added a comment - For some reason, jenkins didn't post the results. It's here: https://builds.apache.org/job/PreCommit-HADOOP-Build/9917/console The unit test failure is the only -1 on the result, and it is unrelated to this patch. Checkstyle is -0 because a unit test class that I touched is lacking javadoc at the class level. I think it is OK to not worry about that one. Regarding the javadoc issue you mentioned Daniel, jenkins returned +1 because javadoc for private methods/fields is not required and they don't generate warnings. It is more for information. Would you prefer that I fix them still? Let me know... Thanks!
          Hide
          sjlee0 Sangjin Lee added a comment -

          Posted patch v.7.

          Fixed javadoc on a few methods in GenericOptionsParser.

          Show
          sjlee0 Sangjin Lee added a comment - Posted patch v.7. Fixed javadoc on a few methods in GenericOptionsParser .
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 20s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 6m 47s trunk passed
          +1 compile 6m 52s trunk passed
          +1 checkstyle 0m 26s trunk passed
          +1 mvnsite 0m 57s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 1m 22s trunk passed
          +1 javadoc 0m 45s trunk passed
          +1 mvninstall 0m 38s the patch passed
          +1 compile 6m 51s the patch passed
          +1 javac 6m 51s the patch passed
          -0 checkstyle 0m 26s hadoop-common-project/hadoop-common: The patch generated 1 new + 181 unchanged - 11 fixed = 182 total (was 192)
          +1 mvnsite 0m 55s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 32s the patch passed
          +1 javadoc 0m 46s the patch passed
          -1 unit 7m 42s hadoop-common in the patch failed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          38m 27s



          Reason Tests
          Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:85209cc
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12815841/HADOOP-12747.07.patch
          JIRA Issue HADOOP-12747
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux d9eaeec9e357 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 0a5def1
          Default Java 1.8.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9918/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9918/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9918/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9918/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 6m 47s trunk passed +1 compile 6m 52s trunk passed +1 checkstyle 0m 26s trunk passed +1 mvnsite 0m 57s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 22s trunk passed +1 javadoc 0m 45s trunk passed +1 mvninstall 0m 38s the patch passed +1 compile 6m 51s the patch passed +1 javac 6m 51s the patch passed -0 checkstyle 0m 26s hadoop-common-project/hadoop-common: The patch generated 1 new + 181 unchanged - 11 fixed = 182 total (was 192) +1 mvnsite 0m 55s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 32s the patch passed +1 javadoc 0m 46s the patch passed -1 unit 7m 42s hadoop-common in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 38m 27s Reason Tests Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics Subsystem Report/Notes Docker Image:yetus/hadoop:85209cc JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12815841/HADOOP-12747.07.patch JIRA Issue HADOOP-12747 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux d9eaeec9e357 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0a5def1 Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9918/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9918/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9918/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9918/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          sjlee0 Sangjin Lee added a comment -

          The latest version (v.7) should be good to review. FWIW, I tested it with a real cluster for local paths and non-local paths.

          Show
          sjlee0 Sangjin Lee added a comment - The latest version (v.7) should be good to review. FWIW, I tested it with a real cluster for local paths and non-local paths.
          Hide
          vicaya Luke Lu added a comment -

          Hey Sangjin Lee, this is a very convenient feature. The patch looks good to me. +1.

          Show
          vicaya Luke Lu added a comment - Hey Sangjin Lee , this is a very convenient feature. The patch looks good to me. +1.
          Hide
          sjlee0 Sangjin Lee added a comment -

          Thanks Luke Lu! Unless there are objections, I'll commit it by EOD today.

          Show
          sjlee0 Sangjin Lee added a comment - Thanks Luke Lu ! Unless there are objections, I'll commit it by EOD today.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #10239 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10239/)
          HADOOP-12747. support wildcard in libjars argument (sjlee) (sjlee: rev 0ad48aa2c8f41196743305c711ea19cc48f186da)

          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileUtil.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/GenericOptionsParser.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ApplicationClassLoader.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestGenericOptionsParser.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #10239 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10239/ ) HADOOP-12747 . support wildcard in libjars argument (sjlee) (sjlee: rev 0ad48aa2c8f41196743305c711ea19cc48f186da) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileUtil.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/GenericOptionsParser.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ApplicationClassLoader.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestGenericOptionsParser.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java
          Hide
          sjlee0 Sangjin Lee added a comment -

          Committed. Thanks Chris Nauroth, Gera Shegalov, and Luke Lu for your reviews and comments.

          Show
          sjlee0 Sangjin Lee added a comment - Committed. Thanks Chris Nauroth , Gera Shegalov , and Luke Lu for your reviews and comments.

            People

            • Assignee:
              sjlee0 Sangjin Lee
              Reporter:
              sjlee0 Sangjin Lee
            • Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development