Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-4958

The file localization process should allow for wildcards to reduce the application footprint in the state store

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.8.0
    • Fix Version/s: 2.9.0, 3.0.0-alpha1
    • Component/s: nodemanager
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      When using the -libjars option to add classes to the classpath, every library so added is explicitly listed in the ContainerLaunchContext's local resources even though they're all uploaded to the same directory in HDFS. When using tools like Crunch without an uber JAR or when trying to take advantage of the shared cache, the number of libraries can be quite large. We've seen many cases where we had to turn down the max number of applications to prevent ZK from running out of heap because of the size of the state store entries.

      Rather than listing all files independently, this JIRA proposes to have the NM allow wildcards in the resource localization paths. Specifically, we propose to allow a path to have a final component (name) set to "*", which is interpreted by the NM as "download the full directory and link to every file in it from the job's working directory." This behavior is the same as the current behavior when using -libjars, but avoids explicitly listing every file.

      This JIRA does not attempt to provide more general purpose wildcards, such as "*.jar" or "file*", as having multiple entries for a single directory presents numerous logistical issues.

      This JIRA also does not attempt to integrate with the shared cache. That work will be left to a future JIRA. Specifically, this JIRA only applies when a full directory is uploaded. Currently the shared cache does not handle directory uploads.

      This JIRA proposes to allow for wildcards both in the internal processing of the -libjars switch and in paths added through the Job and DistributedCache classes.

      The proposed approach is to treat a path, "dir/*", as "dir" for purposes of all file verification and localization. In the final step, the NM will query the localized directory to get a list of the files in "dir" such that each can be linked from the job's working directory. Since $PWD/* is always included on the classpath, all JAR files in "dir" will be in the classpath.

      1. YARN-4958.001.patch
        13 kB
        Daniel Templeton
      2. YARN-4958.002.patch
        49 kB
        Daniel Templeton
      3. YARN-4958.003.patch
        49 kB
        Daniel Templeton
      4. YARN-4958.004.patch
        2 kB
        Daniel Templeton

        Issue Links

          Activity

          Hide
          templedf Daniel Templeton added a comment -

          Here's an initial cut at the patch. There are still a few issues to resolve, but some general testing shows that the patch behaves as expected.

          Show
          templedf Daniel Templeton added a comment - Here's an initial cut at the patch. There are still a few issues to resolve, but some general testing shows that the patch behaves as expected.
          Hide
          templedf Daniel Templeton added a comment -

          This JIRA is in the same space as HADOOP-12747. It's solving the same problem in a completely different way, with different side-effects. I think there is room for and value in both JIRAs.

          Show
          templedf Daniel Templeton added a comment - This JIRA is in the same space as HADOOP-12747 . It's solving the same problem in a completely different way, with different side-effects. I think there is room for and value in both JIRAs.
          Hide
          templedf Daniel Templeton added a comment -

          Gera Shegalov, I just noticed your comments on HADOOP-12747. I'd be curious to have your opinion on this one.

          Show
          templedf Daniel Templeton added a comment - Gera Shegalov , I just noticed your comments on HADOOP-12747 . I'd be curious to have your opinion on this one.
          Hide
          templedf Daniel Templeton added a comment -

          On closer examination, HADOOP-12747 may be solving a slightly different problem. I believe it's solving the problem of reducing the end user pain in specifying a large number of JARs in -libjar. This JIRA is solving the problem of reducing the state store impact of specifying a large number of JARs in -libjar. Based on looking at the implementations, the two JIRAs are related but orthogonal.

          Show
          templedf Daniel Templeton added a comment - On closer examination, HADOOP-12747 may be solving a slightly different problem. I believe it's solving the problem of reducing the end user pain in specifying a large number of JARs in -libjar. This JIRA is solving the problem of reducing the state store impact of specifying a large number of JARs in -libjar. Based on looking at the implementations, the two JIRAs are related but orthogonal.
          Hide
          sjlee0 Sangjin Lee added a comment -

          Thanks for your proposal Daniel Templeton! This is an important improvement. I took a look at the patch, and have some early feedback. I haven't had a chance to run the patch against a cluster yet, however.

          A quick note on the shared cache: I don't think this patch will break the shared cache. This patch goes to work mostly after the shared cache has done its part. The only thing is that jobs with a set of resources that are heavily drawn from the shared cache would not benefit from this patch as there would be few classpath files that will be coming from the staging directory. But that's for later...

          Also, I think HADOOP-12747 is largely orthogonal. It merely gives users a shorthand to address a set of jars.

          More questions and comments:

          • We'll need unit tests for this.
          • I suppose a test will quickly confirm this, but does this work correctly if we're dealing with a non-jar entry in the staging libjars directory? I just wanted to confirm that it gets added to the container-side classpath explicitly. The java classpath of "*" does not include non-jar resources in the directory.
          • Would there be any cross-platform issues? Have you had a chance to test it on Windows specifically? At first glance, there was nothing obvious that might be a platform-specific issue, but it would be good to double check.
          • This is just noting that the size of a wildcard entry (as in mapreduce.job.cache.files.filesizes) would be reported as 0. This is an existing behavior/issue with a directory entry.

          (ClientDistributedCacheManager.java)

          • l.242-243: Would it be simpler to reset current to the parent directory and simply invoke ancestorsHaveExecutePermissions() on it instead? Then, getFileStatus doesn't need to change, and the stat cache would also have only real paths (i.e. no "*" paths). Thoughts?

          (MRApps.java)

          • l.323-329: Would there be a case where there can be multiple attempts for the same directory? Is it for the case both "dir" and "dir/*" are included in cache.files? I'm not sure if you're addressing a new concern or an existing one.
          • l.338-341: Why would there be a wildcard for the paths (which come from mapreduce.job.classpath.files)?

          (ContainerExecutor.java)

          • This would apply to any entries that have the wildcard, and would effect things like PWD/* too?
          Show
          sjlee0 Sangjin Lee added a comment - Thanks for your proposal Daniel Templeton ! This is an important improvement. I took a look at the patch, and have some early feedback. I haven't had a chance to run the patch against a cluster yet, however. A quick note on the shared cache: I don't think this patch will break the shared cache. This patch goes to work mostly after the shared cache has done its part. The only thing is that jobs with a set of resources that are heavily drawn from the shared cache would not benefit from this patch as there would be few classpath files that will be coming from the staging directory. But that's for later... Also, I think HADOOP-12747 is largely orthogonal. It merely gives users a shorthand to address a set of jars. More questions and comments: We'll need unit tests for this. I suppose a test will quickly confirm this, but does this work correctly if we're dealing with a non-jar entry in the staging libjars directory? I just wanted to confirm that it gets added to the container-side classpath explicitly. The java classpath of "*" does not include non-jar resources in the directory. Would there be any cross-platform issues? Have you had a chance to test it on Windows specifically? At first glance, there was nothing obvious that might be a platform-specific issue, but it would be good to double check. This is just noting that the size of a wildcard entry (as in mapreduce.job.cache.files.filesizes ) would be reported as 0. This is an existing behavior/issue with a directory entry. (ClientDistributedCacheManager.java) l.242-243: Would it be simpler to reset current to the parent directory and simply invoke ancestorsHaveExecutePermissions() on it instead? Then, getFileStatus doesn't need to change, and the stat cache would also have only real paths (i.e. no "*" paths). Thoughts? (MRApps.java) l.323-329: Would there be a case where there can be multiple attempts for the same directory? Is it for the case both "dir" and "dir/*" are included in cache.files? I'm not sure if you're addressing a new concern or an existing one. l.338-341: Why would there be a wildcard for the paths (which come from mapreduce.job.classpath.files )? (ContainerExecutor.java) This would apply to any entries that have the wildcard, and would effect things like PWD/* too?
          Hide
          templedf Daniel Templeton added a comment -

          Thanks for the review, Sangjin Lee! I'm working on unit tests and cleaning up the code right now. I hope to have an updated patch by tomorrow.

          does this work correctly if we're dealing with a non-jar entry in the staging libjars directory?

          Depends of the definition of correctly. I defined * similar to your definition of * from HADOOP-12747: all JARs go in the classpath. This patch also links all non-JARs from the working directory, but they are not added to the classpath. I think that behavior is more consistent with HADOOP-12747 than adding everything to the classpath.

          Would there be any cross-platform issues?

          Good question. I was careful to keep the changes agnostic, but who knows. Probably worth testing.

          This is just noting that the size of a wildcard entry (as in mapreduce.job.cache.files.filesizes) would be reported as 0.

          Didn't notice that one. What would be the best behavior? Report the aggregate file size?

          Show
          templedf Daniel Templeton added a comment - Thanks for the review, Sangjin Lee ! I'm working on unit tests and cleaning up the code right now. I hope to have an updated patch by tomorrow. does this work correctly if we're dealing with a non-jar entry in the staging libjars directory? Depends of the definition of correctly. I defined * similar to your definition of * from HADOOP-12747 : all JARs go in the classpath. This patch also links all non-JARs from the working directory, but they are not added to the classpath. I think that behavior is more consistent with HADOOP-12747 than adding everything to the classpath. Would there be any cross-platform issues? Good question. I was careful to keep the changes agnostic, but who knows. Probably worth testing. This is just noting that the size of a wildcard entry (as in mapreduce.job.cache.files.filesizes) would be reported as 0. Didn't notice that one. What would be the best behavior? Report the aggregate file size?
          Hide
          sjlee0 Sangjin Lee added a comment -

          Depends of the definition of correctly. I defined * similar to your definition of * from HADOOP-12747: all JARs go in the classpath. This patch also links all non-JARs from the working directory, but they are not added to the classpath. I think that behavior is more consistent with HADOOP-12747 than adding everything to the classpath.

          Actually there is more to it. HADOOP-12747 is orthogonal to this.

          Suppose the user specified

          -libjars lib/foo.xml
          

          The intent is to upload that file to the staging directory and make it part of the task classpath. So, (1) that file should be uploaded to the staging/libjars directory (and localized of course), and (2) more importantly it should be made part of the task classpath. As we noted, libjars/* will not pick up this file. Therefore the task classpath should look something like

          libjars/*:libjars/foo.xml:...
          

          In other words, non-jar entries in the libjar directory must be explicitly enumerated. I believe this is what the addToClassPathIfNonJar() is all about. This should preserve this behavior. I suspect the current patch does that, but was wondering if you could confirm it with a test.

          Didn't notice that one. What would be the best behavior? Report the aggregate file size?

          The ideal behavior would be the aggregate size under the directory, but there would be complexity. Also, given that this is an existing issue, I'm fine with filing a separate JIRA to discuss and address it.

          Show
          sjlee0 Sangjin Lee added a comment - Depends of the definition of correctly. I defined * similar to your definition of * from HADOOP-12747 : all JARs go in the classpath. This patch also links all non-JARs from the working directory, but they are not added to the classpath. I think that behavior is more consistent with HADOOP-12747 than adding everything to the classpath. Actually there is more to it. HADOOP-12747 is orthogonal to this. Suppose the user specified -libjars lib/foo.xml The intent is to upload that file to the staging directory and make it part of the task classpath. So, (1) that file should be uploaded to the staging/libjars directory (and localized of course), and (2) more importantly it should be made part of the task classpath. As we noted, libjars/* will not pick up this file. Therefore the task classpath should look something like libjars/*:libjars/foo.xml:... In other words, non-jar entries in the libjar directory must be explicitly enumerated. I believe this is what the addToClassPathIfNonJar() is all about. This should preserve this behavior. I suspect the current patch does that, but was wondering if you could confirm it with a test. Didn't notice that one. What would be the best behavior? Report the aggregate file size? The ideal behavior would be the aggregate size under the directory, but there would be complexity. Also, given that this is an existing issue, I'm fine with filing a separate JIRA to discuss and address it.
          Hide
          templedf Daniel Templeton added a comment -

          Actually there is more to it.

          Got it. That scenario works. The dist cache will explicitly add the non-archive to the classpath, and the wildcard will make sure it gets linked on the NM.

          What won't work is if I, as a user, add libs/* to the archives and expect it to also put the non-jars into the classpath.

          Show
          templedf Daniel Templeton added a comment - Actually there is more to it. Got it. That scenario works. The dist cache will explicitly add the non-archive to the classpath, and the wildcard will make sure it gets linked on the NM. What won't work is if I, as a user, add libs/* to the archives and expect it to also put the non-jars into the classpath.
          Hide
          sjlee0 Sangjin Lee added a comment -

          Just to be sure, foo.xml does appear explicitly in the task's final classpath (i.e. in the container launch script), correct?

          What won't work is if I, as a user, add libs/* to the archives and expect it to also put the non-jars into the classpath.

          Understood. And that's fine. Thanks!

          Show
          sjlee0 Sangjin Lee added a comment - Just to be sure, foo.xml does appear explicitly in the task's final classpath (i.e. in the container launch script), correct? What won't work is if I, as a user, add libs/* to the archives and expect it to also put the non-jars into the classpath. Understood. And that's fine. Thanks!
          Hide
          templedf Daniel Templeton added a comment -

          Just to be sure, foo.xml does appear explicitly in the task's final classpath (i.e. in the container launch script), correct?

          Yes. With this patch, I split the definition of the classpath and the cache list. The classpath remains the same as it was before, i.e. all non-archives added via -libjar. Only the cache list gets replaced with tmpjars/*.

          Show
          templedf Daniel Templeton added a comment - Just to be sure, foo.xml does appear explicitly in the task's final classpath (i.e. in the container launch script), correct? Yes. With this patch, I split the definition of the classpath and the cache list. The classpath remains the same as it was before, i.e. all non-archives added via -libjar. Only the cache list gets replaced with tmpjars/*.
          Hide
          templedf Daniel Templeton added a comment -

          Oh, whoops. That was the intended behavior, but it looks like I clobbered it accidentally. I'll add that to the next patch.

          Show
          templedf Daniel Templeton added a comment - Oh, whoops. That was the intended behavior, but it looks like I clobbered it accidentally. I'll add that to the next patch.
          Hide
          templedf Daniel Templeton added a comment -

          l.242-243: Would it be simpler to reset current to the parent directory and simply invoke ancestorsHaveExecutePermissions() on it instead? Then, getFileStatus doesn't need to change, and the stat cache would also have only real paths (i.e. no "*" paths). Thoughts?

          I can't as easily do the path manipulation above the level of getFileStatus() because we're dealing with a URI above that. The net result of the current code, though, is that the wildcard path is swapped with its parent inside getFileStatus(), and only the parent directory is added to the stat cache. Looks like there was a bug there, though, that caused a wildcard path to always have a cache miss.

          l.323-329: Would there be a case where there can be multiple attempts for the same directory? Is it for the case both "dir" and "dir/*" are included in cache.files? I'm not sure if you're addressing a new concern or an existing one.

          It's both. The current behavior is to quietly overwrite a duplicate link. That seems like a bad idea.

          l.338-341: Why would there be a wildcard for the paths (which come from mapreduce.job.classpath.files)?

          Good point. I got a little overzealous. I'll remove that part in the next patch.

          Show
          templedf Daniel Templeton added a comment - l.242-243: Would it be simpler to reset current to the parent directory and simply invoke ancestorsHaveExecutePermissions() on it instead? Then, getFileStatus doesn't need to change, and the stat cache would also have only real paths (i.e. no "*" paths). Thoughts? I can't as easily do the path manipulation above the level of getFileStatus() because we're dealing with a URI above that. The net result of the current code, though, is that the wildcard path is swapped with its parent inside getFileStatus() , and only the parent directory is added to the stat cache. Looks like there was a bug there, though, that caused a wildcard path to always have a cache miss. l.323-329: Would there be a case where there can be multiple attempts for the same directory? Is it for the case both "dir" and "dir/*" are included in cache.files? I'm not sure if you're addressing a new concern or an existing one. It's both. The current behavior is to quietly overwrite a duplicate link. That seems like a bad idea. l.338-341: Why would there be a wildcard for the paths (which come from mapreduce.job.classpath.files)? Good point. I got a little overzealous. I'll remove that part in the next patch.
          Hide
          templedf Daniel Templeton added a comment -

          This patch should address most of the concerns. It adds multiple tests and good documentation. Sangjin Lee, please take a look when you get a chance.

          Show
          templedf Daniel Templeton added a comment - This patch should address most of the concerns. It adds multiple tests and good documentation. Sangjin Lee , please take a look when you get a chance.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
          0 mvndep 0m 16s Maven dependency ordering for branch
          +1 mvninstall 7m 13s trunk passed
          +1 compile 7m 5s trunk passed with JDK v1.8.0_91
          +1 compile 7m 2s trunk passed with JDK v1.7.0_95
          +1 checkstyle 1m 27s trunk passed
          +1 mvnsite 2m 50s trunk passed
          +1 mvneclipse 1m 5s trunk passed
          +1 findbugs 4m 54s trunk passed
          +1 javadoc 2m 26s trunk passed with JDK v1.8.0_91
          +1 javadoc 2m 36s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 2m 20s the patch passed
          +1 compile 6m 33s the patch passed with JDK v1.8.0_91
          -1 javac 9m 7s root-jdk1.8.0_91 with JDK v1.8.0_91 generated 24 new + 671 unchanged - 0 fixed = 695 total (was 671)
          +1 javac 6m 33s the patch passed
          +1 compile 7m 2s the patch passed with JDK v1.7.0_95
          -1 javac 16m 9s root-jdk1.7.0_95 with JDK v1.7.0_95 generated 24 new + 680 unchanged - 0 fixed = 704 total (was 680)
          +1 javac 7m 2s the patch passed
          -1 checkstyle 1m 26s root: patch generated 4 new + 233 unchanged - 16 fixed = 237 total (was 249)
          +1 mvnsite 2m 45s the patch passed
          +1 mvneclipse 1m 2s the patch passed
          -1 whitespace 0m 0s The patch has 3 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 xml 0m 0s The patch has no ill-formed XML file.
          +1 findbugs 5m 44s the patch passed
          +1 javadoc 2m 8s the patch passed with JDK v1.8.0_91
          +1 javadoc 2m 30s the patch passed with JDK v1.7.0_95
          -1 unit 8m 14s hadoop-common in the patch failed with JDK v1.8.0_91.
          +1 unit 11m 9s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_91.
          +1 unit 2m 3s hadoop-mapreduce-client-core in the patch passed with JDK v1.8.0_91.
          +1 unit 0m 42s hadoop-mapreduce-client-common in the patch passed with JDK v1.8.0_91.
          -1 unit 102m 28s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.8.0_91.
          +1 unit 8m 38s hadoop-common in the patch passed with JDK v1.7.0_95.
          +1 unit 11m 46s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_95.
          +1 unit 2m 26s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_95.
          +1 unit 0m 57s hadoop-mapreduce-client-common in the patch passed with JDK v1.7.0_95.
          -1 unit 107m 57s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.7.0_95.
          -1 asflicense 0m 31s Patch generated 1 ASF License warnings.
          327m 42s



          Reason Tests
          JDK v1.8.0_91 Failed junit tests hadoop.ha.TestZKFailoverController
            hadoop.mapred.TestMiniMRChildTask
          JDK v1.7.0_95 Failed junit tests hadoop.mapred.TestMiniMRChildTask



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:cf2ee45
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12803663/YARN-4958.002.patch
          JIRA Issue YARN-4958
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 2f6028f3cbd3 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 3f2816a
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          javac root-jdk1.8.0_91: https://builds.apache.org/job/PreCommit-YARN-Build/11428/artifact/patchprocess/diff-compile-javac-root-jdk1.8.0_91.txt
          javac root-jdk1.7.0_95: https://builds.apache.org/job/PreCommit-YARN-Build/11428/artifact/patchprocess/diff-compile-javac-root-jdk1.7.0_95.txt
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11428/artifact/patchprocess/diff-checkstyle-root.txt
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/11428/artifact/patchprocess/whitespace-eol.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/11428/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_91.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/11428/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_91.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/11428/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11428/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-YARN-Build/11428/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-YARN-Build/11428/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11428/testReport/
          asflicense https://builds.apache.org/job/PreCommit-YARN-Build/11428/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/11428/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. 0 mvndep 0m 16s Maven dependency ordering for branch +1 mvninstall 7m 13s trunk passed +1 compile 7m 5s trunk passed with JDK v1.8.0_91 +1 compile 7m 2s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 27s trunk passed +1 mvnsite 2m 50s trunk passed +1 mvneclipse 1m 5s trunk passed +1 findbugs 4m 54s trunk passed +1 javadoc 2m 26s trunk passed with JDK v1.8.0_91 +1 javadoc 2m 36s trunk passed with JDK v1.7.0_95 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 2m 20s the patch passed +1 compile 6m 33s the patch passed with JDK v1.8.0_91 -1 javac 9m 7s root-jdk1.8.0_91 with JDK v1.8.0_91 generated 24 new + 671 unchanged - 0 fixed = 695 total (was 671) +1 javac 6m 33s the patch passed +1 compile 7m 2s the patch passed with JDK v1.7.0_95 -1 javac 16m 9s root-jdk1.7.0_95 with JDK v1.7.0_95 generated 24 new + 680 unchanged - 0 fixed = 704 total (was 680) +1 javac 7m 2s the patch passed -1 checkstyle 1m 26s root: patch generated 4 new + 233 unchanged - 16 fixed = 237 total (was 249) +1 mvnsite 2m 45s the patch passed +1 mvneclipse 1m 2s the patch passed -1 whitespace 0m 0s The patch has 3 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 xml 0m 0s The patch has no ill-formed XML file. +1 findbugs 5m 44s the patch passed +1 javadoc 2m 8s the patch passed with JDK v1.8.0_91 +1 javadoc 2m 30s the patch passed with JDK v1.7.0_95 -1 unit 8m 14s hadoop-common in the patch failed with JDK v1.8.0_91. +1 unit 11m 9s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_91. +1 unit 2m 3s hadoop-mapreduce-client-core in the patch passed with JDK v1.8.0_91. +1 unit 0m 42s hadoop-mapreduce-client-common in the patch passed with JDK v1.8.0_91. -1 unit 102m 28s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.8.0_91. +1 unit 8m 38s hadoop-common in the patch passed with JDK v1.7.0_95. +1 unit 11m 46s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_95. +1 unit 2m 26s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_95. +1 unit 0m 57s hadoop-mapreduce-client-common in the patch passed with JDK v1.7.0_95. -1 unit 107m 57s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.7.0_95. -1 asflicense 0m 31s Patch generated 1 ASF License warnings. 327m 42s Reason Tests JDK v1.8.0_91 Failed junit tests hadoop.ha.TestZKFailoverController   hadoop.mapred.TestMiniMRChildTask JDK v1.7.0_95 Failed junit tests hadoop.mapred.TestMiniMRChildTask Subsystem Report/Notes Docker Image:yetus/hadoop:cf2ee45 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12803663/YARN-4958.002.patch JIRA Issue YARN-4958 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 2f6028f3cbd3 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 3f2816a Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 javac root-jdk1.8.0_91: https://builds.apache.org/job/PreCommit-YARN-Build/11428/artifact/patchprocess/diff-compile-javac-root-jdk1.8.0_91.txt javac root-jdk1.7.0_95: https://builds.apache.org/job/PreCommit-YARN-Build/11428/artifact/patchprocess/diff-compile-javac-root-jdk1.7.0_95.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11428/artifact/patchprocess/diff-checkstyle-root.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/11428/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/11428/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_91.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/11428/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_91.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/11428/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11428/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-YARN-Build/11428/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-YARN-Build/11428/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11428/testReport/ asflicense https://builds.apache.org/job/PreCommit-YARN-Build/11428/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/11428/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          templedf Daniel Templeton added a comment -

          This should resolve the checkstyle and whitespace issues. The compiler warnings are deprecation warnings which are unavoidable.

          Show
          templedf Daniel Templeton added a comment - This should resolve the checkstyle and whitespace issues. The compiler warnings are deprecation warnings which are unavoidable.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
          0 mvndep 0m 17s Maven dependency ordering for branch
          +1 mvninstall 7m 36s trunk passed
          +1 compile 7m 12s trunk passed with JDK v1.8.0_91
          +1 compile 7m 20s trunk passed with JDK v1.7.0_95
          +1 checkstyle 1m 28s trunk passed
          +1 mvnsite 2m 51s trunk passed
          +1 mvneclipse 1m 9s trunk passed
          +1 findbugs 4m 50s trunk passed
          +1 javadoc 2m 4s trunk passed with JDK v1.8.0_91
          +1 javadoc 2m 32s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 15s Maven dependency ordering for patch
          +1 mvninstall 2m 15s the patch passed
          +1 compile 6m 18s the patch passed with JDK v1.8.0_91
          -1 javac 8m 49s root-jdk1.8.0_91 with JDK v1.8.0_91 generated 23 new + 672 unchanged - 0 fixed = 695 total (was 672)
          +1 javac 6m 18s the patch passed
          +1 compile 7m 5s the patch passed with JDK v1.7.0_95
          -1 javac 15m 54s root-jdk1.7.0_95 with JDK v1.7.0_95 generated 23 new + 681 unchanged - 0 fixed = 704 total (was 681)
          +1 javac 7m 5s the patch passed
          +1 checkstyle 1m 26s root: patch generated 0 new + 229 unchanged - 17 fixed = 229 total (was 246)
          +1 mvnsite 2m 55s the patch passed
          +1 mvneclipse 1m 6s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 5m 59s the patch passed
          +1 javadoc 3m 7s hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_91 with JDK v1.8.0_91 generated 0 new + 2508 unchanged - 1 fixed = 2508 total (was 2509)
          +1 javadoc 0m 55s hadoop-common in the patch passed with JDK v1.8.0_91.
          +1 javadoc 0m 19s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_91.
          +1 javadoc 0m 22s hadoop-mapreduce-client-core in the patch passed with JDK v1.8.0_91.
          +1 javadoc 0m 21s hadoop-mapreduce-client-common in the patch passed with JDK v1.8.0_91.
          +1 javadoc 0m 12s hadoop-mapreduce-client-jobclient in the patch passed with JDK v1.8.0_91.
          +1 javadoc 2m 33s the patch passed with JDK v1.7.0_95
          +1 unit 7m 46s hadoop-common in the patch passed with JDK v1.8.0_91.
          +1 unit 11m 16s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_91.
          +1 unit 2m 0s hadoop-mapreduce-client-core in the patch passed with JDK v1.8.0_91.
          +1 unit 0m 41s hadoop-mapreduce-client-common in the patch passed with JDK v1.8.0_91.
          -1 unit 105m 25s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.8.0_91.
          +1 unit 8m 8s hadoop-common in the patch passed with JDK v1.7.0_95.
          +1 unit 12m 10s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_95.
          +1 unit 2m 30s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_95.
          +1 unit 1m 0s hadoop-mapreduce-client-common in the patch passed with JDK v1.7.0_95.
          -1 unit 109m 38s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.7.0_95.
          -1 asflicense 0m 32s Patch generated 1 ASF License warnings.
          332m 40s



          Reason Tests
          JDK v1.8.0_91 Failed junit tests hadoop.mapred.TestMRCJCFileOutputCommitter
            hadoop.mapred.TestMiniMRChildTask
          JDK v1.7.0_95 Failed junit tests hadoop.mapred.TestMRCJCFileOutputCommitter
            hadoop.mapred.TestMiniMRChildTask



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:cf2ee45
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804266/YARN-4958.003.patch
          JIRA Issue YARN-4958
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 4dc29c81ddfe 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 1217c8f
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          javac root-jdk1.8.0_91: https://builds.apache.org/job/PreCommit-YARN-Build/11483/artifact/patchprocess/diff-compile-javac-root-jdk1.8.0_91.txt
          javac root-jdk1.7.0_95: https://builds.apache.org/job/PreCommit-YARN-Build/11483/artifact/patchprocess/diff-compile-javac-root-jdk1.7.0_95.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/11483/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_91.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/11483/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11483/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-YARN-Build/11483/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11483/testReport/
          asflicense https://builds.apache.org/job/PreCommit-YARN-Build/11483/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/11483/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. 0 mvndep 0m 17s Maven dependency ordering for branch +1 mvninstall 7m 36s trunk passed +1 compile 7m 12s trunk passed with JDK v1.8.0_91 +1 compile 7m 20s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 28s trunk passed +1 mvnsite 2m 51s trunk passed +1 mvneclipse 1m 9s trunk passed +1 findbugs 4m 50s trunk passed +1 javadoc 2m 4s trunk passed with JDK v1.8.0_91 +1 javadoc 2m 32s trunk passed with JDK v1.7.0_95 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 2m 15s the patch passed +1 compile 6m 18s the patch passed with JDK v1.8.0_91 -1 javac 8m 49s root-jdk1.8.0_91 with JDK v1.8.0_91 generated 23 new + 672 unchanged - 0 fixed = 695 total (was 672) +1 javac 6m 18s the patch passed +1 compile 7m 5s the patch passed with JDK v1.7.0_95 -1 javac 15m 54s root-jdk1.7.0_95 with JDK v1.7.0_95 generated 23 new + 681 unchanged - 0 fixed = 704 total (was 681) +1 javac 7m 5s the patch passed +1 checkstyle 1m 26s root: patch generated 0 new + 229 unchanged - 17 fixed = 229 total (was 246) +1 mvnsite 2m 55s the patch passed +1 mvneclipse 1m 6s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 5m 59s the patch passed +1 javadoc 3m 7s hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_91 with JDK v1.8.0_91 generated 0 new + 2508 unchanged - 1 fixed = 2508 total (was 2509) +1 javadoc 0m 55s hadoop-common in the patch passed with JDK v1.8.0_91. +1 javadoc 0m 19s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_91. +1 javadoc 0m 22s hadoop-mapreduce-client-core in the patch passed with JDK v1.8.0_91. +1 javadoc 0m 21s hadoop-mapreduce-client-common in the patch passed with JDK v1.8.0_91. +1 javadoc 0m 12s hadoop-mapreduce-client-jobclient in the patch passed with JDK v1.8.0_91. +1 javadoc 2m 33s the patch passed with JDK v1.7.0_95 +1 unit 7m 46s hadoop-common in the patch passed with JDK v1.8.0_91. +1 unit 11m 16s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_91. +1 unit 2m 0s hadoop-mapreduce-client-core in the patch passed with JDK v1.8.0_91. +1 unit 0m 41s hadoop-mapreduce-client-common in the patch passed with JDK v1.8.0_91. -1 unit 105m 25s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.8.0_91. +1 unit 8m 8s hadoop-common in the patch passed with JDK v1.7.0_95. +1 unit 12m 10s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_95. +1 unit 2m 30s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_95. +1 unit 1m 0s hadoop-mapreduce-client-common in the patch passed with JDK v1.7.0_95. -1 unit 109m 38s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.7.0_95. -1 asflicense 0m 32s Patch generated 1 ASF License warnings. 332m 40s Reason Tests JDK v1.8.0_91 Failed junit tests hadoop.mapred.TestMRCJCFileOutputCommitter   hadoop.mapred.TestMiniMRChildTask JDK v1.7.0_95 Failed junit tests hadoop.mapred.TestMRCJCFileOutputCommitter   hadoop.mapred.TestMiniMRChildTask Subsystem Report/Notes Docker Image:yetus/hadoop:cf2ee45 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804266/YARN-4958.003.patch JIRA Issue YARN-4958 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 4dc29c81ddfe 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 1217c8f Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 javac root-jdk1.8.0_91: https://builds.apache.org/job/PreCommit-YARN-Build/11483/artifact/patchprocess/diff-compile-javac-root-jdk1.8.0_91.txt javac root-jdk1.7.0_95: https://builds.apache.org/job/PreCommit-YARN-Build/11483/artifact/patchprocess/diff-compile-javac-root-jdk1.7.0_95.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/11483/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_91.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/11483/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11483/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-YARN-Build/11483/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11483/testReport/ asflicense https://builds.apache.org/job/PreCommit-YARN-Build/11483/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/11483/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 12s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
          0 mvndep 0m 16s Maven dependency ordering for branch
          +1 mvninstall 9m 26s trunk passed
          +1 compile 9m 48s trunk passed
          +1 checkstyle 1m 45s trunk passed
          +1 mvnsite 3m 33s trunk passed
          +1 mvneclipse 1m 17s trunk passed
          +1 findbugs 5m 13s trunk passed
          +1 javadoc 2m 49s trunk passed
          0 mvndep 0m 15s Maven dependency ordering for patch
          +1 mvninstall 2m 56s the patch passed
          +1 compile 9m 49s the patch passed
          -1 javac 13m 0s root generated 23 new + 671 unchanged - 0 fixed = 694 total (was 671)
          +1 javac 9m 49s the patch passed
          -1 checkstyle 1m 46s root: patch generated 1 new + 228 unchanged - 18 fixed = 229 total (was 246)
          +1 mvnsite 3m 35s the patch passed
          +1 mvneclipse 1m 19s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 6m 27s the patch passed
          +1 javadoc 4m 1s hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core generated 0 new + 2508 unchanged - 1 fixed = 2508 total (was 2509)
          +1 javadoc 1m 12s hadoop-common in the patch passed.
          +1 javadoc 0m 25s hadoop-yarn-server-nodemanager in the patch passed.
          +1 javadoc 0m 30s hadoop-mapreduce-client-core in the patch passed.
          +1 javadoc 0m 29s hadoop-mapreduce-client-common in the patch passed.
          +1 javadoc 0m 17s hadoop-mapreduce-client-jobclient in the patch passed.
          +1 unit 10m 32s hadoop-common in the patch passed.
          -1 unit 12m 35s hadoop-yarn-server-nodemanager in the patch failed.
          +1 unit 2m 48s hadoop-mapreduce-client-core in the patch passed.
          +1 unit 0m 57s hadoop-mapreduce-client-common in the patch passed.
          -1 unit 121m 31s hadoop-mapreduce-client-jobclient in the patch failed.
          +1 asflicense 0m 41s Patch does not generate ASF License warnings.
          214m 12s



          Reason Tests
          Failed junit tests hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainerMetrics
            hadoop.mapred.TestMiniMRChildTask



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:2c91fd8
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804455/YARN-4958.003.patch
          JIRA Issue YARN-4958
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 96887a5fcc84 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 34fddd1
          Default Java 1.8.0_91
          findbugs v3.0.0
          javac root: https://builds.apache.org/job/PreCommit-YARN-Build/11500/artifact/patchprocess/diff-compile-javac-root.txt
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11500/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/11500/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/11500/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11500/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt https://builds.apache.org/job/PreCommit-YARN-Build/11500/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11500/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/11500/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. 0 mvndep 0m 16s Maven dependency ordering for branch +1 mvninstall 9m 26s trunk passed +1 compile 9m 48s trunk passed +1 checkstyle 1m 45s trunk passed +1 mvnsite 3m 33s trunk passed +1 mvneclipse 1m 17s trunk passed +1 findbugs 5m 13s trunk passed +1 javadoc 2m 49s trunk passed 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 2m 56s the patch passed +1 compile 9m 49s the patch passed -1 javac 13m 0s root generated 23 new + 671 unchanged - 0 fixed = 694 total (was 671) +1 javac 9m 49s the patch passed -1 checkstyle 1m 46s root: patch generated 1 new + 228 unchanged - 18 fixed = 229 total (was 246) +1 mvnsite 3m 35s the patch passed +1 mvneclipse 1m 19s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 6m 27s the patch passed +1 javadoc 4m 1s hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core generated 0 new + 2508 unchanged - 1 fixed = 2508 total (was 2509) +1 javadoc 1m 12s hadoop-common in the patch passed. +1 javadoc 0m 25s hadoop-yarn-server-nodemanager in the patch passed. +1 javadoc 0m 30s hadoop-mapreduce-client-core in the patch passed. +1 javadoc 0m 29s hadoop-mapreduce-client-common in the patch passed. +1 javadoc 0m 17s hadoop-mapreduce-client-jobclient in the patch passed. +1 unit 10m 32s hadoop-common in the patch passed. -1 unit 12m 35s hadoop-yarn-server-nodemanager in the patch failed. +1 unit 2m 48s hadoop-mapreduce-client-core in the patch passed. +1 unit 0m 57s hadoop-mapreduce-client-common in the patch passed. -1 unit 121m 31s hadoop-mapreduce-client-jobclient in the patch failed. +1 asflicense 0m 41s Patch does not generate ASF License warnings. 214m 12s Reason Tests Failed junit tests hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainerMetrics   hadoop.mapred.TestMiniMRChildTask Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804455/YARN-4958.003.patch JIRA Issue YARN-4958 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 96887a5fcc84 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 34fddd1 Default Java 1.8.0_91 findbugs v3.0.0 javac root: https://builds.apache.org/job/PreCommit-YARN-Build/11500/artifact/patchprocess/diff-compile-javac-root.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11500/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/11500/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/11500/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11500/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt https://builds.apache.org/job/PreCommit-YARN-Build/11500/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11500/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/11500/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          templedf Daniel Templeton added a comment -

          Looks like I need to dig into some of the unit test failures.

          Show
          templedf Daniel Templeton added a comment - Looks like I need to dig into some of the unit test failures.
          Hide
          templedf Daniel Templeton added a comment -

          The test failures are unrelated. I filed MAPREDUCE-6702 to track.

          Show
          templedf Daniel Templeton added a comment - The test failures are unrelated. I filed MAPREDUCE-6702 to track.
          Hide
          jira.shegalov Gera Shegalov added a comment -

          Hi Daniel Templeton, no particular comment other than there is that workaround that can achieve it with the what i was suggesting in HADOOP-12747, or programmatically, but it would be nice if it can be done in a more obvious way.

          Show
          jira.shegalov Gera Shegalov added a comment - Hi Daniel Templeton , no particular comment other than there is that workaround that can achieve it with the what i was suggesting in HADOOP-12747 , or programmatically, but it would be nice if it can be done in a more obvious way.
          Hide
          sjlee0 Sangjin Lee added a comment -

          Sorry Daniel Templeton, haven't had cycles to get back to this one. I'll definitely review the latest patch next week.

          Just to get data from you, I assume that you tested for cases such as non-jar entries in libjars, etc. Could you kindly confirm?

          Show
          sjlee0 Sangjin Lee added a comment - Sorry Daniel Templeton , haven't had cycles to get back to this one. I'll definitely review the latest patch next week. Just to get data from you, I assume that you tested for cases such as non-jar entries in libjars, etc. Could you kindly confirm?
          Hide
          templedf Daniel Templeton added a comment - - edited

          Sangjin Lee, I did test that non-JAR files work, but I did it by proxy. I submitted a regular file through -libjars and confirmed that the regular file was in the task's classpath. I did not actually check that the file could be seen by the classloader. This patch does not change the way that the classpath is constructed.

          Show
          templedf Daniel Templeton added a comment - - edited Sangjin Lee , I did test that non-JAR files work, but I did it by proxy. I submitted a regular file through -libjars and confirmed that the regular file was in the task's classpath. I did not actually check that the file could be seen by the classloader. This patch does not change the way that the classpath is constructed.
          Hide
          sjlee0 Sangjin Lee added a comment -

          Sorry Daniel Templeton it took me a while to get back to this.

          I finally got around to trying out the patch with a pseudo-distributed setup. I can confirm that the main use cases seem to work correctly, including the case of a non-jar resource in the staging directory. That said, I do have some high level comments as well as a couple of minor nits.

          (1)
          Regarding the decision of determining the public-ness solely based on the parent directory in the case of the wildcard, I'm wondering whether that would have any implications. It's probably not going to be common, but it is possible that the directory is public but there may be files that are not readable by others. Again, it's hard to imagine why one would do this, but if they did, would it cause a security issue on localization or a localization failure? Should we chalk that up to an unsupported setting? To be fair, I can see this being an issue if a directory was specified (not a wildcard), too. In that sense, we could say this is a manifestation of an existing issue... Thoughts?

          (2)
          With ContainerExecutor.java, what happens if the wildcarded directory has further nested directories? It appears we're symlinking only at the immediate child level. I suspect it would work correctly, but wanted to double check.

          (3)
          Were you able to test it in the local job mode?

          (4) ClientDistributedCacheManager.java

          • l.303: change System.out.println() to a logger logging statement

          (5) DistributedCache.java

          • l.295: typo: "it's" -> "its"

          Finally, since most of the changes are really in MAPREDUCE, perhaps this JIRA should be moved to the MAPREDUCE project. What do you think? If we really want to follow the rules to the letter, we would need to create separate JIRAs for all projects involved (HADOOP, YARN, and MAPREDUCE). I'd like to hear what you think. On a related note, you may want to drop the changes to Path.java if you can help it.

          Show
          sjlee0 Sangjin Lee added a comment - Sorry Daniel Templeton it took me a while to get back to this. I finally got around to trying out the patch with a pseudo-distributed setup. I can confirm that the main use cases seem to work correctly, including the case of a non-jar resource in the staging directory. That said, I do have some high level comments as well as a couple of minor nits. (1) Regarding the decision of determining the public-ness solely based on the parent directory in the case of the wildcard, I'm wondering whether that would have any implications. It's probably not going to be common, but it is possible that the directory is public but there may be files that are not readable by others. Again, it's hard to imagine why one would do this, but if they did, would it cause a security issue on localization or a localization failure? Should we chalk that up to an unsupported setting? To be fair, I can see this being an issue if a directory was specified (not a wildcard), too. In that sense, we could say this is a manifestation of an existing issue... Thoughts? (2) With ContainerExecutor.java , what happens if the wildcarded directory has further nested directories? It appears we're symlinking only at the immediate child level. I suspect it would work correctly, but wanted to double check. (3) Were you able to test it in the local job mode? (4) ClientDistributedCacheManager.java l.303: change System.out.println() to a logger logging statement (5) DistributedCache.java l.295: typo: "it's" -> "its" Finally, since most of the changes are really in MAPREDUCE, perhaps this JIRA should be moved to the MAPREDUCE project. What do you think? If we really want to follow the rules to the letter, we would need to create separate JIRAs for all projects involved (HADOOP, YARN, and MAPREDUCE). I'd like to hear what you think. On a related note, you may want to drop the changes to Path.java if you can help it.
          Hide
          templedf Daniel Templeton added a comment -

          No worries. Thanks for the review, Sangjin Lee.

          On point #1, I had the same thought with the same conclusion. Since it's an issue that already exists for directories, I don't see where this patch really changes anything. In the majority of cases, this patch will be used via -libjars, in which case there is no issue. In the remaining cases, the behavior is clearly documented in the javadoc, so I should think that's OK.

          On point #2, I haven't actually tested it, but it should work as one would expect: each subdir should be linked from the working dir.

          #3 is a good point. I think I did, but I don't remember anymore. I'll give that a go and let you know.

          I'll post a patch with fixes for #4 and #5.

          Strictly speaking this should be two separate JIRAs, but I don't think anyone is that fussy about it. I've seen plenty of patches that touch more than one project. I've submitted several myself that touched common and HDFS.

          Show
          templedf Daniel Templeton added a comment - No worries. Thanks for the review, Sangjin Lee . On point #1, I had the same thought with the same conclusion. Since it's an issue that already exists for directories, I don't see where this patch really changes anything. In the majority of cases, this patch will be used via -libjars, in which case there is no issue. In the remaining cases, the behavior is clearly documented in the javadoc, so I should think that's OK. On point #2, I haven't actually tested it, but it should work as one would expect: each subdir should be linked from the working dir. #3 is a good point. I think I did, but I don't remember anymore. I'll give that a go and let you know. I'll post a patch with fixes for #4 and #5. Strictly speaking this should be two separate JIRAs, but I don't think anyone is that fussy about it. I've seen plenty of patches that touch more than one project. I've submitted several myself that touched common and HDFS.
          Hide
          sjlee0 Sangjin Lee added a comment -

          Strictly speaking this should be two separate JIRAs, but I don't think anyone is that fussy about it. I've seen plenty of patches that touch more than one project. I've submitted several myself that touched common and HDFS.

          OK, although it'd be ideal to have two JIRAs (YARN to enable support for wildcards in container launch context and MAPREDUCE to take advantage of it), it might be good to move it to MAPREDUCE at least. The majority of the changes are really in MAPREDUCE. What do you think?

          Show
          sjlee0 Sangjin Lee added a comment - Strictly speaking this should be two separate JIRAs, but I don't think anyone is that fussy about it. I've seen plenty of patches that touch more than one project. I've submitted several myself that touched common and HDFS. OK, although it'd be ideal to have two JIRAs (YARN to enable support for wildcards in container launch context and MAPREDUCE to take advantage of it), it might be good to move it to MAPREDUCE at least. The majority of the changes are really in MAPREDUCE. What do you think?
          Hide
          templedf Daniel Templeton added a comment -

          OK, I moved the MR parts to MAPREDUCE-6719. I'll also file a separate JIRA to clean up the Path javadoc.

          Show
          templedf Daniel Templeton added a comment - OK, I moved the MR parts to MAPREDUCE-6719 . I'll also file a separate JIRA to clean up the Path javadoc.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 29s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 8m 48s trunk passed
          +1 compile 0m 44s trunk passed
          +1 checkstyle 0m 19s trunk passed
          +1 mvnsite 0m 34s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          +1 findbugs 0m 56s trunk passed
          +1 javadoc 0m 19s trunk passed
          +1 mvninstall 0m 30s the patch passed
          +1 compile 0m 29s the patch passed
          +1 javac 0m 29s the patch passed
          +1 checkstyle 0m 16s the patch passed
          +1 mvnsite 0m 34s the patch passed
          +1 mvneclipse 0m 16s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 59s the patch passed
          +1 javadoc 0m 18s the patch passed
          -1 unit 13m 52s hadoop-yarn-server-nodemanager in the patch failed.
          +1 asflicense 0m 22s The patch does not generate ASF License warnings.
          30m 44s



          Reason Tests
          Failed junit tests hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:e2f6409
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12811828/YARN-4958.004.patch
          JIRA Issue YARN-4958
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux f7999c69fd98 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / fc6b50c
          Default Java 1.8.0_91
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-YARN-Build/12080/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/12080/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12080/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/12080/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 29s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 8m 48s trunk passed +1 compile 0m 44s trunk passed +1 checkstyle 0m 19s trunk passed +1 mvnsite 0m 34s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 0m 56s trunk passed +1 javadoc 0m 19s trunk passed +1 mvninstall 0m 30s the patch passed +1 compile 0m 29s the patch passed +1 javac 0m 29s the patch passed +1 checkstyle 0m 16s the patch passed +1 mvnsite 0m 34s the patch passed +1 mvneclipse 0m 16s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 59s the patch passed +1 javadoc 0m 18s the patch passed -1 unit 13m 52s hadoop-yarn-server-nodemanager in the patch failed. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 30m 44s Reason Tests Failed junit tests hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager Subsystem Report/Notes Docker Image:yetus/hadoop:e2f6409 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12811828/YARN-4958.004.patch JIRA Issue YARN-4958 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux f7999c69fd98 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / fc6b50c Default Java 1.8.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/12080/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/12080/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12080/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/12080/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          sjlee0 Sangjin Lee added a comment -

          Thanks for the update Daniel Templeton! The latest patch looks good to me. I'll commit it shortly and look at MAPREDUCE-6719 after that. Should this go into trunk and 2.9.0?

          Show
          sjlee0 Sangjin Lee added a comment - Thanks for the update Daniel Templeton ! The latest patch looks good to me. I'll commit it shortly and look at MAPREDUCE-6719 after that. Should this go into trunk and 2.9.0?
          Hide
          sjlee0 Sangjin Lee added a comment -

          Committed the patch to trunk and branch-2. Thanks Daniel Templeton for your contribution!

          Show
          sjlee0 Sangjin Lee added a comment - Committed the patch to trunk and branch-2. Thanks Daniel Templeton for your contribution!
          Hide
          templedf Daniel Templeton added a comment -

          I filed HADOOP-13296 to deal with the Path changes.

          Show
          templedf Daniel Templeton added a comment - I filed HADOOP-13296 to deal with the Path changes.
          Hide
          templedf Daniel Templeton added a comment -

          I think it's safe for 2.9.

          Show
          templedf Daniel Templeton added a comment - I think it's safe for 2.9.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #9986 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9986/)
          YARN-4958. The file localization process should allow for wildcards to (sjlee: rev 5107a967fa2558deba11c33a326d4d2e5748f452)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #9986 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9986/ ) YARN-4958 . The file localization process should allow for wildcards to (sjlee: rev 5107a967fa2558deba11c33a326d4d2e5748f452) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java

            People

            • Assignee:
              templedf Daniel Templeton
              Reporter:
              templedf Daniel Templeton
            • Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development