Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-alpha2
    • Component/s: None
    • Labels:
      None
    1. YARN-5576.1.patch
      63 kB
      Jian He
    2. YARN-5576.2.patch
      63 kB
      Jian He
    3. YARN-5576.3.patch
      62 kB
      Jian He
    4. YARN-5576.4.branch-2.patch
      64 kB
      Jian He
    5. YARN-5576.4.patch
      65 kB
      Jian He

      Activity

      Hide
      jianhe Jian He added a comment -
      • The patch adds the functionality to localize resource while the container is running.
      • A new ResourceSet class is created to keep track of all resources requested by the container, some refactoring is done in ContainerImpl.
      • LinuxContainerExecutor change will be done separately.
      Show
      jianhe Jian He added a comment - The patch adds the functionality to localize resource while the container is running. A new ResourceSet class is created to keep track of all resources requested by the container, some refactoring is done in ContainerImpl. LinuxContainerExecutor change will be done separately.
      Hide
      hadoopqa Hadoop QA added a comment -
      -1 overall



      Vote Subsystem Runtime Comment
      0 reexec 0m 18s Docker mode activated.
      +1 @author 0m 0s The patch does not contain any @author tags.
      +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
      +1 mvninstall 8m 21s trunk passed
      +1 compile 0m 30s trunk passed
      +1 checkstyle 0m 26s trunk passed
      +1 mvnsite 0m 29s trunk passed
      +1 mvneclipse 0m 14s trunk passed
      +1 findbugs 0m 46s trunk passed
      +1 javadoc 0m 17s trunk passed
      +1 mvninstall 0m 24s the patch passed
      +1 compile 0m 25s the patch passed
      -1 javac 0m 25s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager generated 1 new + 17 unchanged - 0 fixed = 18 total (was 17)
      -1 checkstyle 0m 22s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: The patch generated 12 new + 688 unchanged - 15 fixed = 700 total (was 703)
      +1 mvnsite 0m 27s the patch passed
      +1 mvneclipse 0m 11s the patch passed
      -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
      +1 findbugs 0m 52s the patch passed
      -1 javadoc 0m 15s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager generated 1 new + 242 unchanged - 0 fixed = 243 total (was 242)
      +1 unit 13m 38s hadoop-yarn-server-nodemanager in the patch passed.
      +1 asflicense 0m 17s The patch does not generate ASF License warnings.
      28m 54s



      Subsystem Report/Notes
      Docker Image:yetus/hadoop:9560f25
      JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12825954/YARN-5576.1.patch
      JIRA Issue YARN-5576
      Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
      uname Linux cafdeff88870 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
      Build tool maven
      Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
      git revision trunk / c258171
      Default Java 1.8.0_101
      findbugs v3.0.0
      javac https://builds.apache.org/job/PreCommit-YARN-Build/12924/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
      checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12924/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
      whitespace https://builds.apache.org/job/PreCommit-YARN-Build/12924/artifact/patchprocess/whitespace-eol.txt
      javadoc https://builds.apache.org/job/PreCommit-YARN-Build/12924/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
      Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12924/testReport/
      modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
      Console output https://builds.apache.org/job/PreCommit-YARN-Build/12924/console
      Powered by Apache Yetus 0.3.0 http://yetus.apache.org

      This message was automatically generated.

      Show
      hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files. +1 mvninstall 8m 21s trunk passed +1 compile 0m 30s trunk passed +1 checkstyle 0m 26s trunk passed +1 mvnsite 0m 29s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 0m 46s trunk passed +1 javadoc 0m 17s trunk passed +1 mvninstall 0m 24s the patch passed +1 compile 0m 25s the patch passed -1 javac 0m 25s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager generated 1 new + 17 unchanged - 0 fixed = 18 total (was 17) -1 checkstyle 0m 22s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: The patch generated 12 new + 688 unchanged - 15 fixed = 700 total (was 703) +1 mvnsite 0m 27s the patch passed +1 mvneclipse 0m 11s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 findbugs 0m 52s the patch passed -1 javadoc 0m 15s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager generated 1 new + 242 unchanged - 0 fixed = 243 total (was 242) +1 unit 13m 38s hadoop-yarn-server-nodemanager in the patch passed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 28m 54s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12825954/YARN-5576.1.patch JIRA Issue YARN-5576 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux cafdeff88870 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c258171 Default Java 1.8.0_101 findbugs v3.0.0 javac https://builds.apache.org/job/PreCommit-YARN-Build/12924/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12924/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/12924/artifact/patchprocess/whitespace-eol.txt javadoc https://builds.apache.org/job/PreCommit-YARN-Build/12924/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12924/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/12924/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
      Hide
      vvasudev Varun Vasudev added a comment -

      Thanks for the patch Jian He!
      1) Can you please look at the javac errors?

      2)

      +    if (!container.getContainerState().equals(
      +        org.apache.hadoop.yarn.server.nodemanager.
      +            containermanager.container.ContainerState.RUNNING)) {
      +      throw new YarnException(
      +          containerId + " is at " + container.getContainerState()
      +              + " state. Not able to localize new resources.");
      +    }
      

      Can we move this logic into the ContainerImpl class? Something line

       if(!container.canLocalizeResources()) { 

      It puts all the logic for this stuff in one place. It looks like similar logic is required in the ResourceLocalization class?

      +    EnumSet<ContainerState> set =
      +        EnumSet.of(ContainerState.LOCALIZING, ContainerState.RUNNING);
      +    if (!set.contains(c.getContainerState())) {
      +      LOG.warn(c.getContainerId() + " is at " + c.getContainerState()
      +          + " state, do not localize resources.");
      +      return;
      +    }
      

      3)

      @@ -844,22 +772,21 @@ public ContainerState transition(ContainerImpl container,
             ContainerResourceLocalizedEvent rsrcEvent = (ContainerResourceLocalizedEvent) event;
      
      -      container.localizedResources.put(location, sys);
      

      I couldn't figure out where this logic was moved to - can you please explain?

      4)

      +          if (new File(linkFile).exists()) {
      +            LOG.info("Symlink file already exists: " + linkFile);
      +          } else {
      

      We should throw an error here or at least flag this as a failed localization?

      5)

      +      container.diagnostics.append(failedEvent.getDiagnosticMessage());
      

      We need to check the diagnostics string size - if a AM sends us too many failed requests, the diagnostics string will just balloon in size.

      6)

      +            if (localizer != null && localizer.killContainerLocalizer.get()) {
      +              LOG.info("New " + event.getType() + " localize request for "
      +                  + locId + ", remove old private localizer.");
      +              cleanupPrivLocalizers(locId);
      +              localizer = null;
      +            }
      

      Can you explain the logic for this? I couldn't figure out why we need this.

      7)

      +    System.out.println("==================================");
      +    System.out.println(appDir.getAbsolutePath());
      +    System.out.println(appSysDir.getAbsolutePath());
      +    System.out.println(containerDir.getAbsolutePath());
      +    System.out.println(containerSysDir.getAbsolutePath());
      +    System.out.println(targetFile.getAbsolutePath());
      +    System.out.println("==================================");
      

      Do we need these lines in the test code? Maybe move them to LOG.debug?

      8)
      Can you also add a check in testLocalingResourceWhileContainerRunning to make sure we can’t localize for non-running containers?

      Show
      vvasudev Varun Vasudev added a comment - Thanks for the patch Jian He ! 1) Can you please look at the javac errors? 2) + if (!container.getContainerState().equals( + org.apache.hadoop.yarn.server.nodemanager. + containermanager.container.ContainerState.RUNNING)) { + throw new YarnException( + containerId + " is at " + container.getContainerState() + + " state. Not able to localize new resources." ); + } Can we move this logic into the ContainerImpl class? Something line if (!container.canLocalizeResources()) { It puts all the logic for this stuff in one place. It looks like similar logic is required in the ResourceLocalization class? + EnumSet<ContainerState> set = + EnumSet.of(ContainerState.LOCALIZING, ContainerState.RUNNING); + if (!set.contains(c.getContainerState())) { + LOG.warn(c.getContainerId() + " is at " + c.getContainerState() + + " state, do not localize resources." ); + return ; + } 3) @@ -844,22 +772,21 @@ public ContainerState transition(ContainerImpl container, ContainerResourceLocalizedEvent rsrcEvent = (ContainerResourceLocalizedEvent) event; - container.localizedResources.put(location, sys); I couldn't figure out where this logic was moved to - can you please explain? 4) + if ( new File(linkFile).exists()) { + LOG.info( "Symlink file already exists: " + linkFile); + } else { We should throw an error here or at least flag this as a failed localization? 5) + container.diagnostics.append(failedEvent.getDiagnosticMessage()); We need to check the diagnostics string size - if a AM sends us too many failed requests, the diagnostics string will just balloon in size. 6) + if (localizer != null && localizer.killContainerLocalizer.get()) { + LOG.info( "New " + event.getType() + " localize request for " + + locId + ", remove old private localizer." ); + cleanupPrivLocalizers(locId); + localizer = null ; + } Can you explain the logic for this? I couldn't figure out why we need this. 7) + System .out.println( "==================================" ); + System .out.println(appDir.getAbsolutePath()); + System .out.println(appSysDir.getAbsolutePath()); + System .out.println(containerDir.getAbsolutePath()); + System .out.println(containerSysDir.getAbsolutePath()); + System .out.println(targetFile.getAbsolutePath()); + System .out.println( "==================================" ); Do we need these lines in the test code? Maybe move them to LOG.debug? 8) Can you also add a check in testLocalingResourceWhileContainerRunning to make sure we can’t localize for non-running containers?
      Hide
      jianhe Jian He added a comment - - edited

      Thanks for the review, Varun

      Can we move this logic into the ContainerImpl class?

      It looks like similar logic is required in the ResourceLocalization class?

      The logic is different for these two places. The first one should only allow localizing when the container is running, while the second should allow if the container is either localizing or running. This logic is tailored to the localize API only, I feel adding canLocalizeResources method to the interface of Container makes it a bit confusing, as container can also localize while at localizing state in the normal scenario too. I prefer keep it outside as it should be clear enough to readers

      I couldn't figure out where this logic was moved to - can you please explain?

      It's moved to ResourceSet#resourceLocalized method

      We need to check the diagnostics string size - if a AM sends us too many failed requests, the diagnostics string will just balloon in size.

      Not quite sure how to check though. doesn't seem appropriate to add a new config for the diagnostics size. So just skip appending the diagnostics if it is larger than say 2000 in length?

      We should throw an error here or at least flag this as a failed localization?

      I don't think we should throw error. It's mentioned in the doc that we'll skip updating the symlink for now. This is a valid use case if we want to replacing existing resource while container is running. The changing symlink part, instead of being done here, will be done later when container re-launches.

      Can you explain the logic for this? I couldn't figure out why we need this.

      It's required because, in the normal scenario, when the localization completes before launching the container, the private localizer thread will be interrupted. And when we re-localize, we need to create a new Thread object as the old thread is stopped. Added code comments for this

      Can you also add a check in testLocalingResourceWhileContainerRunning to make sure we can’t localize for non-running containers?

      will do

      Do we need these lines in the test code? Maybe move them to LOG.debug?

      It's useful for debugging test. Anyway, I just removed it.

      Show
      jianhe Jian He added a comment - - edited Thanks for the review, Varun Can we move this logic into the ContainerImpl class? It looks like similar logic is required in the ResourceLocalization class? The logic is different for these two places. The first one should only allow localizing when the container is running, while the second should allow if the container is either localizing or running. This logic is tailored to the localize API only, I feel adding canLocalizeResources method to the interface of Container makes it a bit confusing, as container can also localize while at localizing state in the normal scenario too. I prefer keep it outside as it should be clear enough to readers I couldn't figure out where this logic was moved to - can you please explain? It's moved to ResourceSet#resourceLocalized method We need to check the diagnostics string size - if a AM sends us too many failed requests, the diagnostics string will just balloon in size. Not quite sure how to check though. doesn't seem appropriate to add a new config for the diagnostics size. So just skip appending the diagnostics if it is larger than say 2000 in length? We should throw an error here or at least flag this as a failed localization? I don't think we should throw error. It's mentioned in the doc that we'll skip updating the symlink for now. This is a valid use case if we want to replacing existing resource while container is running. The changing symlink part, instead of being done here, will be done later when container re-launches. Can you explain the logic for this? I couldn't figure out why we need this. It's required because, in the normal scenario, when the localization completes before launching the container, the private localizer thread will be interrupted. And when we re-localize, we need to create a new Thread object as the old thread is stopped. Added code comments for this Can you also add a check in testLocalingResourceWhileContainerRunning to make sure we can’t localize for non-running containers? will do Do we need these lines in the test code? Maybe move them to LOG.debug? It's useful for debugging test. Anyway, I just removed it.
      Hide
      jianhe Jian He added a comment -

      Upload a new path that fixed the comments

      Show
      jianhe Jian He added a comment - Upload a new path that fixed the comments
      Hide
      hadoopqa Hadoop QA added a comment -
      -1 overall



      Vote Subsystem Runtime Comment
      0 reexec 0m 21s Docker mode activated.
      +1 @author 0m 0s The patch does not contain any @author tags.
      +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
      +1 mvninstall 7m 20s trunk passed
      +1 compile 0m 29s trunk passed
      +1 checkstyle 0m 25s trunk passed
      +1 mvnsite 0m 30s trunk passed
      +1 mvneclipse 0m 13s trunk passed
      +1 findbugs 0m 42s trunk passed
      +1 javadoc 0m 18s trunk passed
      +1 mvninstall 0m 23s the patch passed
      +1 compile 0m 24s the patch passed
      +1 javac 0m 24s the patch passed
      -1 checkstyle 0m 23s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: The patch generated 10 new + 689 unchanged - 14 fixed = 699 total (was 703)
      +1 mvnsite 0m 25s the patch passed
      +1 mvneclipse 0m 10s the patch passed
      -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
      +1 findbugs 0m 47s the patch passed
      +1 javadoc 0m 15s the patch passed
      +1 unit 13m 57s hadoop-yarn-server-nodemanager in the patch passed.
      +1 asflicense 0m 15s The patch does not generate ASF License warnings.
      27m 58s



      Subsystem Report/Notes
      Docker Image:yetus/hadoop:9560f25
      JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12826312/YARN-5576.2.patch
      JIRA Issue YARN-5576
      Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
      uname Linux a63a43d89c8e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
      Build tool maven
      Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
      git revision trunk / 20ae1fa
      Default Java 1.8.0_101
      findbugs v3.0.0
      checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12964/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
      whitespace https://builds.apache.org/job/PreCommit-YARN-Build/12964/artifact/patchprocess/whitespace-eol.txt
      Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12964/testReport/
      modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
      Console output https://builds.apache.org/job/PreCommit-YARN-Build/12964/console
      Powered by Apache Yetus 0.3.0 http://yetus.apache.org

      This message was automatically generated.

      Show
      hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files. +1 mvninstall 7m 20s trunk passed +1 compile 0m 29s trunk passed +1 checkstyle 0m 25s trunk passed +1 mvnsite 0m 30s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 42s trunk passed +1 javadoc 0m 18s trunk passed +1 mvninstall 0m 23s the patch passed +1 compile 0m 24s the patch passed +1 javac 0m 24s the patch passed -1 checkstyle 0m 23s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: The patch generated 10 new + 689 unchanged - 14 fixed = 699 total (was 703) +1 mvnsite 0m 25s the patch passed +1 mvneclipse 0m 10s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 findbugs 0m 47s the patch passed +1 javadoc 0m 15s the patch passed +1 unit 13m 57s hadoop-yarn-server-nodemanager in the patch passed. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 27m 58s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12826312/YARN-5576.2.patch JIRA Issue YARN-5576 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux a63a43d89c8e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 20ae1fa Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12964/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/12964/artifact/patchprocess/whitespace-eol.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12964/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/12964/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
      Hide
      vvasudev Varun Vasudev added a comment -

      Thanks for the explanation and the new patch Jian He! It looks mostly fine. Can you take care of the checkstyle and whitespace warnings from Jenkins? It looks like a bunch of lines are more than 80 chars in length. Rest of the patch looks good.

      Show
      vvasudev Varun Vasudev added a comment - Thanks for the explanation and the new patch Jian He ! It looks mostly fine. Can you take care of the checkstyle and whitespace warnings from Jenkins? It looks like a bunch of lines are more than 80 chars in length. Rest of the patch looks good.
      Hide
      jianhe Jian He added a comment -

      Latest patch fixed the styles that I think make sense.

      Show
      jianhe Jian He added a comment - Latest patch fixed the styles that I think make sense.
      Hide
      hadoopqa Hadoop QA added a comment -
      -1 overall



      Vote Subsystem Runtime Comment
      0 reexec 0m 20s Docker mode activated.
      +1 @author 0m 0s The patch does not contain any @author tags.
      +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
      +1 mvninstall 6m 37s trunk passed
      +1 compile 0m 27s trunk passed
      +1 checkstyle 0m 25s trunk passed
      +1 mvnsite 0m 27s trunk passed
      +1 mvneclipse 0m 13s trunk passed
      +1 findbugs 0m 41s trunk passed
      +1 javadoc 0m 16s trunk passed
      +1 mvninstall 0m 22s the patch passed
      +1 compile 0m 24s the patch passed
      +1 javac 0m 24s the patch passed
      -1 checkstyle 0m 22s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: The patch generated 6 new + 689 unchanged - 14 fixed = 695 total (was 703)
      +1 mvnsite 0m 26s the patch passed
      +1 mvneclipse 0m 10s the patch passed
      +1 whitespace 0m 0s The patch has no whitespace issues.
      +1 findbugs 0m 47s the patch passed
      +1 javadoc 0m 14s the patch passed
      +1 unit 13m 20s hadoop-yarn-server-nodemanager in the patch passed.
      +1 asflicense 0m 15s The patch does not generate ASF License warnings.
      26m 25s



      Subsystem Report/Notes
      Docker Image:yetus/hadoop:9560f25
      JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12826615/YARN-5576.3.patch
      JIRA Issue YARN-5576
      Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
      uname Linux b75f5e0c325f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
      Build tool maven
      Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
      git revision trunk / 08f55cc
      Default Java 1.8.0_101
      findbugs v3.0.0
      checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12984/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
      Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12984/testReport/
      modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
      Console output https://builds.apache.org/job/PreCommit-YARN-Build/12984/console
      Powered by Apache Yetus 0.3.0 http://yetus.apache.org

      This message was automatically generated.

      Show
      hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files. +1 mvninstall 6m 37s trunk passed +1 compile 0m 27s trunk passed +1 checkstyle 0m 25s trunk passed +1 mvnsite 0m 27s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 41s trunk passed +1 javadoc 0m 16s trunk passed +1 mvninstall 0m 22s the patch passed +1 compile 0m 24s the patch passed +1 javac 0m 24s the patch passed -1 checkstyle 0m 22s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: The patch generated 6 new + 689 unchanged - 14 fixed = 695 total (was 703) +1 mvnsite 0m 26s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 47s the patch passed +1 javadoc 0m 14s the patch passed +1 unit 13m 20s hadoop-yarn-server-nodemanager in the patch passed. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 26m 25s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12826615/YARN-5576.3.patch JIRA Issue YARN-5576 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux b75f5e0c325f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 08f55cc Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12984/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12984/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/12984/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
      Hide
      vvasudev Varun Vasudev added a comment -

      +1 for the latest patch. I'll commit it tomorrow if no one objects.

      Show
      vvasudev Varun Vasudev added a comment - +1 for the latest patch. I'll commit it tomorrow if no one objects.
      Hide
      asuresh Arun Suresh added a comment -

      Thanks for the patch Jian He!

      Couple of comments:

      1. We need to override the ContainerManagerImpl::localize() method in the QueuingContainerManagerImpl. Re-localization should not be allowed if the container is currently queued (not yet running)
      2. I only see entries added to ResourceSet::resourcesFailedToBeLocalized set. Shouldnt we remove these once the AM is notified of the failure ? Also, Shouldn't these be notified back to the AM ? or we are just relying on the diagnostic string sent to the AM in the GetContainerStatus response to notify the AM ?
      3. wondering if we should have another RE_LOCALIZE_CONTAINER_RESOURCE event in the ResourceLocalizationService to distinguish from the localization needed for container initialization and correspondingly send different events to the Container. Or maybe for the timebeing, we should just rename INIT_CONTAINER_RESOURCE to LOCALIZE_CONTAINER_RESOURCE.
      4. Nit : spurious change in the imports of ContainerImpl and BaseAMRMProxyTest
      Show
      asuresh Arun Suresh added a comment - Thanks for the patch Jian He ! Couple of comments: We need to override the ContainerManagerImpl::localize() method in the QueuingContainerManagerImpl . Re-localization should not be allowed if the container is currently queued (not yet running) I only see entries added to ResourceSet::resourcesFailedToBeLocalized set. Shouldnt we remove these once the AM is notified of the failure ? Also, Shouldn't these be notified back to the AM ? or we are just relying on the diagnostic string sent to the AM in the GetContainerStatus response to notify the AM ? wondering if we should have another RE_LOCALIZE_CONTAINER_RESOURCE event in the ResourceLocalizationService to distinguish from the localization needed for container initialization and correspondingly send different events to the Container. Or maybe for the timebeing, we should just rename INIT_CONTAINER_RESOURCE to LOCALIZE_CONTAINER_RESOURCE . Nit : spurious change in the imports of ContainerImpl and BaseAMRMProxyTest
      Hide
      jianhe Jian He added a comment - - edited

      Thanks for the review, Arun !

      We need to override the ContainerManagerImpl::localize() method in the QueuingContainerManagerImpl. Re-localization should not be allowed if the container is currently queued (not yet running)

      It is not allowed, the method in ContainerManagerImpl only allows localization while running

      I only see entries added to ResourceSet::resourcesFailedToBeLocalized set. Shouldnt we remove these once the AM is notified of the failure ? Also, Shouldn't these be notified back to the AM ? or we are just relying on the diagnostic string sent to the AM in the GetContainerStatus response to notify the AM ?

      The status part is not yet implemented as mentioned in the parent jira. It'll will be done once the requirement is clear. Earlier I was thinking these will be sent as part of container status.

      wondering if we should have another RE_LOCALIZE_CONTAINER_RESOURCE event in the ResourceLocalizationService to distinguish from the localization needed for container initialization and correspondingly send different events to the Container. Or maybe for the timebeing, we should just rename INIT_CONTAINER_RESOURCE to LOCALIZE_CONTAINER_RESOURCE.

      I don't think adding new events type for doing the same thing is necessary at this point. This will also add additional complexity as you need to conditionally sends different types of events. The goal is to reuse existing code. I can rename it.

      : spurious change in the imports of ContainerImpl and BaseAMRMProxyTest

      That's done by IDE auto fixing some unused imports, I edited it manually.

      Show
      jianhe Jian He added a comment - - edited Thanks for the review, Arun ! We need to override the ContainerManagerImpl::localize() method in the QueuingContainerManagerImpl. Re-localization should not be allowed if the container is currently queued (not yet running) It is not allowed, the method in ContainerManagerImpl only allows localization while running I only see entries added to ResourceSet::resourcesFailedToBeLocalized set. Shouldnt we remove these once the AM is notified of the failure ? Also, Shouldn't these be notified back to the AM ? or we are just relying on the diagnostic string sent to the AM in the GetContainerStatus response to notify the AM ? The status part is not yet implemented as mentioned in the parent jira. It'll will be done once the requirement is clear. Earlier I was thinking these will be sent as part of container status. wondering if we should have another RE_LOCALIZE_CONTAINER_RESOURCE event in the ResourceLocalizationService to distinguish from the localization needed for container initialization and correspondingly send different events to the Container. Or maybe for the timebeing, we should just rename INIT_CONTAINER_RESOURCE to LOCALIZE_CONTAINER_RESOURCE. I don't think adding new events type for doing the same thing is necessary at this point. This will also add additional complexity as you need to conditionally sends different types of events. The goal is to reuse existing code. I can rename it. : spurious change in the imports of ContainerImpl and BaseAMRMProxyTest That's done by IDE auto fixing some unused imports, I edited it manually.
      Hide
      hadoopqa Hadoop QA added a comment -
      -1 overall



      Vote Subsystem Runtime Comment
      0 reexec 0m 17s Docker mode activated.
      +1 @author 0m 0s The patch does not contain any @author tags.
      +1 test4tests 0m 0s The patch appears to include 7 new or modified test files.
      +1 mvninstall 7m 1s trunk passed
      +1 compile 0m 27s trunk passed
      +1 checkstyle 0m 26s trunk passed
      +1 mvnsite 0m 28s trunk passed
      +1 mvneclipse 0m 12s trunk passed
      +1 findbugs 0m 42s trunk passed
      +1 javadoc 0m 16s trunk passed
      +1 mvninstall 0m 22s the patch passed
      +1 compile 0m 24s the patch passed
      +1 javac 0m 24s the patch passed
      -1 checkstyle 0m 22s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: The patch generated 6 new + 704 unchanged - 15 fixed = 710 total (was 719)
      +1 mvnsite 0m 24s the patch passed
      +1 mvneclipse 0m 10s the patch passed
      +1 whitespace 0m 0s The patch has no whitespace issues.
      +1 findbugs 0m 45s the patch passed
      +1 javadoc 0m 15s the patch passed
      +1 unit 13m 31s hadoop-yarn-server-nodemanager in the patch passed.
      +1 asflicense 0m 16s The patch does not generate ASF License warnings.
      27m 2s



      Subsystem Report/Notes
      Docker Image:yetus/hadoop:9560f25
      JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12826765/YARN-5576.4.patch
      JIRA Issue YARN-5576
      Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
      uname Linux a1ae6b7e23a4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
      Build tool maven
      Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
      git revision trunk / 0690f09
      Default Java 1.8.0_101
      findbugs v3.0.0
      checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13006/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
      Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13006/testReport/
      modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
      Console output https://builds.apache.org/job/PreCommit-YARN-Build/13006/console
      Powered by Apache Yetus 0.3.0 http://yetus.apache.org

      This message was automatically generated.

      Show
      hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 7 new or modified test files. +1 mvninstall 7m 1s trunk passed +1 compile 0m 27s trunk passed +1 checkstyle 0m 26s trunk passed +1 mvnsite 0m 28s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 0m 42s trunk passed +1 javadoc 0m 16s trunk passed +1 mvninstall 0m 22s the patch passed +1 compile 0m 24s the patch passed +1 javac 0m 24s the patch passed -1 checkstyle 0m 22s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: The patch generated 6 new + 704 unchanged - 15 fixed = 710 total (was 719) +1 mvnsite 0m 24s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 45s the patch passed +1 javadoc 0m 15s the patch passed +1 unit 13m 31s hadoop-yarn-server-nodemanager in the patch passed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 27m 2s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12826765/YARN-5576.4.patch JIRA Issue YARN-5576 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux a1ae6b7e23a4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0690f09 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13006/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13006/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/13006/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
      Hide
      asuresh Arun Suresh added a comment -

      the method in ContainerManagerImpl only allows localization while running

      Agreed.. for queued containers, we might need to queue the localizations too maybe... but we can think about that separately...

      +1

      Show
      asuresh Arun Suresh added a comment - the method in ContainerManagerImpl only allows localization while running Agreed.. for queued containers, we might need to queue the localizations too maybe... but we can think about that separately... +1
      Hide
      vvasudev Varun Vasudev added a comment -

      Arun Suresh - can I go ahead and commit the latest patch?

      Show
      vvasudev Varun Vasudev added a comment - Arun Suresh - can I go ahead and commit the latest patch?
      Hide
      asuresh Arun Suresh added a comment -

      yup... am +1..

      Show
      asuresh Arun Suresh added a comment - yup... am +1..
      Hide
      vvasudev Varun Vasudev added a comment -

      Jian He - the patch can't be cleanly cherry-picked to branch-2. Can you please upload a version for branch-2? Thanks!

      Show
      vvasudev Varun Vasudev added a comment - Jian He - the patch can't be cleanly cherry-picked to branch-2. Can you please upload a version for branch-2? Thanks!
      Hide
      jianhe Jian He added a comment -

      uploaded branch-2 patch

      Show
      jianhe Jian He added a comment - uploaded branch-2 patch
      Hide
      hadoopqa Hadoop QA added a comment -
      -1 overall



      Vote Subsystem Runtime Comment
      0 reexec 15m 43s Docker mode activated.
      +1 @author 0m 0s The patch does not contain any @author tags.
      +1 test4tests 0m 0s The patch appears to include 7 new or modified test files.
      +1 mvninstall 6m 36s branch-2 passed
      +1 compile 0m 23s branch-2 passed with JDK v1.8.0_101
      +1 compile 0m 28s branch-2 passed with JDK v1.7.0_111
      +1 checkstyle 0m 28s branch-2 passed
      +1 mvnsite 0m 29s branch-2 passed
      +1 mvneclipse 0m 14s branch-2 passed
      +1 findbugs 0m 55s branch-2 passed
      +1 javadoc 0m 16s branch-2 passed with JDK v1.8.0_101
      +1 javadoc 0m 20s branch-2 passed with JDK v1.7.0_111
      +1 mvninstall 0m 24s the patch passed
      +1 compile 0m 21s the patch passed with JDK v1.8.0_101
      +1 javac 0m 21s the patch passed
      +1 compile 0m 25s the patch passed with JDK v1.7.0_111
      +1 javac 0m 25s the patch passed
      -1 checkstyle 0m 25s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: The patch generated 7 new + 706 unchanged - 15 fixed = 713 total (was 721)
      +1 mvnsite 0m 27s the patch passed
      +1 mvneclipse 0m 11s the patch passed
      +1 whitespace 0m 0s The patch has no whitespace issues.
      +1 findbugs 1m 3s the patch passed
      +1 javadoc 0m 13s the patch passed with JDK v1.8.0_101
      +1 javadoc 0m 17s the patch passed with JDK v1.7.0_111
      -1 unit 13m 8s hadoop-yarn-server-nodemanager in the patch failed with JDK v1.8.0_101.
      +1 unit 14m 1s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_111.
      +1 asflicense 0m 16s The patch does not generate ASF License warnings.
      58m 13s



      Reason Tests
      JDK v1.8.0_101 Failed junit tests hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager
        hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService



      Subsystem Report/Notes
      Docker Image:yetus/hadoop:b59b8b7
      JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12826982/YARN-5576.4.branch-2.patch
      JIRA Issue YARN-5576
      Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
      uname Linux a3bce2f8889f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
      Build tool maven
      Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
      git revision branch-2 / a5748ee
      Default Java 1.7.0_111
      Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111
      findbugs v3.0.0
      checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13011/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
      unit https://builds.apache.org/job/PreCommit-YARN-Build/13011/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.8.0_101.txt
      unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13011/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.8.0_101.txt
      JDK v1.7.0_111 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13011/testReport/
      modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
      Console output https://builds.apache.org/job/PreCommit-YARN-Build/13011/console
      Powered by Apache Yetus 0.3.0 http://yetus.apache.org

      This message was automatically generated.

      Show
      hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 15m 43s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 7 new or modified test files. +1 mvninstall 6m 36s branch-2 passed +1 compile 0m 23s branch-2 passed with JDK v1.8.0_101 +1 compile 0m 28s branch-2 passed with JDK v1.7.0_111 +1 checkstyle 0m 28s branch-2 passed +1 mvnsite 0m 29s branch-2 passed +1 mvneclipse 0m 14s branch-2 passed +1 findbugs 0m 55s branch-2 passed +1 javadoc 0m 16s branch-2 passed with JDK v1.8.0_101 +1 javadoc 0m 20s branch-2 passed with JDK v1.7.0_111 +1 mvninstall 0m 24s the patch passed +1 compile 0m 21s the patch passed with JDK v1.8.0_101 +1 javac 0m 21s the patch passed +1 compile 0m 25s the patch passed with JDK v1.7.0_111 +1 javac 0m 25s the patch passed -1 checkstyle 0m 25s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: The patch generated 7 new + 706 unchanged - 15 fixed = 713 total (was 721) +1 mvnsite 0m 27s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 3s the patch passed +1 javadoc 0m 13s the patch passed with JDK v1.8.0_101 +1 javadoc 0m 17s the patch passed with JDK v1.7.0_111 -1 unit 13m 8s hadoop-yarn-server-nodemanager in the patch failed with JDK v1.8.0_101. +1 unit 14m 1s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_111. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 58m 13s Reason Tests JDK v1.8.0_101 Failed junit tests hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager   hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12826982/YARN-5576.4.branch-2.patch JIRA Issue YARN-5576 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux a3bce2f8889f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / a5748ee Default Java 1.7.0_111 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13011/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13011/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.8.0_101.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13011/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.8.0_101.txt JDK v1.7.0_111 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13011/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/13011/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
      Hide
      jianhe Jian He added a comment -

      The failed tests are passing locally for me.

      Show
      jianhe Jian He added a comment - The failed tests are passing locally for me.
      Hide
      vvasudev Varun Vasudev added a comment -

      Committed to trunk and branch-2. Thanks for the patch Jian He and for the review Arun Suresh!

      Show
      vvasudev Varun Vasudev added a comment - Committed to trunk and branch-2. Thanks for the patch Jian He and for the review Arun Suresh !
      Hide
      hudson Hudson added a comment -

      SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10397 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10397/)
      YARN-5576. Allow resource localization while container is running. (vvasudev: rev e6fcfe28e304062c7d09231db757acb2953703ce)

      • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java
      • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/Context.java
      • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
      • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/Container.java
      • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/event/ContainerLocalizationRequestEvent.java
      • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitorResourceChange.java
      • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DockerContainerExecutor.java
      • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java
      • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java
      • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/timelineservice/NMTimelinePublisher.java
      • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/MockContainer.java
      • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/BaseAMRMProxyTest.java
      • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestResourceLocalizationService.java
      • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
      • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/DummyContainerManager.java
      • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/event/LocalizationEventType.java
      • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java
      • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceSet.java
      • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java
      • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java
      • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java
      • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestContainerManagerWithLCE.java
      Show
      hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10397 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10397/ ) YARN-5576 . Allow resource localization while container is running. (vvasudev: rev e6fcfe28e304062c7d09231db757acb2953703ce) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/Context.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/Container.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/event/ContainerLocalizationRequestEvent.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitorResourceChange.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DockerContainerExecutor.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/timelineservice/NMTimelinePublisher.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/MockContainer.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/BaseAMRMProxyTest.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestResourceLocalizationService.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/DummyContainerManager.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/event/LocalizationEventType.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceSet.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestContainerManagerWithLCE.java

        People

        • Assignee:
          jianhe Jian He
          Reporter:
          jianhe Jian He
        • Votes:
          0 Vote for this issue
          Watchers:
          6 Start watching this issue

          Dates

          • Created:
            Updated:
            Resolved:

            Development