Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 3.4.0
    • yarn
    • None
    • Reviewed

    Description

      RMNodes in SLS don't keep a track of runningApps on each node. Due to this, graceful decommissioning logic takes a hit as the nodes will decommission if there are no running containers on the node but some shuffle data was present on the node.

      In this Jira, we will add runningApps functionality in SLS for improving decommissioning logic of each node. This will help with autoscaling simulations on SLS.

      Attachments

        1. YARN-10663.0001.patch
          14 kB
          VADAGA ANANYO RAO
        2. YARN-10663.0002.patch
          19 kB
          VADAGA ANANYO RAO

        Issue Links

          Activity

            Recap of how actual impl code handles running and finished apps on each node:

            1. Each RMNode has a list of runningApplications. Running Apps are active apps which have run some container on that node.
            2. Each RMAppImpl maintains a copy `ranNodes` which are all the nodes on which the app has run containers.
            3. When the app is at its `FinalTransition`, the app iterates over all the ranNodes and triggers a `RMNodeCleanupAppEvent` for that node.
            4. RMNodeImpl handles RMNodeCleanupAppEvent by removing apps from `runningApplications` list to `finishedApplications` list.

            Based on this flow, I plan to:

            1. Add a `ranNodes` list in AMSimulator.
            2. Each time AMSimulator starts a container on a node (NMSimulator), we will:
              1. update the runningApps in the NMSimulator and,
              2. update the ranNodes in the AMSimulator
            3. When the app is finishing, for each node in ranNodes list in AMSimulator, we will remove the app from the runningApps list of that node.
            ananyo_rao VADAGA ANANYO RAO added a comment - Recap of how actual impl code handles running and finished apps on each node: Each RMNode has a list of runningApplications. Running Apps are active apps which have run some container on that node. Each RMAppImpl maintains a copy `ranNodes` which are all the nodes on which the app has run containers. When the app is at its `FinalTransition`, the app iterates over all the ranNodes and triggers a `RMNodeCleanupAppEvent` for that node. RMNodeImpl handles RMNodeCleanupAppEvent by removing apps from `runningApplications` list to `finishedApplications` list. Based on this flow, I plan to: Add a `ranNodes` list in AMSimulator. Each time AMSimulator starts a container on a node (NMSimulator), we will: update the runningApps in the NMSimulator and, update the ranNodes in the AMSimulator When the app is finishing, for each node in ranNodes list in AMSimulator, we will remove the app from the runningApps list of that node.
            snemeth Szilard Nemeth added a comment -

            Hi ananyo_rao,
            Change looks good to me.
            One small ask: Could you please add tests to the AMSimulator?
            I can't see the mentioned logic tested anywhere:

            When the app is finishing, for each node in ranNodes list in AMSimulator, we will remove the app from the runningApps list of that node.

            Please also note that I have set this to patch available status so jenkins will trigger.

            snemeth Szilard Nemeth added a comment - Hi ananyo_rao , Change looks good to me. One small ask: Could you please add tests to the AMSimulator? I can't see the mentioned logic tested anywhere: When the app is finishing, for each node in ranNodes list in AMSimulator, we will remove the app from the runningApps list of that node. Please also note that I have set this to patch available status so jenkins will trigger.
            hadoopci Hadoop QA added a comment -
            -1 overall



            Vote Subsystem Runtime Logfile Comment
            0 reexec 1m 38s Docker mode activated.
                  Prechecks  
            +1 dupname 0m 0s No case conflicting files found.
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 0m 0s test4tests The patch appears to include 1 new or modified test files.
                  trunk Compile Tests  
            +1 mvninstall 25m 42s trunk passed
            +1 compile 0m 26s trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
            +1 compile 0m 23s trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
            +1 checkstyle 0m 19s trunk passed
            +1 mvnsite 0m 26s trunk passed
            +1 shadedclient 16m 1s branch has no errors when building and testing our client artifacts.
            +1 javadoc 0m 23s trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
            +1 javadoc 0m 22s trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
            0 spotbugs 17m 29s Both FindBugs and SpotBugs are enabled, using SpotBugs.
            +1 spotbugs 0m 43s trunk passed
                  Patch Compile Tests  
            +1 mvninstall 0m 22s the patch passed
            +1 compile 0m 19s the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
            +1 javac 0m 19s the patch passed
            +1 compile 0m 17s the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
            +1 javac 0m 17s the patch passed
            -0 checkstyle 0m 13s https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/950/artifact/out/diff-checkstyle-hadoop-tools_hadoop-sls.txt hadoop-tools/hadoop-sls: The patch generated 21 new + 48 unchanged - 2 fixed = 69 total (was 50)
            +1 mvnsite 0m 19s the patch passed
            +1 whitespace 0m 0s The patch has no whitespace issues.
            +1 shadedclient 14m 54s patch has no errors when building and testing our client artifacts.
            -1 javadoc 0m 20s https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/950/artifact/out/diff-javadoc-javadoc-hadoop-tools_hadoop-sls-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt hadoop-tools_hadoop-sls-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 generated 1 new + 37 unchanged - 0 fixed = 38 total (was 37)
            -1 javadoc 0m 18s https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/950/artifact/out/diff-javadoc-javadoc-hadoop-tools_hadoop-sls-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08.txt hadoop-tools_hadoop-sls-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 generated 1 new + 37 unchanged - 0 fixed = 38 total (was 37)
            +1 spotbugs 0m 52s the patch passed
                  Other Tests  
            +1 unit 12m 1s hadoop-sls in the patch passed.
            +1 asflicense 0m 29s The patch does not generate ASF License warnings.
            77m 58s



            Subsystem Report/Notes
            Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/950/artifact/out/Dockerfile
            JIRA Issue YARN-10663
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/13021458/YARN-10663.0001.patch
            Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle spotbugs
            uname Linux 96c89b03bd06 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality personality/hadoop.sh
            git revision trunk / 803ac4b1a0f
            Default Java Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
            Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
            Test Results https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/950/testReport/
            Max. process+thread count 514 (vs. ulimit of 5500)
            modules C: hadoop-tools/hadoop-sls U: hadoop-tools/hadoop-sls
            Console output https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/950/console
            versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
            Powered by Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

            This message was automatically generated.

            hadoopci Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Logfile Comment 0 reexec 1m 38s Docker mode activated.       Prechecks   +1 dupname 0m 0s No case conflicting files found. +1 @author 0m 0s The patch does not contain any @author tags. +1 0m 0s test4tests The patch appears to include 1 new or modified test files.       trunk Compile Tests   +1 mvninstall 25m 42s trunk passed +1 compile 0m 26s trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 +1 compile 0m 23s trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 +1 checkstyle 0m 19s trunk passed +1 mvnsite 0m 26s trunk passed +1 shadedclient 16m 1s branch has no errors when building and testing our client artifacts. +1 javadoc 0m 23s trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 +1 javadoc 0m 22s trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 0 spotbugs 17m 29s Both FindBugs and SpotBugs are enabled, using SpotBugs. +1 spotbugs 0m 43s trunk passed       Patch Compile Tests   +1 mvninstall 0m 22s the patch passed +1 compile 0m 19s the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 +1 javac 0m 19s the patch passed +1 compile 0m 17s the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 +1 javac 0m 17s the patch passed -0 checkstyle 0m 13s https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/950/artifact/out/diff-checkstyle-hadoop-tools_hadoop-sls.txt hadoop-tools/hadoop-sls: The patch generated 21 new + 48 unchanged - 2 fixed = 69 total (was 50) +1 mvnsite 0m 19s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 shadedclient 14m 54s patch has no errors when building and testing our client artifacts. -1 javadoc 0m 20s https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/950/artifact/out/diff-javadoc-javadoc-hadoop-tools_hadoop-sls-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt hadoop-tools_hadoop-sls-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 generated 1 new + 37 unchanged - 0 fixed = 38 total (was 37) -1 javadoc 0m 18s https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/950/artifact/out/diff-javadoc-javadoc-hadoop-tools_hadoop-sls-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08.txt hadoop-tools_hadoop-sls-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 generated 1 new + 37 unchanged - 0 fixed = 38 total (was 37) +1 spotbugs 0m 52s the patch passed       Other Tests   +1 unit 12m 1s hadoop-sls in the patch passed. +1 asflicense 0m 29s The patch does not generate ASF License warnings. 77m 58s Subsystem Report/Notes Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/950/artifact/out/Dockerfile JIRA Issue YARN-10663 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/13021458/YARN-10663.0001.patch Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle spotbugs uname Linux 96c89b03bd06 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality personality/hadoop.sh git revision trunk / 803ac4b1a0f Default Java Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 Test Results https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/950/testReport/ Max. process+thread count 514 (vs. ulimit of 5500) modules C: hadoop-tools/hadoop-sls U: hadoop-tools/hadoop-sls Console output https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/950/console versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2 Powered by Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org This message was automatically generated.
            hadoopci Hadoop QA added a comment -
            -1 overall



            Vote Subsystem Runtime Logfile Comment
            0 reexec 23m 40s Docker mode activated.
                  Prechecks  
            +1 dupname 0m 1s No case conflicting files found.
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 0m 0s test4tests The patch appears to include 2 new or modified test files.
                  trunk Compile Tests  
            +1 mvninstall 36m 51s trunk passed
            +1 compile 0m 25s trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
            +1 compile 0m 23s trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
            +1 checkstyle 0m 20s trunk passed
            +1 mvnsite 0m 28s trunk passed
            +1 shadedclient 17m 8s branch has no errors when building and testing our client artifacts.
            +1 javadoc 0m 27s trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
            +1 javadoc 0m 25s trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
            0 spotbugs 18m 50s Both FindBugs and SpotBugs are enabled, using SpotBugs.
            +1 spotbugs 0m 50s trunk passed
                  Patch Compile Tests  
            +1 mvninstall 0m 23s the patch passed
            +1 compile 0m 21s the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
            +1 javac 0m 21s the patch passed
            +1 compile 0m 20s the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
            +1 javac 0m 20s the patch passed
            -0 checkstyle 0m 14s https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/1143/artifact/out/diff-checkstyle-hadoop-tools_hadoop-sls.txt hadoop-tools/hadoop-sls: The patch generated 21 new + 47 unchanged - 2 fixed = 68 total (was 49)
            +1 mvnsite 0m 23s the patch passed
            +1 whitespace 0m 0s The patch has no whitespace issues.
            +1 xml 0m 1s The patch has no ill-formed XML file.
            +1 shadedclient 15m 48s patch has no errors when building and testing our client artifacts.
            -1 javadoc 0m 23s https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/1143/artifact/out/diff-javadoc-javadoc-hadoop-tools_hadoop-sls-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt hadoop-tools_hadoop-sls-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 generated 1 new + 37 unchanged - 0 fixed = 38 total (was 37)
            -1 javadoc 0m 21s https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/1143/artifact/out/diff-javadoc-javadoc-hadoop-tools_hadoop-sls-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt hadoop-tools_hadoop-sls-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 generated 1 new + 37 unchanged - 0 fixed = 38 total (was 37)
            +1 spotbugs 1m 0s the patch passed
                  Other Tests  
            +1 unit 12m 17s hadoop-sls in the patch passed.
            +1 asflicense 0m 37s The patch does not generate ASF License warnings.
            114m 22s



            Subsystem Report/Notes
            Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/1143/artifact/out/Dockerfile
            JIRA Issue YARN-10663
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/13028972/YARN-10663.0002.patch
            Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle spotbugs
            uname Linux 476b699e8588 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality personality/hadoop.sh
            git revision trunk / b038042ece5
            Default Java Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
            Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
            Test Results https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/1143/testReport/
            Max. process+thread count 558 (vs. ulimit of 5500)
            modules C: hadoop-tools/hadoop-sls U: hadoop-tools/hadoop-sls
            Console output https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/1143/console
            versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
            Powered by Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

            This message was automatically generated.

            hadoopci Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Logfile Comment 0 reexec 23m 40s Docker mode activated.       Prechecks   +1 dupname 0m 1s No case conflicting files found. +1 @author 0m 0s The patch does not contain any @author tags. +1 0m 0s test4tests The patch appears to include 2 new or modified test files.       trunk Compile Tests   +1 mvninstall 36m 51s trunk passed +1 compile 0m 25s trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 +1 compile 0m 23s trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 +1 checkstyle 0m 20s trunk passed +1 mvnsite 0m 28s trunk passed +1 shadedclient 17m 8s branch has no errors when building and testing our client artifacts. +1 javadoc 0m 27s trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 +1 javadoc 0m 25s trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 0 spotbugs 18m 50s Both FindBugs and SpotBugs are enabled, using SpotBugs. +1 spotbugs 0m 50s trunk passed       Patch Compile Tests   +1 mvninstall 0m 23s the patch passed +1 compile 0m 21s the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 +1 javac 0m 21s the patch passed +1 compile 0m 20s the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 +1 javac 0m 20s the patch passed -0 checkstyle 0m 14s https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/1143/artifact/out/diff-checkstyle-hadoop-tools_hadoop-sls.txt hadoop-tools/hadoop-sls: The patch generated 21 new + 47 unchanged - 2 fixed = 68 total (was 49) +1 mvnsite 0m 23s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 shadedclient 15m 48s patch has no errors when building and testing our client artifacts. -1 javadoc 0m 23s https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/1143/artifact/out/diff-javadoc-javadoc-hadoop-tools_hadoop-sls-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt hadoop-tools_hadoop-sls-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 generated 1 new + 37 unchanged - 0 fixed = 38 total (was 37) -1 javadoc 0m 21s https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/1143/artifact/out/diff-javadoc-javadoc-hadoop-tools_hadoop-sls-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt hadoop-tools_hadoop-sls-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 generated 1 new + 37 unchanged - 0 fixed = 38 total (was 37) +1 spotbugs 1m 0s the patch passed       Other Tests   +1 unit 12m 17s hadoop-sls in the patch passed. +1 asflicense 0m 37s The patch does not generate ASF License warnings. 114m 22s Subsystem Report/Notes Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/1143/artifact/out/Dockerfile JIRA Issue YARN-10663 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/13028972/YARN-10663.0002.patch Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle spotbugs uname Linux 476b699e8588 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality personality/hadoop.sh git revision trunk / b038042ece5 Default Java Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 Test Results https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/1143/testReport/ Max. process+thread count 558 (vs. ulimit of 5500) modules C: hadoop-tools/hadoop-sls U: hadoop-tools/hadoop-sls Console output https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/1143/console versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2 Powered by Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org This message was automatically generated.
            hadoopci Hadoop QA added a comment -
            -1 overall



            Vote Subsystem Runtime Logfile Comment
            0 reexec 1m 40s Docker mode activated.
                  Prechecks  
            +1 dupname 0m 0s No case conflicting files found.
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 0m 0s test4tests The patch appears to include 2 new or modified test files.
                  trunk Compile Tests  
            +1 mvninstall 25m 7s trunk passed
            +1 compile 0m 29s trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
            +1 compile 0m 27s trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
            +1 checkstyle 0m 22s trunk passed
            +1 mvnsite 0m 32s trunk passed
            +1 shadedclient 18m 49s branch has no errors when building and testing our client artifacts.
            +1 javadoc 0m 27s trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
            +1 javadoc 0m 26s trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
            0 spotbugs 20m 33s Both FindBugs and SpotBugs are enabled, using SpotBugs.
            +1 spotbugs 0m 51s trunk passed
                  Patch Compile Tests  
            +1 mvninstall 0m 25s the patch passed
            +1 compile 0m 22s the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
            +1 javac 0m 22s the patch passed
            +1 compile 0m 19s the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
            +1 javac 0m 19s the patch passed
            -0 checkstyle 0m 14s https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/1147/artifact/out/diff-checkstyle-hadoop-tools_hadoop-sls.txt hadoop-tools/hadoop-sls: The patch generated 21 new + 47 unchanged - 2 fixed = 68 total (was 49)
            +1 mvnsite 0m 23s the patch passed
            +1 whitespace 0m 0s The patch has no whitespace issues.
            +1 xml 0m 1s The patch has no ill-formed XML file.
            +1 shadedclient 16m 41s patch has no errors when building and testing our client artifacts.
            -1 javadoc 0m 23s https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/1147/artifact/out/diff-javadoc-javadoc-hadoop-tools_hadoop-sls-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt hadoop-tools_hadoop-sls-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 generated 1 new + 37 unchanged - 0 fixed = 38 total (was 37)
            -1 javadoc 0m 22s https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/1147/artifact/out/diff-javadoc-javadoc-hadoop-tools_hadoop-sls-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt hadoop-tools_hadoop-sls-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 generated 1 new + 37 unchanged - 0 fixed = 38 total (was 37)
            +1 spotbugs 1m 7s the patch passed
                  Other Tests  
            -1 unit 11m 1s https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/1147/artifact/out/patch-unit-hadoop-tools_hadoop-sls.txt hadoop-sls in the patch passed.
            +1 asflicense 0m 32s The patch does not generate ASF License warnings.
            82m 17s



            Reason Tests
            Failed junit tests hadoop.yarn.sls.appmaster.TestAMSimulator
              hadoop.yarn.sls.TestReservationSystemInvariants



            Subsystem Report/Notes
            Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/1147/artifact/out/Dockerfile
            JIRA Issue YARN-10663
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/13028972/YARN-10663.0002.patch
            Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle spotbugs
            uname Linux 63c8c79e77e9 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality personality/hadoop.sh
            git revision trunk / f8135547696
            Default Java Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
            Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
            Test Results https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/1147/testReport/
            Max. process+thread count 575 (vs. ulimit of 5500)
            modules C: hadoop-tools/hadoop-sls U: hadoop-tools/hadoop-sls
            Console output https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/1147/console
            versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
            Powered by Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

            This message was automatically generated.

            hadoopci Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Logfile Comment 0 reexec 1m 40s Docker mode activated.       Prechecks   +1 dupname 0m 0s No case conflicting files found. +1 @author 0m 0s The patch does not contain any @author tags. +1 0m 0s test4tests The patch appears to include 2 new or modified test files.       trunk Compile Tests   +1 mvninstall 25m 7s trunk passed +1 compile 0m 29s trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 +1 compile 0m 27s trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 +1 checkstyle 0m 22s trunk passed +1 mvnsite 0m 32s trunk passed +1 shadedclient 18m 49s branch has no errors when building and testing our client artifacts. +1 javadoc 0m 27s trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 +1 javadoc 0m 26s trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 0 spotbugs 20m 33s Both FindBugs and SpotBugs are enabled, using SpotBugs. +1 spotbugs 0m 51s trunk passed       Patch Compile Tests   +1 mvninstall 0m 25s the patch passed +1 compile 0m 22s the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 +1 javac 0m 22s the patch passed +1 compile 0m 19s the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 +1 javac 0m 19s the patch passed -0 checkstyle 0m 14s https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/1147/artifact/out/diff-checkstyle-hadoop-tools_hadoop-sls.txt hadoop-tools/hadoop-sls: The patch generated 21 new + 47 unchanged - 2 fixed = 68 total (was 49) +1 mvnsite 0m 23s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 shadedclient 16m 41s patch has no errors when building and testing our client artifacts. -1 javadoc 0m 23s https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/1147/artifact/out/diff-javadoc-javadoc-hadoop-tools_hadoop-sls-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt hadoop-tools_hadoop-sls-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 generated 1 new + 37 unchanged - 0 fixed = 38 total (was 37) -1 javadoc 0m 22s https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/1147/artifact/out/diff-javadoc-javadoc-hadoop-tools_hadoop-sls-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt hadoop-tools_hadoop-sls-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 generated 1 new + 37 unchanged - 0 fixed = 38 total (was 37) +1 spotbugs 1m 7s the patch passed       Other Tests   -1 unit 11m 1s https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/1147/artifact/out/patch-unit-hadoop-tools_hadoop-sls.txt hadoop-sls in the patch passed. +1 asflicense 0m 32s The patch does not generate ASF License warnings. 82m 17s Reason Tests Failed junit tests hadoop.yarn.sls.appmaster.TestAMSimulator   hadoop.yarn.sls.TestReservationSystemInvariants Subsystem Report/Notes Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/1147/artifact/out/Dockerfile JIRA Issue YARN-10663 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/13028972/YARN-10663.0002.patch Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle spotbugs uname Linux 63c8c79e77e9 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality personality/hadoop.sh git revision trunk / f8135547696 Default Java Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 Test Results https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/1147/testReport/ Max. process+thread count 575 (vs. ulimit of 5500) modules C: hadoop-tools/hadoop-sls U: hadoop-tools/hadoop-sls Console output https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/1147/console versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2 Powered by Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org This message was automatically generated.
            snemeth Szilard Nemeth added a comment -

            Hi ananyo_rao,
            Thanks for working on this.
            Latest patch LGTM, committed to trunk.

            One side note:
            Please set up your code formatter rules in your IDE, I fixed some formatting issues like missing spaces after for/while loop expressions, indentation level fixes for multi-line expressions and such kind of things.
            If you would like to get help for the rules I can help you offline.
            For reference, these were the checkstyle issues reported by Jenkins:

            ./hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/nodemanager/NMSimulator.java:274:      && !getNode().getRunningApps().contains(applicationId)) {: '&&' has incorrect indentation level 6, expected level should be 8. [Indentation]
            ./hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/nodemanager/NMSimulator.java:278:      "Updated runningApps on this node are: {}",: '"Updated runningApps on this node are: {}"' has incorrect indentation level 6, expected level should be 8. [Indentation]
            ./hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/nodemanager/NMSimulator.java:279:      applicationId, getNode().getNodeID(), getNode().getRunningApps());: 'applicationId' has incorrect indentation level 6, expected level should be 8. [Indentation]
            ./hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/nodemanager/NMSimulator.java:315:        applicationId, getNode().getNodeID(), getNode().getRunningApps());: 'applicationId' has incorrect indentation level 8, expected level should be 10. [Indentation]
            ./hadoop-tools/hadoop-sls/src/test/java/org/apache/hadoop/yarn/sls/appmaster/TestAMSimulator.java:308:    app.appId = ApplicationId.newInstance(0l, 1);:43: Should use uppercase 'L'. [UpperEll]
            ./hadoop-tools/hadoop-sls/src/test/java/org/apache/hadoop/yarn/sls/nodemanager/TestNMSimulator.java:114:    node1.addNewContainer(container1, 100000l, null);:39: Should use uppercase 'L'. [UpperEll]
            ./hadoop-tools/hadoop-sls/src/test/java/org/apache/hadoop/yarn/sls/nodemanager/TestNMSimulator.java:122:    node1.addNewContainer(container2, -1l, null);:40: Should use uppercase 'L'. [UpperEll]
            ./hadoop-tools/hadoop-sls/src/test/java/org/apache/hadoop/yarn/sls/nodemanager/TestNMSimulator.java:147:      rm, -1f);: 'rm' has incorrect indentation level 6, expected level should be 8. [Indentation]
            ./hadoop-tools/hadoop-sls/src/test/java/org/apache/hadoop/yarn/sls/nodemanager/TestNMSimulator.java:160:    GenericTestUtils.waitFor(new org.apache.hadoop.thirdparty.com.google.common.base.Supplier<Boolean>() {: Line is longer than 100 characters (found 106). [LineLength]
            ./hadoop-tools/hadoop-sls/src/test/java/org/apache/hadoop/yarn/sls/nodemanager/TestNMSimulator.java:168:      node.getNode().getRunningApps().size(), 0);: 'node' has incorrect indentation level 6, expected level should be 8. [Indentation]
            ./hadoop-tools/hadoop-sls/src/test/java/org/apache/hadoop/yarn/sls/nodemanager/TestNMSimulator.java:175:      Resources.createResource(GB, 1), null, null);: 'Resources' has incorrect indentation level 6, expected level should be 8. [Indentation]
            ./hadoop-tools/hadoop-sls/src/test/java/org/apache/hadoop/yarn/sls/nodemanager/TestNMSimulator.java:176:    node.addNewContainer(container, 100000l, appId);:37: Should use uppercase 'L'. [UpperEll]
            ./hadoop-tools/hadoop-sls/src/test/java/org/apache/hadoop/yarn/sls/nodemanager/TestNMSimulator.java:179:      node.getNode().getRunningApps().contains(appId));: 'node' has incorrect indentation level 6, expected level should be 8. [Indentation]
            ./hadoop-tools/hadoop-sls/src/test/java/org/apache/hadoop/yarn/sls/nodemanager/TestNMSimulator.java:187:      node.getNode().getRunningApps().size(), 0);: 'node' has incorrect indentation level 6, expected level should be 8. [Indentation]
            ./hadoop-tools/hadoop-sls/src/test/java/org/apache/hadoop/yarn/sls/nodemanager/TestNMSimulator.java:195:      rm, -1f);: 'rm' has incorrect indentation level 6, expected level should be 8. [Indentation]
            ./hadoop-tools/hadoop-sls/src/test/java/org/apache/hadoop/yarn/sls/nodemanager/TestNMSimulator.java:208:    GenericTestUtils.waitFor(new org.apache.hadoop.thirdparty.com.google.common.base.Supplier<Boolean>() {: Line is longer than 100 characters (found 106). [LineLength]
            ./hadoop-tools/hadoop-sls/src/test/java/org/apache/hadoop/yarn/sls/nodemanager/TestNMSimulator.java:216:      node.getNode().getRunningApps().size(), 0);: 'node' has incorrect indentation level 6, expected level should be 8. [Indentation]
            ./hadoop-tools/hadoop-sls/src/test/java/org/apache/hadoop/yarn/sls/nodemanager/TestNMSimulator.java:221:      Resources.createResource(GB, 1), null, null);: 'Resources' has incorrect indentation level 6, expected level should be 8. [Indentation]
            ./hadoop-tools/hadoop-sls/src/test/java/org/apache/hadoop/yarn/sls/nodemanager/TestNMSimulator.java:222:    node.addNewContainer(container, 100000l, null);:37: Should use uppercase 'L'. [UpperEll]
            ./hadoop-tools/hadoop-sls/src/test/java/org/apache/hadoop/yarn/sls/nodemanager/TestNMSimulator.java:224:      node.getNode().getRunningApps().size(), 0);: 'node' has incorrect indentation level 6, expected level should be 8. [Indentation]
            ./hadoop-tools/hadoop-sls/src/test/java/org/apache/hadoop/yarn/sls/nodemanager/TestNMSimulator.java:230:      node.getNode().getRunningApps().size(), 0);: 'node' has incorrect indentation level 6, expected level should be 8. [Indentation]
            
            snemeth Szilard Nemeth added a comment - Hi ananyo_rao , Thanks for working on this. Latest patch LGTM, committed to trunk. One side note: Please set up your code formatter rules in your IDE, I fixed some formatting issues like missing spaces after for/while loop expressions, indentation level fixes for multi-line expressions and such kind of things. If you would like to get help for the rules I can help you offline. For reference, these were the checkstyle issues reported by Jenkins: ./hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/nodemanager/NMSimulator.java:274: && !getNode().getRunningApps().contains(applicationId)) {: '&&' has incorrect indentation level 6, expected level should be 8. [Indentation] ./hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/nodemanager/NMSimulator.java:278: "Updated runningApps on this node are: {}" ,: ' "Updated runningApps on this node are: {}" ' has incorrect indentation level 6, expected level should be 8. [Indentation] ./hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/nodemanager/NMSimulator.java:279: applicationId, getNode().getNodeID(), getNode().getRunningApps());: 'applicationId' has incorrect indentation level 6, expected level should be 8. [Indentation] ./hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/nodemanager/NMSimulator.java:315: applicationId, getNode().getNodeID(), getNode().getRunningApps());: 'applicationId' has incorrect indentation level 8, expected level should be 10. [Indentation] ./hadoop-tools/hadoop-sls/src/test/java/org/apache/hadoop/yarn/sls/appmaster/TestAMSimulator.java:308: app.appId = ApplicationId.newInstance(0l, 1);:43: Should use uppercase 'L' . [UpperEll] ./hadoop-tools/hadoop-sls/src/test/java/org/apache/hadoop/yarn/sls/nodemanager/TestNMSimulator.java:114: node1.addNewContainer(container1, 100000l, null );:39: Should use uppercase 'L' . [UpperEll] ./hadoop-tools/hadoop-sls/src/test/java/org/apache/hadoop/yarn/sls/nodemanager/TestNMSimulator.java:122: node1.addNewContainer(container2, -1l, null );:40: Should use uppercase 'L' . [UpperEll] ./hadoop-tools/hadoop-sls/src/test/java/org/apache/hadoop/yarn/sls/nodemanager/TestNMSimulator.java:147: rm, -1f);: 'rm' has incorrect indentation level 6, expected level should be 8. [Indentation] ./hadoop-tools/hadoop-sls/src/test/java/org/apache/hadoop/yarn/sls/nodemanager/TestNMSimulator.java:160: GenericTestUtils.waitFor( new org.apache.hadoop.thirdparty.com.google.common.base.Supplier< Boolean >() {: Line is longer than 100 characters (found 106). [LineLength] ./hadoop-tools/hadoop-sls/src/test/java/org/apache/hadoop/yarn/sls/nodemanager/TestNMSimulator.java:168: node.getNode().getRunningApps().size(), 0);: 'node' has incorrect indentation level 6, expected level should be 8. [Indentation] ./hadoop-tools/hadoop-sls/src/test/java/org/apache/hadoop/yarn/sls/nodemanager/TestNMSimulator.java:175: Resources.createResource(GB, 1), null , null );: 'Resources' has incorrect indentation level 6, expected level should be 8. [Indentation] ./hadoop-tools/hadoop-sls/src/test/java/org/apache/hadoop/yarn/sls/nodemanager/TestNMSimulator.java:176: node.addNewContainer(container, 100000l, appId);:37: Should use uppercase 'L' . [UpperEll] ./hadoop-tools/hadoop-sls/src/test/java/org/apache/hadoop/yarn/sls/nodemanager/TestNMSimulator.java:179: node.getNode().getRunningApps().contains(appId));: 'node' has incorrect indentation level 6, expected level should be 8. [Indentation] ./hadoop-tools/hadoop-sls/src/test/java/org/apache/hadoop/yarn/sls/nodemanager/TestNMSimulator.java:187: node.getNode().getRunningApps().size(), 0);: 'node' has incorrect indentation level 6, expected level should be 8. [Indentation] ./hadoop-tools/hadoop-sls/src/test/java/org/apache/hadoop/yarn/sls/nodemanager/TestNMSimulator.java:195: rm, -1f);: 'rm' has incorrect indentation level 6, expected level should be 8. [Indentation] ./hadoop-tools/hadoop-sls/src/test/java/org/apache/hadoop/yarn/sls/nodemanager/TestNMSimulator.java:208: GenericTestUtils.waitFor( new org.apache.hadoop.thirdparty.com.google.common.base.Supplier< Boolean >() {: Line is longer than 100 characters (found 106). [LineLength] ./hadoop-tools/hadoop-sls/src/test/java/org/apache/hadoop/yarn/sls/nodemanager/TestNMSimulator.java:216: node.getNode().getRunningApps().size(), 0);: 'node' has incorrect indentation level 6, expected level should be 8. [Indentation] ./hadoop-tools/hadoop-sls/src/test/java/org/apache/hadoop/yarn/sls/nodemanager/TestNMSimulator.java:221: Resources.createResource(GB, 1), null , null );: 'Resources' has incorrect indentation level 6, expected level should be 8. [Indentation] ./hadoop-tools/hadoop-sls/src/test/java/org/apache/hadoop/yarn/sls/nodemanager/TestNMSimulator.java:222: node.addNewContainer(container, 100000l, null );:37: Should use uppercase 'L' . [UpperEll] ./hadoop-tools/hadoop-sls/src/test/java/org/apache/hadoop/yarn/sls/nodemanager/TestNMSimulator.java:224: node.getNode().getRunningApps().size(), 0);: 'node' has incorrect indentation level 6, expected level should be 8. [Indentation] ./hadoop-tools/hadoop-sls/src/test/java/org/apache/hadoop/yarn/sls/nodemanager/TestNMSimulator.java:230: node.getNode().getRunningApps().size(), 0);: 'node' has incorrect indentation level 6, expected level should be 8. [Indentation]
            stevel@apache.org Steve Loughran added a comment -

            FYI. there's refs to com.google in the code which MUST be org.apache.thirdparty.com.google; you do a build right now and maven ends up patching TestNMSimulator

            stevel@apache.org Steve Loughran added a comment - FYI. there's refs to com.google in the code which MUST be org.apache.thirdparty.com.google; you do a build right now and maven ends up patching TestNMSimulator
            snemeth Szilard Nemeth added a comment -

            Hi stevel@apache.org,
            Thanks for reporting this.
            Do you suggest a follow-up jira to fix those imports?

            snemeth Szilard Nemeth added a comment - Hi stevel@apache.org , Thanks for reporting this. Do you suggest a follow-up jira to fix those imports?
            stevel@apache.org Steve Loughran added a comment - YARN-10878

            People

              ananyo_rao VADAGA ANANYO RAO
              ananyo_rao VADAGA ANANYO RAO
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: