Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-5375

invoke MockRM#drainEvents implicitly in MockRM methods to reduce test failures

    Details

    • Hadoop Flags:
      Reviewed

      Description

      seen many test failures related to RMApp/RMAppattempt comes to some state but some event are not processed in rm event queue or scheduler event queue, cause test failure, seems we could implicitly invokes drainEvents(should also drain sheduler event) in some mockRM method like waitForState

      1. YARN-5375.01.patch
        33 kB
        sandflee
      2. YARN-5375.03.patch
        39 kB
        sandflee
      3. YARN-5375.04.patch
        33 kB
        sandflee
      4. YARN-5375.05.patch
        36 kB
        sandflee
      5. YARN-5375.06.patch
        36 kB
        sandflee
      6. YARN-5375.07-drain-statestore.patch
        38 kB
        sandflee
      7. YARN-5375.07-sync-statestore.patch
        39 kB
        sandflee
      8. YARN-5375.08.patch
        30 kB
        sandflee
      9. YARN-5375.09.patch
        30 kB
        sandflee
      10. YARN-5375.10.patch
        30 kB
        Rohith Sharma K S
      11. YARN-5375.11.patch
        34 kB
        sandflee
      12. YARN-5375.12.new.patch
        34 kB
        sandflee
      13. YARN-5375.12.patch
        34 kB
        sandflee

        Issue Links

          Activity

          Hide
          sandflee sandflee added a comment -

          1, add drainSchedulerEvents and drainStateStoreEvents to drainEvents
          2, add drainEvents to almost all MockRM methods
          3, fix test compile/runtime failure that conficts with the patch

          Show
          sandflee sandflee added a comment - 1, add drainSchedulerEvents and drainStateStoreEvents to drainEvents 2, add drainEvents to almost all MockRM methods 3, fix test compile/runtime failure that conficts with the patch
          Hide
          sandflee sandflee added a comment -

          there still had a very little chance that events are left in eventQueue event if drainEvents returned. Thought not related to this issue, if drainEvents could granted that all event are processed completed, we could use drainEvents in waitForState to replace wait-sleep way. like

          waitForState(app, state) {
             drainEvents()
             assert.equals(app.state, state)
          }
          

          this may reduce the test times, and wondered whether we should do that.

          Show
          sandflee sandflee added a comment - there still had a very little chance that events are left in eventQueue event if drainEvents returned. Thought not related to this issue, if drainEvents could granted that all event are processed completed, we could use drainEvents in waitForState to replace wait-sleep way. like waitForState(app, state) { drainEvents() assert .equals(app.state, state) } this may reduce the test times, and wondered whether we should do that.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 4s YARN-5375 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          Subsystem Report/Notes
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12818452/YARN-5375.01.patch
          JIRA Issue YARN-5375
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/12349/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 4s YARN-5375 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12818452/YARN-5375.01.patch JIRA Issue YARN-5375 Console output https://builds.apache.org/job/PreCommit-YARN-Build/12349/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 29s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 11 new or modified test files.
          0 mvndep 0m 9s Maven dependency ordering for branch
          +1 mvninstall 6m 41s trunk passed
          +1 compile 2m 17s trunk passed
          +1 checkstyle 0m 41s trunk passed
          +1 mvnsite 1m 8s trunk passed
          +1 mvneclipse 0m 30s trunk passed
          +1 findbugs 1m 49s trunk passed
          +1 javadoc 0m 47s trunk passed
          0 mvndep 0m 9s Maven dependency ordering for patch
          +1 mvninstall 0m 56s the patch passed
          -1 compile 1m 49s hadoop-yarn in the patch failed.
          -1 javac 1m 49s hadoop-yarn in the patch failed.
          -1 checkstyle 0m 40s hadoop-yarn-project/hadoop-yarn: The patch generated 14 new + 351 unchanged - 4 fixed = 365 total (was 355)
          +1 mvnsite 1m 3s the patch passed
          +1 mvneclipse 0m 26s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 3s the patch passed
          +1 javadoc 0m 44s the patch passed
          +1 unit 2m 13s hadoop-yarn-common in the patch passed.
          -1 unit 51m 4s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          76m 44s



          Reason Tests
          Timed out junit tests org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12818464/YARN-5375.02.patch
          JIRA Issue YARN-5375
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 63e6b0fb68ae 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 5b4a708
          Default Java 1.8.0_91
          findbugs v3.0.0
          compile https://builds.apache.org/job/PreCommit-YARN-Build/12350/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn.txt
          javac https://builds.apache.org/job/PreCommit-YARN-Build/12350/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn.txt
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12350/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/12350/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/12350/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12350/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/12350/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 29s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 11 new or modified test files. 0 mvndep 0m 9s Maven dependency ordering for branch +1 mvninstall 6m 41s trunk passed +1 compile 2m 17s trunk passed +1 checkstyle 0m 41s trunk passed +1 mvnsite 1m 8s trunk passed +1 mvneclipse 0m 30s trunk passed +1 findbugs 1m 49s trunk passed +1 javadoc 0m 47s trunk passed 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 0m 56s the patch passed -1 compile 1m 49s hadoop-yarn in the patch failed. -1 javac 1m 49s hadoop-yarn in the patch failed. -1 checkstyle 0m 40s hadoop-yarn-project/hadoop-yarn: The patch generated 14 new + 351 unchanged - 4 fixed = 365 total (was 355) +1 mvnsite 1m 3s the patch passed +1 mvneclipse 0m 26s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 3s the patch passed +1 javadoc 0m 44s the patch passed +1 unit 2m 13s hadoop-yarn-common in the patch passed. -1 unit 51m 4s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 76m 44s Reason Tests Timed out junit tests org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12818464/YARN-5375.02.patch JIRA Issue YARN-5375 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 63e6b0fb68ae 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 5b4a708 Default Java 1.8.0_91 findbugs v3.0.0 compile https://builds.apache.org/job/PreCommit-YARN-Build/12350/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn.txt javac https://builds.apache.org/job/PreCommit-YARN-Build/12350/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12350/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/12350/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/12350/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12350/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/12350/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          sandflee sandflee added a comment - - edited

          seems we could use a sync way to process scheduler event , this will simple the process and code. Thought?

              @Override
              protected EventHandler<SchedulerEvent> createSchedulerEventDispatcher() {
                // Dispatch inline for test sanity
                return new EventHandler<SchedulerEvent>() {
                  @Override
                  public void handle(SchedulerEvent event) {
                    scheduler.handle(event);
                  }
                };
              }
          
          Show
          sandflee sandflee added a comment - - edited seems we could use a sync way to process scheduler event , this will simple the process and code. Thought? @Override protected EventHandler<SchedulerEvent> createSchedulerEventDispatcher() { // Dispatch inline for test sanity return new EventHandler<SchedulerEvent>() { @Override public void handle(SchedulerEvent event) { scheduler.handle(event); } }; }
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 23s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 13 new or modified test files.
          0 mvndep 0m 43s Maven dependency ordering for branch
          +1 mvninstall 6m 53s trunk passed
          +1 compile 7m 20s trunk passed
          +1 checkstyle 1m 35s trunk passed
          +1 mvnsite 2m 7s trunk passed
          +1 mvneclipse 1m 2s trunk passed
          +1 findbugs 3m 41s trunk passed
          +1 javadoc 1m 38s trunk passed
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvninstall 1m 59s the patch passed
          +1 compile 8m 54s the patch passed
          -1 javac 8m 54s root generated 1 new + 709 unchanged - 0 fixed = 710 total (was 709)
          -1 checkstyle 1m 40s root: The patch generated 3 new + 512 unchanged - 5 fixed = 515 total (was 517)
          +1 mvnsite 2m 12s the patch passed
          +1 mvneclipse 1m 1s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 37s the patch passed
          +1 javadoc 1m 27s the patch passed
          +1 unit 2m 20s hadoop-yarn-common in the patch passed.
          -1 unit 51m 29s hadoop-yarn-server-resourcemanager in the patch failed.
          -1 unit 8m 18s hadoop-yarn-client in the patch failed.
          -1 unit 8m 39s hadoop-mapreduce-client-app in the patch failed.
          +1 asflicense 0m 22s The patch does not generate ASF License warnings.
          118m 54s



          Reason Tests
          Failed junit tests hadoop.yarn.client.cli.TestLogsCLI
            hadoop.yarn.client.api.impl.TestYarnClient
            hadoop.mapreduce.v2.app.rm.TestRMContainerAllocator
          Timed out junit tests org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12818502/YARN-5375.03.patch
          JIRA Issue YARN-5375
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux f13388b3d46a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 5b4a708
          Default Java 1.8.0_91
          findbugs v3.0.0
          javac https://builds.apache.org/job/PreCommit-YARN-Build/12353/artifact/patchprocess/diff-compile-javac-root.txt
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12353/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/12353/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/12353/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/12353/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/12353/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt https://builds.apache.org/job/PreCommit-YARN-Build/12353/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt https://builds.apache.org/job/PreCommit-YARN-Build/12353/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12353/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/12353/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 23s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 13 new or modified test files. 0 mvndep 0m 43s Maven dependency ordering for branch +1 mvninstall 6m 53s trunk passed +1 compile 7m 20s trunk passed +1 checkstyle 1m 35s trunk passed +1 mvnsite 2m 7s trunk passed +1 mvneclipse 1m 2s trunk passed +1 findbugs 3m 41s trunk passed +1 javadoc 1m 38s trunk passed 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 1m 59s the patch passed +1 compile 8m 54s the patch passed -1 javac 8m 54s root generated 1 new + 709 unchanged - 0 fixed = 710 total (was 709) -1 checkstyle 1m 40s root: The patch generated 3 new + 512 unchanged - 5 fixed = 515 total (was 517) +1 mvnsite 2m 12s the patch passed +1 mvneclipse 1m 1s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 37s the patch passed +1 javadoc 1m 27s the patch passed +1 unit 2m 20s hadoop-yarn-common in the patch passed. -1 unit 51m 29s hadoop-yarn-server-resourcemanager in the patch failed. -1 unit 8m 18s hadoop-yarn-client in the patch failed. -1 unit 8m 39s hadoop-mapreduce-client-app in the patch failed. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 118m 54s Reason Tests Failed junit tests hadoop.yarn.client.cli.TestLogsCLI   hadoop.yarn.client.api.impl.TestYarnClient   hadoop.mapreduce.v2.app.rm.TestRMContainerAllocator Timed out junit tests org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12818502/YARN-5375.03.patch JIRA Issue YARN-5375 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux f13388b3d46a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 5b4a708 Default Java 1.8.0_91 findbugs v3.0.0 javac https://builds.apache.org/job/PreCommit-YARN-Build/12353/artifact/patchprocess/diff-compile-javac-root.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12353/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/12353/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/12353/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/12353/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/12353/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt https://builds.apache.org/job/PreCommit-YARN-Build/12353/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt https://builds.apache.org/job/PreCommit-YARN-Build/12353/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12353/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/12353/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          sunilg Sunil G added a comment -

          Hi sandflee
          Thanks for the effort. Its a good initiative.

          This patch exposes a new api in RMStateStore to set dispatcher. I think its better and clean to have a mock class extended from RMStateStore, and override to set dispatcher. YARN-5043 has some done a similar effort for state store. I think there is bit of duplicate effort only for statestore changes with YARN-5043. I suggest you can check that point. cc/Jun Gong

          Show
          sunilg Sunil G added a comment - Hi sandflee Thanks for the effort. Its a good initiative. This patch exposes a new api in RMStateStore to set dispatcher. I think its better and clean to have a mock class extended from RMStateStore, and override to set dispatcher. YARN-5043 has some done a similar effort for state store. I think there is bit of duplicate effort only for statestore changes with YARN-5043 . I suggest you can check that point. cc/ Jun Gong
          Hide
          sandflee sandflee added a comment -

          thanks for your suggestion, yes, It's more clean to use a MockRMStateStore, but there is a problem, many RMStateStore are created out of MockRM , especially for RM restart test. Thought?

          Show
          sandflee sandflee added a comment - thanks for your suggestion, yes, It's more clean to use a MockRMStateStore, but there is a problem, many RMStateStore are created out of MockRM , especially for RM restart test. Thought?
          Hide
          sandflee sandflee added a comment -

          or could we change the dispatcher of MemoryStateStore to drainDispatcher? this will reduce the change and simple the design.

          Show
          sandflee sandflee added a comment - or could we change the dispatcher of MemoryStateStore to drainDispatcher? this will reduce the change and simple the design.
          Hide
          sandflee sandflee added a comment - - edited

          update the patch
          1, drop scheduler async event handler, use a sync way
          2, use RMStateStore.dispatcher#isDrained to drain rmStateStore events, not using a MockRMStateStore because RMStateStore are created outside of MockRM in many test case.

          Show
          sandflee sandflee added a comment - - edited update the patch 1, drop scheduler async event handler, use a sync way 2, use RMStateStore.dispatcher#isDrained to drain rmStateStore events, not using a MockRMStateStore because RMStateStore are created outside of MockRM in many test case.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 23s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 9 new or modified test files.
          0 mvndep 0m 11s Maven dependency ordering for branch
          +1 mvninstall 7m 7s trunk passed
          +1 compile 2m 17s trunk passed
          +1 checkstyle 0m 41s trunk passed
          +1 mvnsite 1m 7s trunk passed
          +1 mvneclipse 0m 29s trunk passed
          +1 findbugs 1m 53s trunk passed
          +1 javadoc 0m 47s trunk passed
          0 mvndep 0m 9s Maven dependency ordering for patch
          +1 mvninstall 0m 54s the patch passed
          +1 compile 2m 13s the patch passed
          +1 javac 2m 13s the patch passed
          -1 checkstyle 0m 39s hadoop-yarn-project/hadoop-yarn: The patch generated 1 new + 261 unchanged - 3 fixed = 262 total (was 264)
          +1 mvnsite 1m 3s the patch passed
          +1 mvneclipse 0m 25s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 28s the patch passed
          +1 javadoc 0m 43s the patch passed
          +1 unit 2m 15s hadoop-yarn-common in the patch passed.
          -1 unit 47m 25s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          75m 20s



          Reason Tests
          Timed out junit tests org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler
            org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12819745/YARN-5375.04.patch
          JIRA Issue YARN-5375
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux d3b7e23e0704 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / c6e3a00
          Default Java 1.8.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12474/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/12474/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/12474/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12474/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/12474/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 23s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 9 new or modified test files. 0 mvndep 0m 11s Maven dependency ordering for branch +1 mvninstall 7m 7s trunk passed +1 compile 2m 17s trunk passed +1 checkstyle 0m 41s trunk passed +1 mvnsite 1m 7s trunk passed +1 mvneclipse 0m 29s trunk passed +1 findbugs 1m 53s trunk passed +1 javadoc 0m 47s trunk passed 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 0m 54s the patch passed +1 compile 2m 13s the patch passed +1 javac 2m 13s the patch passed -1 checkstyle 0m 39s hadoop-yarn-project/hadoop-yarn: The patch generated 1 new + 261 unchanged - 3 fixed = 262 total (was 264) +1 mvnsite 1m 3s the patch passed +1 mvneclipse 0m 25s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 28s the patch passed +1 javadoc 0m 43s the patch passed +1 unit 2m 15s hadoop-yarn-common in the patch passed. -1 unit 47m 25s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 75m 20s Reason Tests Timed out junit tests org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler   org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12819745/YARN-5375.04.patch JIRA Issue YARN-5375 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux d3b7e23e0704 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c6e3a00 Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12474/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/12474/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/12474/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12474/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/12474/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          sandflee sandflee added a comment - - edited

          update the patch to fix test failures.

          org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler

          scheduler event handler throws exceptions cause test failed, disable Dispatcher.DISPATCHER_EXIT_ON_ERROR_KEY in drainDispatcher like Scheduler event dispatcher.

          org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes

          MockRM is not started, so waitForState will be blocked, just start the rm.

          Show
          sandflee sandflee added a comment - - edited update the patch to fix test failures. org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler scheduler event handler throws exceptions cause test failed, disable Dispatcher.DISPATCHER_EXIT_ON_ERROR_KEY in drainDispatcher like Scheduler event dispatcher. org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes MockRM is not started, so waitForState will be blocked, just start the rm.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 11 new or modified test files.
          0 mvndep 0m 9s Maven dependency ordering for branch
          +1 mvninstall 6m 42s trunk passed
          +1 compile 2m 17s trunk passed
          +1 checkstyle 0m 41s trunk passed
          +1 mvnsite 1m 8s trunk passed
          +1 mvneclipse 0m 30s trunk passed
          +1 findbugs 1m 50s trunk passed
          +1 javadoc 0m 47s trunk passed
          0 mvndep 0m 9s Maven dependency ordering for patch
          +1 mvninstall 0m 56s the patch passed
          +1 compile 2m 14s the patch passed
          +1 javac 2m 14s the patch passed
          -1 checkstyle 0m 38s hadoop-yarn-project/hadoop-yarn: The patch generated 1 new + 270 unchanged - 3 fixed = 271 total (was 273)
          +1 mvnsite 1m 3s the patch passed
          +1 mvneclipse 0m 26s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 32s the patch passed
          +1 javadoc 0m 51s the patch passed
          +1 unit 2m 32s hadoop-yarn-common in the patch passed.
          +1 unit 36m 29s hadoop-yarn-server-resourcemanager in the patch passed.
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          63m 14s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12820730/YARN-5375.05.patch
          JIRA Issue YARN-5375
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 9e46bec91aed 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 7f3c306
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12539/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12539/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/12539/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 11 new or modified test files. 0 mvndep 0m 9s Maven dependency ordering for branch +1 mvninstall 6m 42s trunk passed +1 compile 2m 17s trunk passed +1 checkstyle 0m 41s trunk passed +1 mvnsite 1m 8s trunk passed +1 mvneclipse 0m 30s trunk passed +1 findbugs 1m 50s trunk passed +1 javadoc 0m 47s trunk passed 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 0m 56s the patch passed +1 compile 2m 14s the patch passed +1 javac 2m 14s the patch passed -1 checkstyle 0m 38s hadoop-yarn-project/hadoop-yarn: The patch generated 1 new + 270 unchanged - 3 fixed = 271 total (was 273) +1 mvnsite 1m 3s the patch passed +1 mvneclipse 0m 26s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 32s the patch passed +1 javadoc 0m 51s the patch passed +1 unit 2m 32s hadoop-yarn-common in the patch passed. +1 unit 36m 29s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 63m 14s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12820730/YARN-5375.05.patch JIRA Issue YARN-5375 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 9e46bec91aed 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 7f3c306 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12539/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12539/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/12539/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          varun_saxena Varun Saxena added a comment -

          sandflee, after YARN-5436, the patch here doesnt apply cleanly.
          Can you rebase it ?

          Show
          varun_saxena Varun Saxena added a comment - sandflee , after YARN-5436 , the patch here doesnt apply cleanly. Can you rebase it ?
          Hide
          sandflee sandflee added a comment -

          1, replace rmStateStore#AsyncDispatcher with DrainDispatcher, yes the code is not clean, welcome suggestions!
          2, set DrainDispacher#isDrain default value to true, and sleep a while if not drained to reduce cpu usage.
          3, not invoke setDrainEventsOnStop in RMStateStore#DrainDispatcher creation , for DrainDispacher will take 300s to stop if enabled setDrainEventsOnStop, file YARN-5526 to track.

          Show
          sandflee sandflee added a comment - 1, replace rmStateStore#AsyncDispatcher with DrainDispatcher, yes the code is not clean, welcome suggestions! 2, set DrainDispacher#isDrain default value to true, and sleep a while if not drained to reduce cpu usage. 3, not invoke setDrainEventsOnStop in RMStateStore#DrainDispatcher creation , for DrainDispacher will take 300s to stop if enabled setDrainEventsOnStop, file YARN-5526 to track.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 11 new or modified test files.
          0 mvndep 0m 9s Maven dependency ordering for branch
          +1 mvninstall 7m 7s trunk passed
          +1 compile 2m 26s trunk passed
          +1 checkstyle 0m 41s trunk passed
          +1 mvnsite 1m 10s trunk passed
          +1 mvneclipse 0m 31s trunk passed
          +1 findbugs 1m 58s trunk passed
          +1 javadoc 0m 48s trunk passed
          0 mvndep 0m 11s Maven dependency ordering for patch
          +1 mvninstall 0m 57s the patch passed
          +1 compile 2m 23s the patch passed
          +1 javac 2m 23s the patch passed
          +1 checkstyle 0m 39s hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 259 unchanged - 3 fixed = 259 total (was 262)
          +1 mvnsite 1m 7s the patch passed
          +1 mvneclipse 0m 27s the patch passed
          +1 whitespace 0m 1s The patch has no whitespace issues.
          +1 findbugs 2m 11s the patch passed
          +1 javadoc 0m 45s the patch passed
          -1 unit 2m 18s hadoop-yarn-common in the patch failed.
          -1 unit 37m 54s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          65m 10s



          Reason Tests
          Failed junit tests hadoop.yarn.logaggregation.TestAggregatedLogFormat
            hadoop.yarn.server.resourcemanager.TestRMRestart



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12823885/YARN-5375.06.patch
          JIRA Issue YARN-5375
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 24822e76236f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / ffe1fff
          Default Java 1.8.0_101
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-YARN-Build/12784/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/12784/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/12784/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt https://builds.apache.org/job/PreCommit-YARN-Build/12784/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12784/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/12784/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 11 new or modified test files. 0 mvndep 0m 9s Maven dependency ordering for branch +1 mvninstall 7m 7s trunk passed +1 compile 2m 26s trunk passed +1 checkstyle 0m 41s trunk passed +1 mvnsite 1m 10s trunk passed +1 mvneclipse 0m 31s trunk passed +1 findbugs 1m 58s trunk passed +1 javadoc 0m 48s trunk passed 0 mvndep 0m 11s Maven dependency ordering for patch +1 mvninstall 0m 57s the patch passed +1 compile 2m 23s the patch passed +1 javac 2m 23s the patch passed +1 checkstyle 0m 39s hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 259 unchanged - 3 fixed = 259 total (was 262) +1 mvnsite 1m 7s the patch passed +1 mvneclipse 0m 27s the patch passed +1 whitespace 0m 1s The patch has no whitespace issues. +1 findbugs 2m 11s the patch passed +1 javadoc 0m 45s the patch passed -1 unit 2m 18s hadoop-yarn-common in the patch failed. -1 unit 37m 54s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 65m 10s Reason Tests Failed junit tests hadoop.yarn.logaggregation.TestAggregatedLogFormat   hadoop.yarn.server.resourcemanager.TestRMRestart Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12823885/YARN-5375.06.patch JIRA Issue YARN-5375 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 24822e76236f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ffe1fff Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/12784/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/12784/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/12784/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt https://builds.apache.org/job/PreCommit-YARN-Build/12784/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12784/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/12784/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          varun_saxena Varun Saxena added a comment -

          I agree with Sunil here. Too much of code change in main class for tests is not desirable.
          I think we can create a subclass for MemoryRMStateStore (i.e. what almost all the test cases use) which uses DrainDispatcher.
          This will increase code change but should be only one line change in the caller (replace MemoryRMStateStore by the new class).

          Show
          varun_saxena Varun Saxena added a comment - I agree with Sunil here. Too much of code change in main class for tests is not desirable. I think we can create a subclass for MemoryRMStateStore (i.e. what almost all the test cases use) which uses DrainDispatcher. This will increase code change but should be only one line change in the caller (replace MemoryRMStateStore by the new class).
          Hide
          sandflee sandflee added a comment -

          Thanks Varun Saxena, yes this will reduce the change for main class and much cleaner, thought? Sunil GRohith Sharma K S

          Show
          sandflee sandflee added a comment - Thanks Varun Saxena , yes this will reduce the change for main class and much cleaner, thought? Sunil G Rohith Sharma K S
          Hide
          sunilg Sunil G added a comment -

          Thanks sandflee
          Sorry for coming in late.
          +1 for the approach suggested by Varun and you. Eventhough code is more, in log run this will be helpful to use. This also will help MockRM to be almost same as now.

          Show
          sunilg Sunil G added a comment - Thanks sandflee Sorry for coming in late. +1 for the approach suggested by Varun and you. Eventhough code is more, in log run this will be helpful to use. This also will help MockRM to be almost same as now.
          Hide
          sunilg Sunil G added a comment -

          Thanks sandflee
          Sorry for coming in late.
          +1 for the approach suggested by Varun and you. Eventhough code is more, in log run this will be helpful to use. This also will help MockRM to be almost same as now.

          Show
          sunilg Sunil G added a comment - Thanks sandflee Sorry for coming in late. +1 for the approach suggested by Varun and you. Eventhough code is more, in log run this will be helpful to use. This also will help MockRM to be almost same as now.
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          Thanks sandflee for working on this. One doubt is do we see any test case failure is because RMStateStore asyncDispatcher?
          AFAIK random ness was seen many places where in scheduler-dispatcher handling and in RM dispatcher.

          Show
          rohithsharma Rohith Sharma K S added a comment - Thanks sandflee for working on this. One doubt is do we see any test case failure is because RMStateStore asyncDispatcher? AFAIK random ness was seen many places where in scheduler-dispatcher handling and in RM dispatcher.
          Hide
          varun_saxena Varun Saxena added a comment -

          Yes, refer to YARN-5548 for instance.
          That seems to be due to state store dispatcher.

          Show
          varun_saxena Varun Saxena added a comment - Yes, refer to YARN-5548 for instance. That seems to be due to state store dispatcher.
          Hide
          sandflee sandflee added a comment -

          also see YARN-5043, if StateStore Event is not processed, more likely it will produce another RM Event.

          Show
          sandflee sandflee added a comment - also see YARN-5043 , if StateStore Event is not processed, more likely it will produce another RM Event.
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          I am fine going with subclass of memoryRMStateStore and it can be used everywhere.
          One comment from earlier patch in MockRM is need to take care whenever drain for Stastore dispatcher, again need to wait for draining rm-dispatcher. This is required because state store trigger another event to rm-dispatcher.

          And another approach I was thinking to override the dispatcher in RMStateStore while creating itself. But issue with that state-store is created using reflection where I could not override methods in RMstatestore.

          Show
          rohithsharma Rohith Sharma K S added a comment - I am fine going with subclass of memoryRMStateStore and it can be used everywhere. One comment from earlier patch in MockRM is need to take care whenever drain for Stastore dispatcher, again need to wait for draining rm-dispatcher. This is required because state store trigger another event to rm-dispatcher. And another approach I was thinking to override the dispatcher in RMStateStore while creating itself. But issue with that state-store is created using reflection where I could not override methods in RMstatestore.
          Hide
          sandflee sandflee added a comment -

          One comment from earlier patch in MockRM is need to take care whenever drain for Stastore dispatcher, again need to wait for draining rm-dispatcher. This is required because state store trigger another event to rm-dispatcher.

          yes, we should take care of this. but double drain drain rm-dispatcher may not help, because rm state store may have new Event again. One approach is to add a timestamp to drainDispatcher to record newEvent add time, after drain StateStore event, we could check the timestamp of rm-disptacher, if not updated, we could make sure that all events are drained.

          Show
          sandflee sandflee added a comment - One comment from earlier patch in MockRM is need to take care whenever drain for Stastore dispatcher, again need to wait for draining rm-dispatcher. This is required because state store trigger another event to rm-dispatcher. yes, we should take care of this. but double drain drain rm-dispatcher may not help, because rm state store may have new Event again. One approach is to add a timestamp to drainDispatcher to record newEvent add time, after drain StateStore event, we could check the timestamp of rm-disptacher, if not updated, we could make sure that all events are drained.
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          because rm state store may have new Event again

          I think it should be fine since these are new events. Another way is if we drain RMStateStore#dispatcher first would solve the issue.

          Show
          rohithsharma Rohith Sharma K S added a comment - because rm state store may have new Event again I think it should be fine since these are new events. Another way is if we drain RMStateStore#dispatcher first would solve the issue.
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          I think the order of rmDispatcher.await --> stateStoreDispatcher --> rmDispatcher would be more accurate.

          Show
          rohithsharma Rohith Sharma K S added a comment - I think the order of rmDispatcher.await --> stateStoreDispatcher --> rmDispatcher would be more accurate.
          Hide
          sandflee sandflee added a comment -

          thanks Rohith Sharma K S, even if we using the order rmDispatcher.await --> stateStoreDispatcher --> rmDispatcher , there are still very very little change that events are not processed in stateStoreDispatcher or rmDispatcher, agree?
          I prefer a accurate way to really drain all events if MockRM#drainEvents returned. based on this we could use drainEvents to replace sleep-wait way in MockRM#waitForState, this may help to reduce test time, thought?

          Show
          sandflee sandflee added a comment - thanks Rohith Sharma K S , even if we using the order rmDispatcher.await --> stateStoreDispatcher --> rmDispatcher , there are still very very little change that events are not processed in stateStoreDispatcher or rmDispatcher, agree? I prefer a accurate way to really drain all events if MockRM#drainEvents returned. based on this we could use drainEvents to replace sleep-wait way in MockRM#waitForState, this may help to reduce test time, thought?
          Hide
          varun_saxena Varun Saxena added a comment -

          There are many events where draining of both rm dispatcher and state store dispatcher is not required and as Rohith pointed out in some places, we may need to drain RM Dispatcher after state store dispatcher drain as well. There could be different sequences as well. Should we create 2 drain methods and call each one as per need ?

          Also many of the waitForState can be eliminated by calling a sequence of drains. For different kind of events, we may have to decide what do we do.
          Maybe for some regularly checked states, we can add some specific methods in MockRM in addition to generic waitForState methods with customary draining of RM dispatcher and state store dispatcher and again RM dispatcher. This as you said can reduce test times.
          Thoughts ?

          For instance, after Node HB we typically check if an attempt has transferred to ALLOCATED state in many of the tests. Here, first rm dispatcher must be drained to process node update to be sent to Scheduler. As with the changes in patch, scheduler events will be processed immediately, it will lead to App Attempt event for Container being allocated (from RM Container). This would again require draining of RM dispatcher queue which would lead to a state store event and from State store we will again send a ATTEMPT_SAVED event back to RM App attempt, which would again require draining of RM dispatcher queue. And only then will attempt state change to ALLOCATED.

          Show
          varun_saxena Varun Saxena added a comment - There are many events where draining of both rm dispatcher and state store dispatcher is not required and as Rohith pointed out in some places, we may need to drain RM Dispatcher after state store dispatcher drain as well. There could be different sequences as well. Should we create 2 drain methods and call each one as per need ? Also many of the waitForState can be eliminated by calling a sequence of drains. For different kind of events, we may have to decide what do we do. Maybe for some regularly checked states, we can add some specific methods in MockRM in addition to generic waitForState methods with customary draining of RM dispatcher and state store dispatcher and again RM dispatcher. This as you said can reduce test times. Thoughts ? For instance, after Node HB we typically check if an attempt has transferred to ALLOCATED state in many of the tests. Here, first rm dispatcher must be drained to process node update to be sent to Scheduler. As with the changes in patch, scheduler events will be processed immediately, it will lead to App Attempt event for Container being allocated (from RM Container). This would again require draining of RM dispatcher queue which would lead to a state store event and from State store we will again send a ATTEMPT_SAVED event back to RM App attempt, which would again require draining of RM dispatcher queue. And only then will attempt state change to ALLOCATED.
          Hide
          sunilg Sunil G added a comment -

          Nice catch Rohith Sharma K S.

          Overall i feel we need to only drain rm dispatcher and the state store for most of the cases. But when we have a saving transition as explained by Varun Saxena in detail, we can expect a piggy back event from state store to rm dispatcher.

          Could we try a waitForState only for this case?. Yes, it will not be clean and may little more expensive in terms of sleep time, but code may be simpler to understand. Thoughts?

          Show
          sunilg Sunil G added a comment - Nice catch Rohith Sharma K S . Overall i feel we need to only drain rm dispatcher and the state store for most of the cases. But when we have a saving transition as explained by Varun Saxena in detail, we can expect a piggy back event from state store to rm dispatcher. Could we try a waitForState only for this case?. Yes, it will not be clean and may little more expensive in terms of sleep time, but code may be simpler to understand. Thoughts?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 21s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 11 new or modified test files.
          0 mvndep 0m 9s Maven dependency ordering for branch
          +1 mvninstall 6m 45s trunk passed
          +1 compile 2m 18s trunk passed
          +1 checkstyle 0m 40s trunk passed
          +1 mvnsite 1m 8s trunk passed
          +1 mvneclipse 0m 30s trunk passed
          +1 findbugs 1m 52s trunk passed
          +1 javadoc 0m 48s trunk passed
          0 mvndep 0m 9s Maven dependency ordering for patch
          +1 mvninstall 0m 56s the patch passed
          +1 compile 2m 14s the patch passed
          +1 javac 2m 14s the patch passed
          -1 checkstyle 0m 39s hadoop-yarn-project/hadoop-yarn: The patch generated 3 new + 259 unchanged - 3 fixed = 262 total (was 262)
          +1 mvnsite 1m 4s the patch passed
          +1 mvneclipse 0m 26s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 5s the patch passed
          +1 javadoc 0m 45s the patch passed
          +1 unit 2m 16s hadoop-yarn-common in the patch passed.
          +1 unit 38m 25s hadoop-yarn-server-resourcemanager in the patch passed.
          +1 asflicense 0m 19s The patch does not generate ASF License warnings.
          64m 39s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12825909/YARN-5375.07-drain-statestore.patch
          JIRA Issue YARN-5375
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 27522beffd0e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / c258171
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12916/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12916/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/12916/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 11 new or modified test files. 0 mvndep 0m 9s Maven dependency ordering for branch +1 mvninstall 6m 45s trunk passed +1 compile 2m 18s trunk passed +1 checkstyle 0m 40s trunk passed +1 mvnsite 1m 8s trunk passed +1 mvneclipse 0m 30s trunk passed +1 findbugs 1m 52s trunk passed +1 javadoc 0m 48s trunk passed 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 0m 56s the patch passed +1 compile 2m 14s the patch passed +1 javac 2m 14s the patch passed -1 checkstyle 0m 39s hadoop-yarn-project/hadoop-yarn: The patch generated 3 new + 259 unchanged - 3 fixed = 262 total (was 262) +1 mvnsite 1m 4s the patch passed +1 mvneclipse 0m 26s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 5s the patch passed +1 javadoc 0m 45s the patch passed +1 unit 2m 16s hadoop-yarn-common in the patch passed. +1 unit 38m 25s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 64m 39s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12825909/YARN-5375.07-drain-statestore.patch JIRA Issue YARN-5375 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 27522beffd0e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c258171 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12916/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12916/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/12916/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 11 new or modified test files.
          0 mvndep 0m 9s Maven dependency ordering for branch
          +1 mvninstall 6m 39s trunk passed
          +1 compile 2m 16s trunk passed
          +1 checkstyle 0m 41s trunk passed
          +1 mvnsite 1m 8s trunk passed
          +1 mvneclipse 0m 29s trunk passed
          +1 findbugs 1m 52s trunk passed
          +1 javadoc 0m 47s trunk passed
          0 mvndep 0m 9s Maven dependency ordering for patch
          +1 mvninstall 0m 55s the patch passed
          +1 compile 2m 14s the patch passed
          +1 javac 2m 14s the patch passed
          -1 checkstyle 0m 38s hadoop-yarn-project/hadoop-yarn: The patch generated 2 new + 257 unchanged - 5 fixed = 259 total (was 262)
          +1 mvnsite 1m 5s the patch passed
          +1 mvneclipse 0m 26s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 31s the patch passed
          +1 javadoc 0m 53s the patch passed
          +1 unit 2m 29s hadoop-yarn-common in the patch passed.
          +1 unit 35m 15s hadoop-yarn-server-resourcemanager in the patch passed.
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          62m 3s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12825937/YARN-5375.07-sync-statestore.patch
          JIRA Issue YARN-5375
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 62dd3488b850 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / c258171
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12923/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12923/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/12923/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 11 new or modified test files. 0 mvndep 0m 9s Maven dependency ordering for branch +1 mvninstall 6m 39s trunk passed +1 compile 2m 16s trunk passed +1 checkstyle 0m 41s trunk passed +1 mvnsite 1m 8s trunk passed +1 mvneclipse 0m 29s trunk passed +1 findbugs 1m 52s trunk passed +1 javadoc 0m 47s trunk passed 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 0m 55s the patch passed +1 compile 2m 14s the patch passed +1 javac 2m 14s the patch passed -1 checkstyle 0m 38s hadoop-yarn-project/hadoop-yarn: The patch generated 2 new + 257 unchanged - 5 fixed = 259 total (was 262) +1 mvnsite 1m 5s the patch passed +1 mvneclipse 0m 26s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 31s the patch passed +1 javadoc 0m 53s the patch passed +1 unit 2m 29s hadoop-yarn-common in the patch passed. +1 unit 35m 15s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 62m 3s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12825937/YARN-5375.07-sync-statestore.patch JIRA Issue YARN-5375 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 62dd3488b850 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c258171 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12923/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12923/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/12923/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          sandflee sandflee added a comment - - edited

          thanks Varun Saxena, Sunil G, Rohith Sharma K S for your comment and suggest ! , update two patches.
          1, add MockRMMemoryStateStore in MockRM
          "drain patch" adds a DrainDispatcher and will call rm-dispatcher.await, statestore-dispatcher.await rm-dispatcher.await when drainEvents. this works for almost all of cases
          "sync patch" makes stateStore Event processed in a sync way. so drainEvents will drain all events, this will drain some unnessesary events, but seems a more general way.
          2, accessing DrainDispatcher#drained should be protected by mutex, or there will be a race condition.

          Show
          sandflee sandflee added a comment - - edited thanks Varun Saxena , Sunil G , Rohith Sharma K S for your comment and suggest ! , update two patches. 1, add MockRMMemoryStateStore in MockRM "drain patch" adds a DrainDispatcher and will call rm-dispatcher.await, statestore-dispatcher.await rm-dispatcher.await when drainEvents. this works for almost all of cases "sync patch" makes stateStore Event processed in a sync way. so drainEvents will drain all events, this will drain some unnessesary events, but seems a more general way. 2, accessing DrainDispatcher#drained should be protected by mutex, or there will be a race condition.
          Hide
          sandflee sandflee added a comment -

          I'm ok for both method, any thought?

          Show
          sandflee sandflee added a comment - I'm ok for both method, any thought?
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          I am +1 for sync-statestore patch. I just went through the patch and some comments from the patch.

          1. private volatile boolean drained = true; default value has been changed. Would you tell why this change required?
          2. I think change in the method static to non-static not necessarily required in MockRM#waitForState. Lets keep it as it is. As a result, MockAM modifications are not at all required.
          3. nit: couple of changes which are not modified are appeared in patch. May be check those also, else patch looks very huge. Ex : MockRM class, line no 349, 332
          4. One doubt, if once * disableDrainEventsImplicitly* set then there is no way to enable it. Should we provide enabling method also?
          5. After this patch, can sleeps can be avoided ? If yes, I think we need to remove so that test execute faster.
          Show
          rohithsharma Rohith Sharma K S added a comment - I am +1 for sync-statestore patch. I just went through the patch and some comments from the patch. private volatile boolean drained = true; default value has been changed. Would you tell why this change required? I think change in the method static to non-static not necessarily required in MockRM#waitForState. Lets keep it as it is. As a result, MockAM modifications are not at all required. nit: couple of changes which are not modified are appeared in patch. May be check those also, else patch looks very huge. Ex : MockRM class, line no 349, 332 One doubt, if once * disableDrainEventsImplicitly* set then there is no way to enable it. Should we provide enabling method also? After this patch, can sleeps can be avoided ? If yes, I think we need to remove so that test execute faster.
          Hide
          sunilg Sunil G added a comment -

          +1 for state-store approach..

          With this approach, we can now make sure that a target state is reached for scheduler or state-store. So overall we could improve on test case duration too. I havent looked patch in detail, will do that soon. Thank You.

          Show
          sunilg Sunil G added a comment - +1 for state-store approach.. With this approach, we can now make sure that a target state is reached for scheduler or state-store. So overall we could improve on test case duration too. I havent looked patch in detail, will do that soon. Thank You.
          Hide
          sandflee sandflee added a comment -

          Thanks Rohith Sharma K S for your review !

          private volatile boolean drained = true; default value has been changed. Would you tell why this change required?

          to be consistent with AsyncDispatcher#drained. and seems a default value of true is more reasonable.

          I think change in the method static to non-static not necessarily required in MockRM#waitForState. Lets keep it as it is. As a result, MockAM modifications are not at all required.

          change from static to non-static is to add drainEventsImplicitly(). if keep it as it is, the invoker (MockAM) maybe have to explicitly call rm#drainEvents

          nit: couple of changes which are not modified are appeared in patch. May be check those also, else patch looks very huge. Ex : MockRM class, line no 349, 332

          will do

          One doubt, if once * disableDrainEventsImplicitly* set then there is no way to enable it. Should we provide enabling method also?

          couldn't figure out the scene to disable and then enable, but I'm ok to add enable method

          After this patch, can sleeps can be avoided ? If yes, I think we need to remove so that test execute faster.

          yes, after drainEvents, all events are processed, no need to sleep-wait anymore

          Show
          sandflee sandflee added a comment - Thanks Rohith Sharma K S for your review ! private volatile boolean drained = true; default value has been changed. Would you tell why this change required? to be consistent with AsyncDispatcher#drained. and seems a default value of true is more reasonable. I think change in the method static to non-static not necessarily required in MockRM#waitForState. Lets keep it as it is. As a result, MockAM modifications are not at all required. change from static to non-static is to add drainEventsImplicitly(). if keep it as it is, the invoker (MockAM) maybe have to explicitly call rm#drainEvents nit: couple of changes which are not modified are appeared in patch. May be check those also, else patch looks very huge. Ex : MockRM class, line no 349, 332 will do One doubt, if once * disableDrainEventsImplicitly* set then there is no way to enable it. Should we provide enabling method also? couldn't figure out the scene to disable and then enable, but I'm ok to add enable method After this patch, can sleeps can be avoided ? If yes, I think we need to remove so that test execute faster. yes, after drainEvents, all events are processed, no need to sleep-wait anymore
          Hide
          sunilg Sunil G added a comment -

          I think we need to get this in as many tests are failing randomly.

          sandflee. seems like we have a consensus for state-store patch approach. In that case, could you please make this as a proper patch here.

          Show
          sunilg Sunil G added a comment - I think we need to get this in as many tests are failing randomly. sandflee . seems like we have a consensus for state-store patch approach. In that case, could you please make this as a proper patch here.
          Hide
          sandflee sandflee added a comment -

          sorry for the delay, will do this in these days

          Show
          sandflee sandflee added a comment - sorry for the delay, will do this in these days
          Hide
          sandflee sandflee added a comment -

          update YARN-5375.08.patch to address the comment of Rohith Sharma K S, and
          1. add MockRMNullStateStore since NullStateStore is mostly used
          2. simple fix TestFairScheduler deadlock

          Show
          sandflee sandflee added a comment - update YARN-5375 .08.patch to address the comment of Rohith Sharma K S , and 1. add MockRMNullStateStore since NullStateStore is mostly used 2. simple fix TestFairScheduler deadlock
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 9 new or modified test files.
          0 mvndep 0m 10s Maven dependency ordering for branch
          +1 mvninstall 6m 42s trunk passed
          +1 compile 2m 18s trunk passed
          +1 checkstyle 0m 43s trunk passed
          +1 mvnsite 1m 8s trunk passed
          +1 mvneclipse 0m 29s trunk passed
          +1 findbugs 1m 50s trunk passed
          +1 javadoc 0m 47s trunk passed
          0 mvndep 0m 10s Maven dependency ordering for patch
          +1 mvninstall 0m 56s the patch passed
          +1 compile 2m 13s the patch passed
          +1 javac 2m 13s the patch passed
          -1 checkstyle 0m 40s hadoop-yarn-project/hadoop-yarn: The patch generated 4 new + 414 unchanged - 3 fixed = 418 total (was 417)
          +1 mvnsite 1m 5s the patch passed
          +1 mvneclipse 0m 25s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 3s the patch passed
          +1 javadoc 0m 43s the patch passed
          +1 unit 2m 37s hadoop-yarn-common in the patch passed.
          +1 unit 39m 17s hadoop-yarn-server-resourcemanager in the patch passed.
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          65m 39s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12835111/YARN-5375.08.patch
          JIRA Issue YARN-5375
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 827e9d9d63c5 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / dbd2057
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13501/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13501/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/13501/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 9 new or modified test files. 0 mvndep 0m 10s Maven dependency ordering for branch +1 mvninstall 6m 42s trunk passed +1 compile 2m 18s trunk passed +1 checkstyle 0m 43s trunk passed +1 mvnsite 1m 8s trunk passed +1 mvneclipse 0m 29s trunk passed +1 findbugs 1m 50s trunk passed +1 javadoc 0m 47s trunk passed 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 0m 56s the patch passed +1 compile 2m 13s the patch passed +1 javac 2m 13s the patch passed -1 checkstyle 0m 40s hadoop-yarn-project/hadoop-yarn: The patch generated 4 new + 414 unchanged - 3 fixed = 418 total (was 417) +1 mvnsite 1m 5s the patch passed +1 mvneclipse 0m 25s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 3s the patch passed +1 javadoc 0m 43s the patch passed +1 unit 2m 37s hadoop-yarn-common in the patch passed. +1 unit 39m 17s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 65m 39s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12835111/YARN-5375.08.patch JIRA Issue YARN-5375 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 827e9d9d63c5 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / dbd2057 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13501/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13501/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/13501/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          Overall patch looks good..
          one doubt,

          simple fix TestFairScheduler deadlock

          Could explain why deadlock detected? Is it because of this patch change?

          Show
          rohithsharma Rohith Sharma K S added a comment - Overall patch looks good.. one doubt, simple fix TestFairScheduler deadlock Could explain why deadlock detected? Is it because of this patch change?
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          Opps, I checked the code. It is issue in fair scheduler.

          Show
          rohithsharma Rohith Sharma K S added a comment - Opps, I checked the code. It is issue in fair scheduler.
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          Though in FairScheduler dead lock code exist, in reality deadlock can't not occur. Because all the events are handled in sequence. I checked the test fair scheduler code too. There also, scheduler event are processed by sequence.
          Would you attach dead lock trace ?

          Show
          rohithsharma Rohith Sharma K S added a comment - Though in FairScheduler dead lock code exist, in reality deadlock can't not occur. Because all the events are handled in sequence. I checked the test fair scheduler code too. There also, scheduler event are processed by sequence. Would you attach dead lock trace ?
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          I debugged more finding reason for deadlock. It is because, patch adds MockRM#createSchedulerEventDispatcher. I believe sleep might not solve deadlock.
          One of the solution I think is since after this patch however scheduler event processing becomes sequential, need to change the test code to trigger an event rather than resourceManager.getResourceScheduler().handle(nodeUpdate);

          Show
          rohithsharma Rohith Sharma K S added a comment - I debugged more finding reason for deadlock. It is because, patch adds MockRM#createSchedulerEventDispatcher. I believe sleep might not solve deadlock. One of the solution I think is since after this patch however scheduler event processing becomes sequential, need to change the test code to trigger an event rather than resourceManager.getResourceScheduler().handle(nodeUpdate);
          Hide
          sandflee sandflee added a comment -

          Thanks Rohith Sharma K S, yes, MockRM#createSchedulerEventDispatcher triggers the deadlock. NodeAddEvent used to left in schedule event queue now get processed. agree that using resourcemanager.handle(nodeUpdate) could solve this bug.
          and this test seems very special, only rm dispatcher started.

          Show
          sandflee sandflee added a comment - Thanks Rohith Sharma K S , yes, MockRM#createSchedulerEventDispatcher triggers the deadlock. NodeAddEvent used to left in schedule event queue now get processed. agree that using resourcemanager.handle(nodeUpdate) could solve this bug. and this test seems very special, only rm dispatcher started.
          Hide
          sandflee sandflee added a comment -

          update the patch to fix TestFairScheduler, adding drainEvents after a node is registered. not using a rm.dispatcher.handle() for most TestFairScheduler not using this way.

          Show
          sandflee sandflee added a comment - update the patch to fix TestFairScheduler, adding drainEvents after a node is registered. not using a rm.dispatcher.handle() for most TestFairScheduler not using this way.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 9 new or modified test files.
          0 mvndep 0m 10s Maven dependency ordering for branch
          +1 mvninstall 8m 24s trunk passed
          +1 compile 6m 56s trunk passed
          +1 checkstyle 0m 55s trunk passed
          +1 mvnsite 1m 36s trunk passed
          +1 mvneclipse 0m 47s trunk passed
          +1 findbugs 2m 12s trunk passed
          +1 javadoc 1m 8s trunk passed
          0 mvndep 0m 10s Maven dependency ordering for patch
          +1 mvninstall 1m 0s the patch passed
          +1 compile 4m 57s the patch passed
          +1 javac 4m 57s the patch passed
          -0 checkstyle 0m 52s hadoop-yarn-project/hadoop-yarn: The patch generated 3 new + 409 unchanged - 3 fixed = 412 total (was 412)
          +1 mvnsite 1m 27s the patch passed
          +1 mvneclipse 0m 47s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 30s the patch passed
          +1 javadoc 1m 7s the patch passed
          +1 unit 2m 33s hadoop-yarn-common in the patch passed.
          +1 unit 38m 49s hadoop-yarn-server-resourcemanager in the patch passed.
          +1 asflicense 0m 37s The patch does not generate ASF License warnings.
          85m 32s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:e809691
          JIRA Issue YARN-5375
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12837704/YARN-5375.09.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 4bf671c1ceaf 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / ca33bdd
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13805/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13805/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/13805/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 9 new or modified test files. 0 mvndep 0m 10s Maven dependency ordering for branch +1 mvninstall 8m 24s trunk passed +1 compile 6m 56s trunk passed +1 checkstyle 0m 55s trunk passed +1 mvnsite 1m 36s trunk passed +1 mvneclipse 0m 47s trunk passed +1 findbugs 2m 12s trunk passed +1 javadoc 1m 8s trunk passed 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 1m 0s the patch passed +1 compile 4m 57s the patch passed +1 javac 4m 57s the patch passed -0 checkstyle 0m 52s hadoop-yarn-project/hadoop-yarn: The patch generated 3 new + 409 unchanged - 3 fixed = 412 total (was 412) +1 mvnsite 1m 27s the patch passed +1 mvneclipse 0m 47s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 30s the patch passed +1 javadoc 1m 7s the patch passed +1 unit 2m 33s hadoop-yarn-common in the patch passed. +1 unit 38m 49s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 37s The patch does not generate ASF License warnings. 85m 32s Subsystem Report/Notes Docker Image:yetus/hadoop:e809691 JIRA Issue YARN-5375 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12837704/YARN-5375.09.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 4bf671c1ceaf 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ca33bdd Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13805/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13805/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/13805/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          Overall patch looks good to me.. I will do testing on random test failures applying this patch. And also I believe many places MockRM#waitFor**** are sleeping, this need to be re looked how sleep time can be reduced.

          Show
          rohithsharma Rohith Sharma K S added a comment - Overall patch looks good to me.. I will do testing on random test failures applying this patch. And also I believe many places MockRM#waitFor**** are sleeping, this need to be re looked how sleep time can be reduced.
          Hide
          sunilg Sunil G added a comment -

          Patch looks good for me too. I also will share a feedback with some tests to see whether we have some more random issues still.

          Show
          sunilg Sunil G added a comment - Patch looks good for me too. I also will share a feedback with some tests to see whether we have some more random issues still.
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          I ran all the tests package by package. It looks all tests ran other than Kerberos tests. But patch need to be rebased. Could you rebase patch please?

          Show
          rohithsharma Rohith Sharma K S added a comment - I ran all the tests package by package. It looks all tests ran other than Kerberos tests. But patch need to be rebased. Could you rebase patch please?
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          Myself rebased patch against latest trunk on behalf of sandflee .. Let wait for Jenkins report.

          Show
          rohithsharma Rohith Sharma K S added a comment - Myself rebased patch against latest trunk on behalf of sandflee .. Let wait for Jenkins report.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 9 new or modified test files.
          0 mvndep 0m 10s Maven dependency ordering for branch
          +1 mvninstall 7m 9s trunk passed
          +1 compile 5m 2s trunk passed
          +1 checkstyle 0m 49s trunk passed
          +1 mvnsite 1m 22s trunk passed
          +1 mvneclipse 0m 42s trunk passed
          +1 findbugs 2m 9s trunk passed
          +1 javadoc 1m 4s trunk passed
          0 mvndep 0m 10s Maven dependency ordering for patch
          +1 mvninstall 1m 0s the patch passed
          +1 compile 4m 47s the patch passed
          +1 javac 4m 47s the patch passed
          -0 checkstyle 0m 49s hadoop-yarn-project/hadoop-yarn: The patch generated 3 new + 409 unchanged - 2 fixed = 412 total (was 411)
          +1 mvnsite 1m 19s the patch passed
          +1 mvneclipse 0m 39s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 25s the patch passed
          +1 javadoc 1m 1s the patch passed
          +1 unit 2m 24s hadoop-yarn-common in the patch passed.
          -1 unit 40m 56s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 asflicense 0m 29s The patch does not generate ASF License warnings.
          82m 50s



          Reason Tests
          Failed junit tests hadoop.yarn.server.resourcemanager.TestTokenClientRMService



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue YARN-5375
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838737/YARN-5375.10.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 25216010d2f2 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 79448d4
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13894/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/13894/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13894/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/13894/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 9 new or modified test files. 0 mvndep 0m 10s Maven dependency ordering for branch +1 mvninstall 7m 9s trunk passed +1 compile 5m 2s trunk passed +1 checkstyle 0m 49s trunk passed +1 mvnsite 1m 22s trunk passed +1 mvneclipse 0m 42s trunk passed +1 findbugs 2m 9s trunk passed +1 javadoc 1m 4s trunk passed 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 1m 0s the patch passed +1 compile 4m 47s the patch passed +1 javac 4m 47s the patch passed -0 checkstyle 0m 49s hadoop-yarn-project/hadoop-yarn: The patch generated 3 new + 409 unchanged - 2 fixed = 412 total (was 411) +1 mvnsite 1m 19s the patch passed +1 mvneclipse 0m 39s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 25s the patch passed +1 javadoc 1m 1s the patch passed +1 unit 2m 24s hadoop-yarn-common in the patch passed. -1 unit 40m 56s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 29s The patch does not generate ASF License warnings. 82m 50s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestTokenClientRMService Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-5375 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838737/YARN-5375.10.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 25216010d2f2 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 79448d4 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13894/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13894/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13894/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/13894/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          sandflee sandflee added a comment -

          thanks Rohith Sharma K S, TestTokenClientRMService test failure seems not related to this issue, it couldn't run pass locally and is tracked by YARN-5875

          Show
          sandflee sandflee added a comment - thanks Rohith Sharma K S , TestTokenClientRMService test failure seems not related to this issue, it couldn't run pass locally and is tracked by YARN-5875
          Hide
          sunilg Sunil G added a comment -

          +1 from my end. Thanks.

          Show
          sunilg Sunil G added a comment - +1 from my end. Thanks.
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          I was re-looking the patch especially MockRM class changes. I think we should use drainEventsImplicitly() before any method waitFotState() rather draining at the end of methods. As I see most of the waitForState() methods are not triggering any events.
          And in some of the waitForState methods are sending a nodeHeartBeat internally. I think we should drain for every heartbeat sent to RM in waitForState Methods.

          Show
          rohithsharma Rohith Sharma K S added a comment - I was re-looking the patch especially MockRM class changes. I think we should use drainEventsImplicitly() before any method waitFotState() rather draining at the end of methods. As I see most of the waitForState() methods are not triggering any events. And in some of the waitForState methods are sending a nodeHeartBeat internally. I think we should drain for every heartbeat sent to RM in waitForState Methods.
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          The overall patch looks good. One quick comment on patch need to change as below.

           for (MockNM nm : nms) {
                  nm.nodeHeartbeat(true);
                }
          drainEventsImplicitly(); // Add here after all NM's heart beat is sent
          
          Show
          rohithsharma Rohith Sharma K S added a comment - The overall patch looks good. One quick comment on patch need to change as below. for (MockNM nm : nms) { nm.nodeHeartbeat( true ); } drainEventsImplicitly(); // Add here after all NM's heart beat is sent
          Hide
          sandflee sandflee added a comment -

          update the patch to invoke drainEventsImplicitly() at the front of waitForState(). and fix a test failure caused by this

          TestAbstractYarnScheduler.java
               // AM crashes, and a new app-attempt gets created
                node.nodeHeartbeat(applicationAttemptOneID, 1, ContainerState.COMPLETE);
                rm.waitForState(node, am1ContainerID, RMContainerState.COMPLETED, 30 * 1000);
                RMAppAttempt rmAppAttempt2 = MockRM.waitForAttemptScheduled(rmApp, rm);
          

          waitForState will drain all events first leading completed container not exist in new schedulerAppAttempt, and the coresponding waitForState() code will invoke node hearbeat, appattempt will be allocated state, the check for scheduled state will fail

          Show
          sandflee sandflee added a comment - update the patch to invoke drainEventsImplicitly() at the front of waitForState(). and fix a test failure caused by this TestAbstractYarnScheduler.java // AM crashes, and a new app-attempt gets created node.nodeHeartbeat(applicationAttemptOneID, 1, ContainerState.COMPLETE); rm.waitForState(node, am1ContainerID, RMContainerState.COMPLETED, 30 * 1000); RMAppAttempt rmAppAttempt2 = MockRM.waitForAttemptScheduled(rmApp, rm); waitForState will drain all events first leading completed container not exist in new schedulerAppAttempt, and the coresponding waitForState() code will invoke node hearbeat, appattempt will be allocated state, the check for scheduled state will fail
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          +1, pending jenkins

          Show
          rohithsharma Rohith Sharma K S added a comment - +1, pending jenkins
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 docker 3m 46s Docker failed to build yetus/hadoop:a9ad5d6.



          Subsystem Report/Notes
          JIRA Issue YARN-5375
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838951/YARN-5375.12.patch
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/13920/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 docker 3m 46s Docker failed to build yetus/hadoop:a9ad5d6. Subsystem Report/Notes JIRA Issue YARN-5375 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838951/YARN-5375.12.patch Console output https://builds.apache.org/job/PreCommit-YARN-Build/13920/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 22s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 10 new or modified test files.
          0 mvndep 0m 11s Maven dependency ordering for branch
          +1 mvninstall 7m 26s trunk passed
          +1 compile 5m 29s trunk passed
          +1 checkstyle 0m 54s trunk passed
          +1 mvnsite 1m 30s trunk passed
          +1 mvneclipse 0m 42s trunk passed
          +1 findbugs 2m 7s trunk passed
          +1 javadoc 1m 1s trunk passed
          0 mvndep 0m 10s Maven dependency ordering for patch
          +1 mvninstall 0m 59s the patch passed
          +1 compile 4m 42s the patch passed
          +1 javac 4m 42s the patch passed
          -0 checkstyle 0m 53s hadoop-yarn-project/hadoop-yarn: The patch generated 2 new + 423 unchanged - 3 fixed = 425 total (was 426)
          +1 mvnsite 1m 31s the patch passed
          +1 mvneclipse 0m 40s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 35s the patch passed
          +1 javadoc 1m 5s the patch passed
          +1 unit 2m 30s hadoop-yarn-common in the patch passed.
          -1 unit 42m 2s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 asflicense 0m 29s The patch does not generate ASF License warnings.
          85m 25s



          Reason Tests
          Failed junit tests hadoop.yarn.server.resourcemanager.TestTokenClientRMService



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue YARN-5375
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838947/YARN-5375.11.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux a088b853af80 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 7ffb994
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13918/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/13918/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13918/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/13918/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 22s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 10 new or modified test files. 0 mvndep 0m 11s Maven dependency ordering for branch +1 mvninstall 7m 26s trunk passed +1 compile 5m 29s trunk passed +1 checkstyle 0m 54s trunk passed +1 mvnsite 1m 30s trunk passed +1 mvneclipse 0m 42s trunk passed +1 findbugs 2m 7s trunk passed +1 javadoc 1m 1s trunk passed 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 0m 59s the patch passed +1 compile 4m 42s the patch passed +1 javac 4m 42s the patch passed -0 checkstyle 0m 53s hadoop-yarn-project/hadoop-yarn: The patch generated 2 new + 423 unchanged - 3 fixed = 425 total (was 426) +1 mvnsite 1m 31s the patch passed +1 mvneclipse 0m 40s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 35s the patch passed +1 javadoc 1m 5s the patch passed +1 unit 2m 30s hadoop-yarn-common in the patch passed. -1 unit 42m 2s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 29s The patch does not generate ASF License warnings. 85m 25s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestTokenClientRMService Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-5375 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838947/YARN-5375.11.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux a088b853af80 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 7ffb994 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13918/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13918/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13918/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/13918/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          sandflee latest patch YARN-5375.12.patch, HadoopQA has failed. Could you attach new patch?

          Show
          rohithsharma Rohith Sharma K S added a comment - sandflee latest patch YARN-5375 .12.patch , HadoopQA has failed. Could you attach new patch?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 21s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 10 new or modified test files.
          0 mvndep 0m 56s Maven dependency ordering for branch
          +1 mvninstall 9m 4s trunk passed
          +1 compile 6m 42s trunk passed
          +1 checkstyle 1m 2s trunk passed
          +1 mvnsite 1m 43s trunk passed
          +1 mvneclipse 0m 48s trunk passed
          +1 findbugs 2m 35s trunk passed
          +1 javadoc 1m 12s trunk passed
          0 mvndep 0m 12s Maven dependency ordering for patch
          +1 mvninstall 1m 11s the patch passed
          +1 compile 5m 54s the patch passed
          +1 javac 5m 54s the patch passed
          -0 checkstyle 0m 59s hadoop-yarn-project/hadoop-yarn: The patch generated 2 new + 423 unchanged - 3 fixed = 425 total (was 426)
          +1 mvnsite 1m 39s the patch passed
          +1 mvneclipse 0m 52s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 49s the patch passed
          +1 javadoc 1m 3s the patch passed
          +1 unit 2m 31s hadoop-yarn-common in the patch passed.
          -1 unit 41m 49s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 asflicense 0m 29s The patch does not generate ASF License warnings.
          92m 18s



          Reason Tests
          Failed junit tests hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue YARN-5375
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839103/YARN-5375.12.new.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 399f12ebab62 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 61c0bed
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13936/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/13936/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13936/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/13936/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 10 new or modified test files. 0 mvndep 0m 56s Maven dependency ordering for branch +1 mvninstall 9m 4s trunk passed +1 compile 6m 42s trunk passed +1 checkstyle 1m 2s trunk passed +1 mvnsite 1m 43s trunk passed +1 mvneclipse 0m 48s trunk passed +1 findbugs 2m 35s trunk passed +1 javadoc 1m 12s trunk passed 0 mvndep 0m 12s Maven dependency ordering for patch +1 mvninstall 1m 11s the patch passed +1 compile 5m 54s the patch passed +1 javac 5m 54s the patch passed -0 checkstyle 0m 59s hadoop-yarn-project/hadoop-yarn: The patch generated 2 new + 423 unchanged - 3 fixed = 425 total (was 426) +1 mvnsite 1m 39s the patch passed +1 mvneclipse 0m 52s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 49s the patch passed +1 javadoc 1m 3s the patch passed +1 unit 2m 31s hadoop-yarn-common in the patch passed. -1 unit 41m 49s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 29s The patch does not generate ASF License warnings. 92m 18s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-5375 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839103/YARN-5375.12.new.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 399f12ebab62 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 61c0bed Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13936/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13936/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13936/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/13936/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          sandflee sandflee added a comment -

          update YARN-5375.12.new.patch to trigger jenkins

          Show
          sandflee sandflee added a comment - update YARN-5375 .12.new.patch to trigger jenkins
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          committed to trunk/branch-2. thanks sandflee for the patch.. thanks Sunil and Varun for the review..

          Show
          rohithsharma Rohith Sharma K S added a comment - committed to trunk/branch-2. thanks sandflee for the patch.. thanks Sunil and Varun for the review..
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10844 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10844/)
          YARN-5375. invoke MockRM#drainEvents implicitly in MockRM methods to (rohithsharmaks: rev d65603517e52843f11cd9d3b6f6e28fca9336ee3)

          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestAbstractYarnScheduler.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/event/DrainDispatcher.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationCleanup.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRM.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestNodeBlacklistingOnAMFailures.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesNodes.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMRMRPCNodeUpdates.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10844 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10844/ ) YARN-5375 . invoke MockRM#drainEvents implicitly in MockRM methods to (rohithsharmaks: rev d65603517e52843f11cd9d3b6f6e28fca9336ee3) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestAbstractYarnScheduler.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/event/DrainDispatcher.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationCleanup.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRM.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestNodeBlacklistingOnAMFailures.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesNodes.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMRMRPCNodeUpdates.java
          Hide
          sandflee sandflee added a comment -

          Thanks Rohith,Sunil, Varun for review and commit ! after this path most random test failure should be resolved. will open another jira to fix RMRestart and MockAM.

          Show
          sandflee sandflee added a comment - Thanks Rohith,Sunil, Varun for review and commit ! after this path most random test failure should be resolved. will open another jira to fix RMRestart and MockAM.
          Hide
          varun_saxena Varun Saxena added a comment -

          sandflee,Rohith Sharma K S looking at the patch, it seems we haven't really replaced tests to use new MockRMMemoryStateStore instead of MemoryRMStateStore. Do we plan to do this in another JIRA ?

          Show
          varun_saxena Varun Saxena added a comment - sandflee , Rohith Sharma K S looking at the patch, it seems we haven't really replaced tests to use new MockRMMemoryStateStore instead of MemoryRMStateStore. Do we plan to do this in another JIRA ?
          Hide
          sandflee sandflee added a comment -

          yes, to keep the patch simple, not replace MemoryRMStateStore, let's go to YARN-5548

          Show
          sandflee sandflee added a comment - yes, to keep the patch simple, not replace MemoryRMStateStore, let's go to YARN-5548

            People

            • Assignee:
              sandflee sandflee
              Reporter:
              sandflee sandflee
            • Votes:
              3 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development