Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-4398

Yarn recover functionality causes the cluster running slowly and the cluster usage rate is far below 100

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      In my hadoop cluster, the resourceManager recover functionality is enabled with FileSystemRMStateStore.
      I found this cause the yarn cluster running slowly and cluster usage rate is just 50 even there are many pending Apps.

      The scenario is below.
      In thread A, the RMAppImpl$RMAppNewlySavingTransition is calling storeNewApplication method defined in RMStateStore. This storeNewApplication method is synchronized.

      RMAppImpl.java
        private static final class RMAppNewlySavingTransition extends RMAppTransition {
          @Override
          public void transition(RMAppImpl app, RMAppEvent event) {
      
            // If recovery is enabled then store the application information in a
            // non-blocking call so make sure that RM has stored the information
            // needed to restart the AM after RM restart without further client
            // communication
            LOG.info("Storing application with id " + app.applicationId);
            app.rmContext.getStateStore().storeNewApplication(app);
          }
        }
      
      RMStateStore.java
      public synchronized void storeNewApplication(RMApp app) {
          ApplicationSubmissionContext context = app
                                                  .getApplicationSubmissionContext();
          assert context instanceof ApplicationSubmissionContextPBImpl;
          ApplicationStateData appState =
              ApplicationStateData.newInstance(
                  app.getSubmitTime(), app.getStartTime(), context, app.getUser());
          dispatcher.getEventHandler().handle(new RMStateStoreAppEvent(appState));
        }
      

      In thread B, the FileSystemRMStateStore is calling storeApplicationStateInternal method. It's also synchronized.
      This storeApplicationStateInternal method saves an ApplicationStateData into HDFS and it normally costs 90~300 milliseconds in my hadoop cluster.

      FileSystemRMStateStore.java
      public synchronized void storeApplicationStateInternal(ApplicationId appId,
            ApplicationStateData appStateDataPB) throws Exception {
          Path appDirPath = getAppDir(rmAppRoot, appId);
          mkdirsWithRetries(appDirPath);
          Path nodeCreatePath = getNodePath(appDirPath, appId.toString());
      
          LOG.info("Storing info for app: " + appId + " at: " + nodeCreatePath);
          byte[] appStateData = appStateDataPB.getProto().toByteArray();
          try {
            // currently throw all exceptions. May need to respond differently for HA
            // based on whether we have lost the right to write to FS
            writeFileWithRetries(nodeCreatePath, appStateData, true);
          } catch (Exception e) {
            LOG.info("Error storing info for app: " + appId, e);
            throw e;
          }
        }
      

      Think thread B firstly comes into FileSystemRMStateStore.storeApplicationStateInternal method, then thread A will be blocked for a while because of synchronization. In ResourceManager there is only one RMStateStore instance. In my cluster it's FileSystemRMStateStore type.
      Debug the RMAppNewlySavingTransition.transition method, the thread stack shows it's called form AsyncDispatcher.dispatch method. This method code is as below.

      AsyncDispatcher.java
        protected void dispatch(Event event) {
          //all events go thru this loop
          if (LOG.isDebugEnabled()) {
            LOG.debug("Dispatching the event " + event.getClass().getName() + "."
                + event.toString());
          }
      
          Class<? extends Enum> type = event.getType().getDeclaringClass();
      
          try{
            EventHandler handler = eventDispatchers.get(type);
            if(handler != null) {
              handler.handle(event);
            } else {
              throw new Exception("No handler for registered for " + type);
            }
          } catch (Throwable t) {
            //TODO Maybe log the state of the queue
            LOG.fatal("Error in dispatcher thread", t);
            // If serviceStop is called, we should exit this thread gracefully.
            if (exitOnDispatchException
                && (ShutdownHookManager.get().isShutdownInProgress()) == false
                && stopped == false) {
              Thread shutDownThread = new Thread(createShutDownThread());
              shutDownThread.setName("AsyncDispatcher ShutDown handler");
              shutDownThread.start();
            }
          }
        }
      

      Above code shows AsyncDispatcher.dispatch method can process different type events.
      In fact this AsyncDispatcher instance is just ResourceManager.rmDispatcher created in ResourceManager.serviceInit method.
      You can find many eventTypes and handlers are registered in ResourceManager.rmDispatcher.
      In above scenario thread B blocks thread A, then many following events processing are blocked.

      In my testing cluster, there is only one queue and the client submits 1000 applications concurrently, the yarn cluster usage rate is 50. Many apps are pending. If I disable resourceManager recover functionality, the cluster usage can be 100.

      To solve this issue, I removed synchronized modifier on some methods defined in RMStateStore.
      Instead, in these methods I defined a dedicated lock object before calling dispatcher.getEventHandler().handle.
      In this way, the yarn cluster usage rate can be 100 and the whole cluster is good running.
      Please see my attached patch.

      1. YARN-4398.2.patch
        4 kB
        NING DING
      2. YARN-4398.3.patch
        4 kB
        NING DING
      3. YARN-4398.4.patch
        4 kB
        NING DING

        Activity

        Hide
        iceberg565 NING DING added a comment -

        Jian He would you kindly help to take a look on this issue?

        Show
        iceberg565 NING DING added a comment - Jian He would you kindly help to take a look on this issue?
        Hide
        templedf Daniel Templeton added a comment -

        The AsyncDispatcher.GenericEventHandler.handle() method is MT safe. The AsyncDispatcher.getEventHandler() is the unsafe call, and it's only unsafe because of the lazy initialization. Prior to YARN-1121, it was returning a new object every time, which was thread safe. I see two obvious options: revert the YARN-1121 optimization in the AsyncDispatcher.getEventHandler() method or do eager initialization into a final member variable. Either way, the calls become MT-safe, letting you just drop the synchronization.

        Show
        templedf Daniel Templeton added a comment - The AsyncDispatcher.GenericEventHandler.handle() method is MT safe. The AsyncDispatcher.getEventHandler() is the unsafe call, and it's only unsafe because of the lazy initialization. Prior to YARN-1121 , it was returning a new object every time, which was thread safe. I see two obvious options: revert the YARN-1121 optimization in the AsyncDispatcher.getEventHandler() method or do eager initialization into a final member variable. Either way, the calls become MT-safe, letting you just drop the synchronization.
        Hide
        jianhe Jian He added a comment -

        NING DING, thanks for looking into this. analysis makes sense to me. I think we can just remove the synchronized keyword ?

        the AsyncDispatcher.getEventHandler() is the unsafe call

        Suppose the call is unsafe, in the worst case when contention happens, separate new objects will return to each caller instead of one, which is equivalent to new object every time as before ?

        Show
        jianhe Jian He added a comment - NING DING , thanks for looking into this. analysis makes sense to me. I think we can just remove the synchronized keyword ? the AsyncDispatcher.getEventHandler() is the unsafe call Suppose the call is unsafe, in the worst case when contention happens, separate new objects will return to each caller instead of one, which is equivalent to new object every time as before ?
        Hide
        templedf Daniel Templeton added a comment -

        Jian He, you are correct, but that approach just smells bad to me. It's behavior that someone will be confused by later. It would be better to do something intentional than something that accidentally works for a non-obvious reason.

        Show
        templedf Daniel Templeton added a comment - Jian He , you are correct, but that approach just smells bad to me. It's behavior that someone will be confused by later. It would be better to do something intentional than something that accidentally works for a non-obvious reason.
        Hide
        iceberg565 NING DING added a comment -

        Thanks for all your comments.
        I prefer to do eager initialization handlerInstance in AsyncDispatcher, then remove synchronized modifier in RMStateStore.
        Pelese see my new patch.

        Show
        iceberg565 NING DING added a comment - Thanks for all your comments. I prefer to do eager initialization handlerInstance in AsyncDispatcher, then remove synchronized modifier in RMStateStore. Pelese see my new patch.
        Hide
        jianhe Jian He added a comment -

        patch looks good to me. thanks NING DING, and Daniel Templeton for the review.

        Show
        jianhe Jian He added a comment - patch looks good to me. thanks NING DING , and Daniel Templeton for the review.
        Hide
        templedf Daniel Templeton added a comment -

        Yep. +1 (non-binding)

        Show
        templedf Daniel Templeton added a comment - Yep. +1 (non-binding)
        Hide
        templedf Daniel Templeton added a comment -

        Actually, before I +1 that, NING DING, did you run the full set of RM unit tests against the patch?

        Show
        templedf Daniel Templeton added a comment - Actually, before I +1 that, NING DING , did you run the full set of RM unit tests against the patch?
        Hide
        jianhe Jian He added a comment -

        NING DING, after you upload the patch, you can click the "Submit Patch" button, which will trigger jenkins to run the unit tests.

        Show
        jianhe Jian He added a comment - NING DING , after you upload the patch, you can click the "Submit Patch" button, which will trigger jenkins to run the unit tests.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 9m 49s trunk passed
        +1 compile 3m 5s trunk passed with JDK v1.8.0_66
        +1 compile 2m 49s trunk passed with JDK v1.7.0_85
        +1 checkstyle 0m 35s trunk passed
        +1 mvnsite 1m 28s trunk passed
        +1 mvneclipse 0m 35s trunk passed
        +1 findbugs 3m 13s trunk passed
        +1 javadoc 1m 13s trunk passed with JDK v1.8.0_66
        +1 javadoc 1m 17s trunk passed with JDK v1.7.0_85
        +1 mvninstall 1m 23s the patch passed
        +1 compile 3m 4s the patch passed with JDK v1.8.0_66
        +1 javac 3m 4s the patch passed
        +1 compile 2m 50s the patch passed with JDK v1.7.0_85
        +1 javac 2m 50s the patch passed
        +1 checkstyle 0m 35s the patch passed
        +1 mvnsite 1m 27s the patch passed
        +1 mvneclipse 0m 35s the patch passed
        -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 findbugs 3m 33s the patch passed
        +1 javadoc 1m 14s the patch passed with JDK v1.8.0_66
        +1 javadoc 1m 16s the patch passed with JDK v1.7.0_85
        +1 unit 2m 35s hadoop-yarn-common in the patch passed with JDK v1.8.0_66.
        -1 unit 65m 28s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66.
        +1 unit 2m 36s hadoop-yarn-common in the patch passed with JDK v1.7.0_85.
        -1 unit 65m 3s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_85.
        +1 asflicense 0m 28s Patch does not generate ASF License warnings.
        178m 3s



        Reason Tests
        JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
          hadoop.yarn.server.resourcemanager.rmapp.TestRMAppTransitions
          hadoop.yarn.server.resourcemanager.TestAMAuthorization
        JDK v1.7.0_85 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
          hadoop.yarn.server.resourcemanager.rmapp.TestRMAppTransitions
          hadoop.yarn.server.resourcemanager.TestAMAuthorization



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12774937/YARN-4398.3.patch
        JIRA Issue YARN-4398
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux c0bfc366ed04 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 830eb25
        findbugs v3.0.0
        whitespace https://builds.apache.org/job/PreCommit-YARN-Build/9829/artifact/patchprocess/whitespace-eol.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/9829/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/9829/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_85.txt
        unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/9829/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/9829/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_85.txt
        JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9829/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
        Max memory used 76MB
        Powered by Apache Yetus http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/9829/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 9m 49s trunk passed +1 compile 3m 5s trunk passed with JDK v1.8.0_66 +1 compile 2m 49s trunk passed with JDK v1.7.0_85 +1 checkstyle 0m 35s trunk passed +1 mvnsite 1m 28s trunk passed +1 mvneclipse 0m 35s trunk passed +1 findbugs 3m 13s trunk passed +1 javadoc 1m 13s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 17s trunk passed with JDK v1.7.0_85 +1 mvninstall 1m 23s the patch passed +1 compile 3m 4s the patch passed with JDK v1.8.0_66 +1 javac 3m 4s the patch passed +1 compile 2m 50s the patch passed with JDK v1.7.0_85 +1 javac 2m 50s the patch passed +1 checkstyle 0m 35s the patch passed +1 mvnsite 1m 27s the patch passed +1 mvneclipse 0m 35s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 findbugs 3m 33s the patch passed +1 javadoc 1m 14s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 16s the patch passed with JDK v1.7.0_85 +1 unit 2m 35s hadoop-yarn-common in the patch passed with JDK v1.8.0_66. -1 unit 65m 28s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66. +1 unit 2m 36s hadoop-yarn-common in the patch passed with JDK v1.7.0_85. -1 unit 65m 3s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_85. +1 asflicense 0m 28s Patch does not generate ASF License warnings. 178m 3s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.rmapp.TestRMAppTransitions   hadoop.yarn.server.resourcemanager.TestAMAuthorization JDK v1.7.0_85 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.rmapp.TestRMAppTransitions   hadoop.yarn.server.resourcemanager.TestAMAuthorization Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12774937/YARN-4398.3.patch JIRA Issue YARN-4398 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux c0bfc366ed04 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 830eb25 findbugs v3.0.0 whitespace https://builds.apache.org/job/PreCommit-YARN-Build/9829/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/9829/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/9829/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_85.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/9829/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/9829/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_85.txt JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9829/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Max memory used 76MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/9829/console This message was automatically generated.
        Hide
        iceberg565 NING DING added a comment -

        I uploaded a new patch that removed useless whitespace.
        The current test cases can cover the modified code in this patch. This patch resolved performance issue. So no new unit test cases.

        Show
        iceberg565 NING DING added a comment - I uploaded a new patch that removed useless whitespace. The current test cases can cover the modified code in this patch. This patch resolved performance issue. So no new unit test cases.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 8m 9s trunk passed
        +1 compile 2m 10s trunk passed with JDK v1.8.0_66
        +1 compile 2m 22s trunk passed with JDK v1.7.0_85
        +1 checkstyle 0m 30s trunk passed
        +1 mvnsite 1m 20s trunk passed
        +1 mvneclipse 0m 29s trunk passed
        +1 findbugs 2m 46s trunk passed
        +1 javadoc 1m 16s trunk passed with JDK v1.8.0_66
        +1 javadoc 1m 1s trunk passed with JDK v1.7.0_85
        +1 mvninstall 1m 7s the patch passed
        +1 compile 1m 59s the patch passed with JDK v1.8.0_66
        +1 javac 1m 59s the patch passed
        +1 compile 2m 14s the patch passed with JDK v1.7.0_85
        +1 javac 2m 14s the patch passed
        +1 checkstyle 0m 28s the patch passed
        +1 mvnsite 1m 12s the patch passed
        +1 mvneclipse 0m 28s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 2m 55s the patch passed
        +1 javadoc 0m 56s the patch passed with JDK v1.8.0_66
        +1 javadoc 1m 6s the patch passed with JDK v1.7.0_85
        +1 unit 2m 9s hadoop-yarn-common in the patch passed with JDK v1.8.0_66.
        -1 unit 64m 58s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66.
        +1 unit 2m 13s hadoop-yarn-common in the patch passed with JDK v1.7.0_85.
        -1 unit 65m 29s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_85.
        +1 asflicense 0m 22s Patch does not generate ASF License warnings.
        169m 7s



        Reason Tests
        JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
          hadoop.yarn.server.resourcemanager.TestAMAuthorization
        JDK v1.7.0_85 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
          hadoop.yarn.server.resourcemanager.security.TestRMDelegationTokens
          hadoop.yarn.server.resourcemanager.TestAMAuthorization
          hadoop.yarn.server.resourcemanager.TestRM



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12775241/YARN-4398.4.patch
        JIRA Issue YARN-4398
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux bc3bf54a8c60 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 53e3bf7
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-YARN-Build/9834/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/9834/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_85.txt
        unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/9834/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/9834/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_85.txt
        JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9834/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
        Max memory used 76MB
        Powered by Apache Yetus http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/9834/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 8m 9s trunk passed +1 compile 2m 10s trunk passed with JDK v1.8.0_66 +1 compile 2m 22s trunk passed with JDK v1.7.0_85 +1 checkstyle 0m 30s trunk passed +1 mvnsite 1m 20s trunk passed +1 mvneclipse 0m 29s trunk passed +1 findbugs 2m 46s trunk passed +1 javadoc 1m 16s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 1s trunk passed with JDK v1.7.0_85 +1 mvninstall 1m 7s the patch passed +1 compile 1m 59s the patch passed with JDK v1.8.0_66 +1 javac 1m 59s the patch passed +1 compile 2m 14s the patch passed with JDK v1.7.0_85 +1 javac 2m 14s the patch passed +1 checkstyle 0m 28s the patch passed +1 mvnsite 1m 12s the patch passed +1 mvneclipse 0m 28s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 55s the patch passed +1 javadoc 0m 56s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 6s the patch passed with JDK v1.7.0_85 +1 unit 2m 9s hadoop-yarn-common in the patch passed with JDK v1.8.0_66. -1 unit 64m 58s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66. +1 unit 2m 13s hadoop-yarn-common in the patch passed with JDK v1.7.0_85. -1 unit 65m 29s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_85. +1 asflicense 0m 22s Patch does not generate ASF License warnings. 169m 7s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization JDK v1.7.0_85 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.security.TestRMDelegationTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestRM Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12775241/YARN-4398.4.patch JIRA Issue YARN-4398 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux bc3bf54a8c60 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 53e3bf7 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/9834/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/9834/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_85.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/9834/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/9834/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_85.txt JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9834/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Max memory used 76MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/9834/console This message was automatically generated.
        Hide
        iceberg565 NING DING added a comment -

        Jian He and Daniel Templeton, could you assign this jira to me and help to check the patch into trunk?
        Thanks.

        Show
        iceberg565 NING DING added a comment - Jian He and Daniel Templeton , could you assign this jira to me and help to check the patch into trunk? Thanks.
        Hide
        jianhe Jian He added a comment -

        NING DING, added you to the contributor list. Assigned this to you. You can also now assign jira to yourself.
        Committing this.

        Show
        jianhe Jian He added a comment - NING DING , added you to the contributor list. Assigned this to you. You can also now assign jira to yourself. Committing this.
        Hide
        jianhe Jian He added a comment -

        Committed to trunk, branch-2, branch-2.7.
        NING DING, congratulations on your first YARN patch !

        Show
        jianhe Jian He added a comment - Committed to trunk, branch-2, branch-2.7. NING DING , congratulations on your first YARN patch !
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #8910 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8910/)
        YARN-4398. Remove unnecessary synchronization in RMStateStore. (jianhe: rev 6b9a5beb2b2f9589ef86670f2d763e8488ee5e90)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/AsyncDispatcher.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java
        • hadoop-yarn-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8910 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8910/ ) YARN-4398 . Remove unnecessary synchronization in RMStateStore. (jianhe: rev 6b9a5beb2b2f9589ef86670f2d763e8488ee5e90) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/AsyncDispatcher.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java hadoop-yarn-project/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        ABORTED: Integrated in Hadoop-Hdfs-trunk-Java8 #658 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/658/)
        YARN-4398. Remove unnecessary synchronization in RMStateStore. (jianhe: rev 6b9a5beb2b2f9589ef86670f2d763e8488ee5e90)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/AsyncDispatcher.java
        Show
        hudson Hudson added a comment - ABORTED: Integrated in Hadoop-Hdfs-trunk-Java8 #658 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/658/ ) YARN-4398 . Remove unnecessary synchronization in RMStateStore. (jianhe: rev 6b9a5beb2b2f9589ef86670f2d763e8488ee5e90) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/AsyncDispatcher.java
        Hide
        iceberg565 NING DING added a comment -

        Jian He, thank you.

        Show
        iceberg565 NING DING added a comment - Jian He , thank you.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Committed to branch-2.8.

        Show
        leftnoteasy Wangda Tan added a comment - Committed to branch-2.8.
        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        Closing the JIRA as part of 2.7.3 release.

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - Closing the JIRA as part of 2.7.3 release.

          People

          • Assignee:
            iceberg565 NING DING
            Reporter:
            iceberg565 NING DING
          • Votes:
            0 Vote for this issue
            Watchers:
            12 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development