Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-3857

Memory leak in ResourceManager with SIMPLE mode

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      We register the ClientTokenMasterKey to avoid client may hold an invalid ClientToken after RM restarts. In SIMPLE mode, we register Pair<ApplicationAttemptId, null> , But we never remove it from HashMap, as unregister only runing while in Security mode, so memory leak coming.

      1. hadoop-yarn-server-resourcemanager.patch
        1 kB
        mujunchao
      2. YARN-3857-1.patch
        6 kB
        mujunchao
      3. YARN-3857-2.patch
        5 kB
        mujunchao
      4. YARN-3857-3.patch
        5 kB
        mujunchao
      5. YARN-3857-4.patch
        5 kB
        mujunchao

        Activity

        Hide
        mujunchao mujunchao added a comment -

        never register org.apache.hadoop.yarn.server.resourcemanager.security.ClientToAMTokenSecretManagerInRM.masterKeys while SecretKey is null.

        Show
        mujunchao mujunchao added a comment - never register org.apache.hadoop.yarn.server.resourcemanager.security.ClientToAMTokenSecretManagerInRM.masterKeys while SecretKey is null.
        Hide
        zxu zhihai xu added a comment -

        Hi mujunchao, thanks for reporting and working on this issue.
        It is a nice catch. I see why this is a critical issue. For non-secure cluster, the more completed jobs, the more entries with null value will be left in ClientToAMTokenSecretManagerInRM#masterKeys. You patch makes sense to me, since we only call unRegisterApplication in secure mode, we should also call registerApplication in secure mode to match unRegisterApplication.
        Could you add a test case in your patch? You can do something similar as TestRMAppAttemptTransitions#testGetClientToken for non-secure mode.

        Show
        zxu zhihai xu added a comment - Hi mujunchao , thanks for reporting and working on this issue. It is a nice catch. I see why this is a critical issue. For non-secure cluster, the more completed jobs, the more entries with null value will be left in ClientToAMTokenSecretManagerInRM#masterKeys . You patch makes sense to me, since we only call unRegisterApplication in secure mode, we should also call registerApplication in secure mode to match unRegisterApplication . Could you add a test case in your patch? You can do something similar as TestRMAppAttemptTransitions#testGetClientToken for non-secure mode.
        Hide
        devaraj.k Devaraj K added a comment -

        Thanks mujunchao for reporting and working in this. I am assigning this issue to you.

        Adding to zhihai xu, Can you also take care of the naming convention for the patch like <JIRA-ID>-<patch-version>.patch?

        Show
        devaraj.k Devaraj K added a comment - Thanks mujunchao for reporting and working in this. I am assigning this issue to you. Adding to zhihai xu , Can you also take care of the naming convention for the patch like <JIRA-ID>-<patch-version>.patch?
        Hide
        brahmareddy Brahma Reddy Battula added a comment -

        Nice Catch!!

        Show
        brahmareddy Brahma Reddy Battula added a comment - Nice Catch!!
        Hide
        mujunchao mujunchao added a comment -

        Thanks for your reviewing, I will give the test case.

        Show
        mujunchao mujunchao added a comment - Thanks for your reviewing, I will give the test case.
        Hide
        mujunchao mujunchao added a comment -

        Ok, got, thanks Devaraj for reviewing.

        Show
        mujunchao mujunchao added a comment - Ok, got, thanks Devaraj for reviewing.
        Hide
        mujunchao mujunchao added a comment -

        add test case.

        Show
        mujunchao mujunchao added a comment - add test case.
        Hide
        zxu zhihai xu added a comment -

        thanks for the updated patch mujunchao!
        The patch looks most good to me, some nits:

        1. Add @VisibleForTesting before function hasMasterKey to mark this function used for test only. So you can remove the comment // Only for test
        2. It looks like the code in testNoSecureNoRegistClientToken are similar as testRegistClientTokenInSecure. Can we merge testNoSecureNoRegistClientToken with testRegistClientTokenInSecure to one test? We can rename the test as testApplicationAttemptMasterKey. You can check isMasterKeyExisted based on isSecurityEnabled. You can change your comments can not get ClientToken/can get ClientToken to can not get MasterKey/can get MasterKey
        Show
        zxu zhihai xu added a comment - thanks for the updated patch mujunchao ! The patch looks most good to me, some nits: Add @VisibleForTesting before function hasMasterKey to mark this function used for test only. So you can remove the comment // Only for test It looks like the code in testNoSecureNoRegistClientToken are similar as testRegistClientTokenInSecure . Can we merge testNoSecureNoRegistClientToken with testRegistClientTokenInSecure to one test? We can rename the test as testApplicationAttemptMasterKey . You can check isMasterKeyExisted based on isSecurityEnabled . You can change your comments can not get ClientToken / can get ClientToken to can not get MasterKey / can get MasterKey
        Hide
        mujunchao mujunchao added a comment -

        update test case, thanks for xuzhihai's reviewing.

        Show
        mujunchao mujunchao added a comment - update test case, thanks for xuzhihai's reviewing.
        Hide
        mujunchao mujunchao added a comment -

        thanks for your reviewing, have updated the test case, thanks!

        Show
        mujunchao mujunchao added a comment - thanks for your reviewing, have updated the test case, thanks!
        Hide
        mujunchao mujunchao added a comment -

        thanks for your reviewing, have updated the test case, thanks!

        Show
        mujunchao mujunchao added a comment - thanks for your reviewing, have updated the test case, thanks!
        Hide
        zxu zhihai xu added a comment -

        thanks for updating the patch quickly.
        The new patch looks good in general. It'd be nice to address a few small nits, mostly regarding the comment and style.

        1. Change // this is to test master key is saved in the state store to // this is to test master key is saved in the secret manager
        2. Remove comment //assumeTrue(!isSecurityEnabled);
        3. Change if(isSecurityEnabled) to if (isSecurityEnabled)
        4. Could you fix the indentation? It should be indented by 2(new paragraph) or 4 bytes.
          +1 non-binding once these are addressed. Thanks again for working on this.
        Show
        zxu zhihai xu added a comment - thanks for updating the patch quickly. The new patch looks good in general. It'd be nice to address a few small nits, mostly regarding the comment and style. Change // this is to test master key is saved in the state store to // this is to test master key is saved in the secret manager Remove comment //assumeTrue(!isSecurityEnabled); Change if(isSecurityEnabled) to if (isSecurityEnabled) Could you fix the indentation? It should be indented by 2(new paragraph) or 4 bytes. +1 non-binding once these are addressed. Thanks again for working on this.
        Hide
        devaraj.k Devaraj K added a comment -

        Thanks mujunchao for updated patch with test.

        Please take care of these comments also along with the zhihai xu comments fix.

        1.I don't think adding this new method is required. Can we just use the ClientToAMTokenSecretManagerInRM#getMasterKey() to know whether the master key present or not?

        +  
        +  @VisibleForTesting
        +  public synchronized boolean hasMasterKey(
        +	  ApplicationAttemptId applicationAttemptID) {
        +	return this.masterKeys.containsKey(applicationAttemptID);
        +  }
        

        2. I see there are some format issues in the patch w.r.t braces and indentation with spaces. Please go through the 'Making Changes' section in https://wiki.apache.org/hadoop/HowToContribute and configure your IDE according. It will be one time job and you don't have to worry next time for creating patches.

        +    if(isSecurityEnabled)
        +    {
        
        +    }
        +    else
        +    {
        

        4. Remove unused imports in RMAppAttemptImpl.java.

        Show
        devaraj.k Devaraj K added a comment - Thanks mujunchao for updated patch with test. Please take care of these comments also along with the zhihai xu comments fix. 1.I don't think adding this new method is required. Can we just use the ClientToAMTokenSecretManagerInRM#getMasterKey() to know whether the master key present or not? + + @VisibleForTesting + public synchronized boolean hasMasterKey( + ApplicationAttemptId applicationAttemptID) { + return this.masterKeys.containsKey(applicationAttemptID); + } 2. I see there are some format issues in the patch w.r.t braces and indentation with spaces. Please go through the 'Making Changes' section in https://wiki.apache.org/hadoop/HowToContribute and configure your IDE according. It will be one time job and you don't have to worry next time for creating patches. + if(isSecurityEnabled) + { + } + else + { 4. Remove unused imports in RMAppAttemptImpl.java.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 16m 9s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 7m 41s There were no new javac warning messages.
        +1 javadoc 9m 38s There were no new javadoc warning messages.
        +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 0m 48s The applied patch generated 7 new checkstyle issues (total was 125, now 129).
        -1 whitespace 0m 0s The patch has 6 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 21s mvn install still works.
        +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
        +1 findbugs 1m 25s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        -1 yarn tests 51m 1s Tests failed in hadoop-yarn-server-resourcemanager.
            89m 3s  



        Reason Tests
        Failed unit tests hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRMRPCNodeUpdates
          hadoop.yarn.server.resourcemanager.TestResourceTrackerService
          hadoop.yarn.server.resourcemanager.rmapp.TestRMAppTransitions



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12744651/YARN-3857-2.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / d66302e
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8492/artifact/patchprocess/diffcheckstylehadoop-yarn-server-resourcemanager.txt
        whitespace https://builds.apache.org/job/PreCommit-YARN-Build/8492/artifact/patchprocess/whitespace.txt
        hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8492/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8492/testReport/
        Java 1.7.0_55
        uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/8492/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 9s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 41s There were no new javac warning messages. +1 javadoc 9m 38s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 0m 48s The applied patch generated 7 new checkstyle issues (total was 125, now 129). -1 whitespace 0m 0s The patch has 6 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 21s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 1m 25s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 yarn tests 51m 1s Tests failed in hadoop-yarn-server-resourcemanager.     89m 3s   Reason Tests Failed unit tests hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRMRPCNodeUpdates   hadoop.yarn.server.resourcemanager.TestResourceTrackerService   hadoop.yarn.server.resourcemanager.rmapp.TestRMAppTransitions Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12744651/YARN-3857-2.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / d66302e checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8492/artifact/patchprocess/diffcheckstylehadoop-yarn-server-resourcemanager.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/8492/artifact/patchprocess/whitespace.txt hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8492/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8492/testReport/ Java 1.7.0_55 uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8492/console This message was automatically generated.
        Hide
        devaraj.k Devaraj K added a comment -

        Cancelling patch to address the above comments.

        Show
        devaraj.k Devaraj K added a comment - Cancelling patch to address the above comments.
        Hide
        mujunchao mujunchao added a comment -

        fix the comments and format the code.

        Show
        mujunchao mujunchao added a comment - fix the comments and format the code.
        Hide
        mujunchao mujunchao added a comment -

        thanks for your reviewing, have fixed the comments and the indentation.

        Show
        mujunchao mujunchao added a comment - thanks for your reviewing, have fixed the comments and the indentation.
        Hide
        mujunchao mujunchao added a comment -

        Thanks Devar for reviewing,
        1. I think ClientToAMTokenSecretManagerInRM#hasMasterKey() is necessary, in this case the value is null, so i new function to recognize it.
        2. have fixed.
        3. As i use annotation @VisibleForTesting,in my ide, need to import com.google.common.annotations.VisibleForTesting.

        Show
        mujunchao mujunchao added a comment - Thanks Devar for reviewing, 1. I think ClientToAMTokenSecretManagerInRM#hasMasterKey() is necessary, in this case the value is null, so i new function to recognize it. 2. have fixed. 3. As i use annotation @VisibleForTesting,in my ide, need to import com.google.common.annotations.VisibleForTesting.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 16m 24s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 7m 52s There were no new javac warning messages.
        +1 javadoc 9m 54s There were no new javadoc warning messages.
        +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 0m 50s There were no new checkstyle issues.
        -1 whitespace 0m 0s The patch has 5 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 23s mvn install still works.
        +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
        +1 findbugs 1m 28s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 yarn tests 53m 37s Tests passed in hadoop-yarn-server-resourcemanager.
            92m 29s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12747980/YARN-3857-3.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 90b5104
        whitespace https://builds.apache.org/job/PreCommit-YARN-Build/8751/artifact/patchprocess/whitespace.txt
        hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8751/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8751/testReport/
        Java 1.7.0_55
        uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/8751/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 24s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 52s There were no new javac warning messages. +1 javadoc 9m 54s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 50s There were no new checkstyle issues. -1 whitespace 0m 0s The patch has 5 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 23s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 1m 28s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 53m 37s Tests passed in hadoop-yarn-server-resourcemanager.     92m 29s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12747980/YARN-3857-3.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 90b5104 whitespace https://builds.apache.org/job/PreCommit-YARN-Build/8751/artifact/patchprocess/whitespace.txt hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8751/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8751/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8751/console This message was automatically generated.
        Hide
        mujunchao mujunchao added a comment -

        remove the whitespace.

        Show
        mujunchao mujunchao added a comment - remove the whitespace.
        Hide
        mujunchao mujunchao added a comment -

        remove the whitespace.

        Show
        mujunchao mujunchao added a comment - remove the whitespace.
        Hide
        hadoopqa Hadoop QA added a comment -



        +1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 16m 8s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 7m 41s There were no new javac warning messages.
        +1 javadoc 9m 38s There were no new javadoc warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 0m 47s There were no new checkstyle issues.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 23s mvn install still works.
        +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse.
        +1 findbugs 1m 27s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 yarn tests 52m 42s Tests passed in hadoop-yarn-server-resourcemanager.
            90m 47s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12748411/YARN-3857-4.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 90b5104
        hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8752/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8752/testReport/
        Java 1.7.0_55
        uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/8752/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 8s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 41s There were no new javac warning messages. +1 javadoc 9m 38s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 47s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 23s mvn install still works. +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse. +1 findbugs 1m 27s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 52m 42s Tests passed in hadoop-yarn-server-resourcemanager.     90m 47s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12748411/YARN-3857-4.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 90b5104 hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8752/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8752/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8752/console This message was automatically generated.
        Hide
        zxu zhihai xu added a comment -

        +1 for the latest patch, If no objection, I will commit it tomorrow.

        Show
        zxu zhihai xu added a comment - +1 for the latest patch, If no objection, I will commit it tomorrow.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #8317 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8317/)
        YARN-3857: Memory leak in ResourceManager with SIMPLE mode. Contributed by mujunchao. (zxu: rev 3a76a010b85176f2bcb85ed6f74c25dcb8acfe4d)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/TestRMAppAttemptTransitions.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/ClientToAMTokenSecretManagerInRM.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8317 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8317/ ) YARN-3857 : Memory leak in ResourceManager with SIMPLE mode. Contributed by mujunchao. (zxu: rev 3a76a010b85176f2bcb85ed6f74c25dcb8acfe4d) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/TestRMAppAttemptTransitions.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/ClientToAMTokenSecretManagerInRM.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java
        Hide
        zxu zhihai xu added a comment -

        Thanks to mujunchao for the contribution and to Devaraj for additional review! I committed this to trunk, branch-2 and branch-2.7.

        Show
        zxu zhihai xu added a comment - Thanks to mujunchao for the contribution and to Devaraj for additional review! I committed this to trunk, branch-2 and branch-2.7.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #1022 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1022/)
        YARN-3857: Memory leak in ResourceManager with SIMPLE mode. Contributed by mujunchao. (zxu: rev 3a76a010b85176f2bcb85ed6f74c25dcb8acfe4d)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/ClientToAMTokenSecretManagerInRM.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/TestRMAppAttemptTransitions.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1022 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1022/ ) YARN-3857 : Memory leak in ResourceManager with SIMPLE mode. Contributed by mujunchao. (zxu: rev 3a76a010b85176f2bcb85ed6f74c25dcb8acfe4d) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/ClientToAMTokenSecretManagerInRM.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/TestRMAppAttemptTransitions.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #292 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/292/)
        YARN-3857: Memory leak in ResourceManager with SIMPLE mode. Contributed by mujunchao. (zxu: rev 3a76a010b85176f2bcb85ed6f74c25dcb8acfe4d)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/ClientToAMTokenSecretManagerInRM.java
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/TestRMAppAttemptTransitions.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #292 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/292/ ) YARN-3857 : Memory leak in ResourceManager with SIMPLE mode. Contributed by mujunchao. (zxu: rev 3a76a010b85176f2bcb85ed6f74c25dcb8acfe4d) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/ClientToAMTokenSecretManagerInRM.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/TestRMAppAttemptTransitions.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2219 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2219/)
        YARN-3857: Memory leak in ResourceManager with SIMPLE mode. Contributed by mujunchao. (zxu: rev 3a76a010b85176f2bcb85ed6f74c25dcb8acfe4d)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/ClientToAMTokenSecretManagerInRM.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/TestRMAppAttemptTransitions.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2219 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2219/ ) YARN-3857 : Memory leak in ResourceManager with SIMPLE mode. Contributed by mujunchao. (zxu: rev 3a76a010b85176f2bcb85ed6f74c25dcb8acfe4d) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/ClientToAMTokenSecretManagerInRM.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/TestRMAppAttemptTransitions.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #281 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/281/)
        YARN-3857: Memory leak in ResourceManager with SIMPLE mode. Contributed by mujunchao. (zxu: rev 3a76a010b85176f2bcb85ed6f74c25dcb8acfe4d)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/ClientToAMTokenSecretManagerInRM.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/TestRMAppAttemptTransitions.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java
        • hadoop-yarn-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #281 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/281/ ) YARN-3857 : Memory leak in ResourceManager with SIMPLE mode. Contributed by mujunchao. (zxu: rev 3a76a010b85176f2bcb85ed6f74c25dcb8acfe4d) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/ClientToAMTokenSecretManagerInRM.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/TestRMAppAttemptTransitions.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java hadoop-yarn-project/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #2238 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2238/)
        YARN-3857: Memory leak in ResourceManager with SIMPLE mode. Contributed by mujunchao. (zxu: rev 3a76a010b85176f2bcb85ed6f74c25dcb8acfe4d)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/ClientToAMTokenSecretManagerInRM.java
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/TestRMAppAttemptTransitions.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2238 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2238/ ) YARN-3857 : Memory leak in ResourceManager with SIMPLE mode. Contributed by mujunchao. (zxu: rev 3a76a010b85176f2bcb85ed6f74c25dcb8acfe4d) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/ClientToAMTokenSecretManagerInRM.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/TestRMAppAttemptTransitions.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #289 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/289/)
        YARN-3857: Memory leak in ResourceManager with SIMPLE mode. Contributed by mujunchao. (zxu: rev 3a76a010b85176f2bcb85ed6f74c25dcb8acfe4d)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/TestRMAppAttemptTransitions.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/ClientToAMTokenSecretManagerInRM.java
        • hadoop-yarn-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #289 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/289/ ) YARN-3857 : Memory leak in ResourceManager with SIMPLE mode. Contributed by mujunchao. (zxu: rev 3a76a010b85176f2bcb85ed6f74c25dcb8acfe4d) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/TestRMAppAttemptTransitions.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/ClientToAMTokenSecretManagerInRM.java hadoop-yarn-project/CHANGES.txt
        Hide
        sjlee0 Sangjin Lee added a comment -

        Does this issue exist in 2.6.x? Should this be backported to branch-2.6?

        Show
        sjlee0 Sangjin Lee added a comment - Does this issue exist in 2.6.x? Should this be backported to branch-2.6?
        Hide
        zxu zhihai xu added a comment -

        Yes, this issue exists in 2.6.x, I just committed this patch to branch-2.6.

        Show
        zxu zhihai xu added a comment - Yes, this issue exists in 2.6.x, I just committed this patch to branch-2.6.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #8969 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8969/)
        Update CHANGES.txt to add YARN-3857 and YARN-3535 to branch-2.6 (zxu: rev 0c3a53e5a978140e56b9ebbc82c8d04fc978e640)

        • hadoop-yarn-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8969 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8969/ ) Update CHANGES.txt to add YARN-3857 and YARN-3535 to branch-2.6 (zxu: rev 0c3a53e5a978140e56b9ebbc82c8d04fc978e640) hadoop-yarn-project/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #694 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/694/)
        Update CHANGES.txt to add YARN-3857 and YARN-3535 to branch-2.6 (zxu: rev 0c3a53e5a978140e56b9ebbc82c8d04fc978e640)

        • hadoop-yarn-project/CHANGES.txt
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #694 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/694/ ) Update CHANGES.txt to add YARN-3857 and YARN-3535 to branch-2.6 (zxu: rev 0c3a53e5a978140e56b9ebbc82c8d04fc978e640) hadoop-yarn-project/CHANGES.txt

          People

          • Assignee:
            mujunchao mujunchao
            Reporter:
            mujunchao mujunchao
          • Votes:
            0 Vote for this issue
            Watchers:
            12 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development