Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-6982

Potential issue on setting AMContainerSpec#tokenConf to null before app is completed

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-beta1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      While reviewing patch for YARN-65, I found that many places RMAppImpl#submissionContext sets containerLaunchcontext#setTokensConf has been set to null i.e

      // set the memory free
      app.submissionContext.getAMContainerSpec().setTokensConf(null);
      

      This appears be a issue if application is updated may be because queue move or lifetime or priority, then submission context will be restored again into state store. Consider after app update, if RM is restarted then submission context will have null tokenConf. This could be a potential issue.

        Issue Links

          Activity

          Hide
          manirajv06@gmail.com Manikandan R added a comment - - edited

          Rohith Sharma K S Thanks for review and filing this.

          As this is related to YARN-65 and almost nearing completion, I am assigning this JIRA to myself. Please re-assign if someone is already working on it.

          Show
          manirajv06@gmail.com Manikandan R added a comment - - edited Rohith Sharma K S Thanks for review and filing this. As this is related to YARN-65 and almost nearing completion, I am assigning this JIRA to myself. Please re-assign if someone is already working on it.
          Hide
          manirajv06@gmail.com Manikandan R added a comment -

          Attached patch for review.

          Overall approach is to remove those lines where tokensconf has been set to null and ensuring the same has been retained even after update (for ex, change in priority) and recover events through test cases.

          Show
          manirajv06@gmail.com Manikandan R added a comment - Attached patch for review. Overall approach is to remove those lines where tokensconf has been set to null and ensuring the same has been retained even after update (for ex, change in priority) and recover events through test cases.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
                trunk Compile Tests
          +1 mvninstall 14m 54s trunk passed
          +1 compile 0m 37s trunk passed
          +1 checkstyle 0m 29s trunk passed
          +1 mvnsite 0m 38s trunk passed
          +1 findbugs 1m 3s trunk passed
          +1 javadoc 0m 23s trunk passed
                Patch Compile Tests
          +1 mvninstall 0m 35s the patch passed
          +1 compile 0m 34s the patch passed
          +1 javac 0m 34s the patch passed
          +1 checkstyle 0m 36s the patch passed
          +1 mvnsite 0m 40s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 29s the patch passed
          +1 javadoc 0m 24s the patch passed
                Other Tests
          -1 unit 46m 18s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          70m 38s



          Reason Tests
          Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation
            hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation
          Timed out junit tests org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue YARN-6982
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12882342/YARN-6982.001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 2ce95d95421a 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / f9a0e23
          Default Java 1.8.0_144
          findbugs v3.1.0-RC1
          unit https://builds.apache.org/job/PreCommit-YARN-Build/16958/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16958/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/16958/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.       trunk Compile Tests +1 mvninstall 14m 54s trunk passed +1 compile 0m 37s trunk passed +1 checkstyle 0m 29s trunk passed +1 mvnsite 0m 38s trunk passed +1 findbugs 1m 3s trunk passed +1 javadoc 0m 23s trunk passed       Patch Compile Tests +1 mvninstall 0m 35s the patch passed +1 compile 0m 34s the patch passed +1 javac 0m 34s the patch passed +1 checkstyle 0m 36s the patch passed +1 mvnsite 0m 40s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 29s the patch passed +1 javadoc 0m 24s the patch passed       Other Tests -1 unit 46m 18s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 70m 38s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation   hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation Timed out junit tests org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-6982 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12882342/YARN-6982.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 2ce95d95421a 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f9a0e23 Default Java 1.8.0_144 findbugs v3.1.0-RC1 unit https://builds.apache.org/job/PreCommit-YARN-Build/16958/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16958/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/16958/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          thanks Manikandan R for the patch!
          +1 lgtm. Would you mind taking a look at the failed tests?

          Show
          rohithsharma Rohith Sharma K S added a comment - thanks Manikandan R for the patch! +1 lgtm. Would you mind taking a look at the failed tests?
          Hide
          manirajv06@gmail.com Manikandan R added a comment -

          Rohith Sharma K S Thanks for your review. Junits failures are not related to this patch.

          Show
          manirajv06@gmail.com Manikandan R added a comment - Rohith Sharma K S Thanks for your review. Junits failures are not related to this patch.
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          committed to trunk and branch-2. thanks to Manikandan R for the patch!

          Show
          rohithsharma Rohith Sharma K S added a comment - committed to trunk and branch-2. thanks to Manikandan R for the patch!
          Hide
          hudson Hudson added a comment -

          ABORTED: Integrated in Jenkins build Hadoop-trunk-Commit #12270 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12270/)
          YARN-6982. Potential issue on setting AMContainerSpec#tokenConf to null (rohithsharmaks: rev 4cae120c619811006b26b9a95680a98732572af6)

          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java
          Show
          hudson Hudson added a comment - ABORTED: Integrated in Jenkins build Hadoop-trunk-Commit #12270 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12270/ ) YARN-6982 . Potential issue on setting AMContainerSpec#tokenConf to null (rohithsharmaks: rev 4cae120c619811006b26b9a95680a98732572af6) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java
          Hide
          manirajv06@gmail.com Manikandan R added a comment -

          Rohith Sharma K S Thanks for review and commits

          Show
          manirajv06@gmail.com Manikandan R added a comment - Rohith Sharma K S Thanks for review and commits

            People

            • Assignee:
              manirajv06@gmail.com Manikandan R
              Reporter:
              rohithsharma Rohith Sharma K S
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development