Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.23.0
    • Component/s: mrv2
    • Labels:
      None
    • Release Note:
      Hide
      New config added:

         // the maximum number of completed applications the RM keeps <name>yarn.server.resourcemanager.expire.applications.completed.max</name>
      Show
      New config added:    // the maximum number of completed applications the RM keeps <name>yarn.server.resourcemanager.expire.applications.completed.max</name>

      Description

      Today RM keeps the references of finished application for ever. Though this is not sustainable long term, it keeps
      the user experience saner. Users can revisit RM UI and check the status of their apps.

      We need to think of purging old references yet keeping the UX sane.

      1. MAPREDUCE-2649-patch-mr279.txt
        34 kB
        Thomas Graves
      2. MAPREDUCE-2649-v2.patch
        30 kB
        Thomas Graves
      3. MAPREDUCE-2649-v3.patch
        24 kB
        Thomas Graves
      4. MAPREDUCE-2649-v4.patch
        25 kB
        Thomas Graves
      5. MAPREDUCE-2649-v5.patch
        24 kB
        Thomas Graves
      6. MAPREDUCE-2649-v6.patch
        47 kB
        Thomas Graves
      7. MAPREDUCE-2649-v7.patch
        48 kB
        Thomas Graves

        Issue Links

          Activity

          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Vinod Kumar Vavilapalli made changes -
          Link This issue relates to MAPREDUCE-2941 [ MAPREDUCE-2941 ]
          Vinod Kumar Vavilapalli made changes -
          Link This issue relates to MAPREDUCE-2953 [ MAPREDUCE-2953 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #779 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/779/)
          MAPREDUCE-2649. Handling of finished applications in RM. Contributed by Thomas Graves.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1160521
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java
          • /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
          • /hadoop/common/trunk/hadoop-mapreduce/hadoop-mr-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
          • /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml
          • /hadoop/common/trunk/hadoop-mapreduce/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManagerEvent.java
          • /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java
          • /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMConfig.java
          • /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java
          • /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
          • /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestRMAppTransitions.java
          • /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManagerEventType.java
          • /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java
          • /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManagerSubmitEvent.java
          • /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/MockRMApp.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #779 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/779/ ) MAPREDUCE-2649 . Handling of finished applications in RM. Contributed by Thomas Graves. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1160521 Files : /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java /hadoop/common/trunk/hadoop-mapreduce/hadoop-mr-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml /hadoop/common/trunk/hadoop-mapreduce/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManagerEvent.java /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMConfig.java /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestRMAppTransitions.java /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManagerEventType.java /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManagerSubmitEvent.java /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/MockRMApp.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #763 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/763/)
          MAPREDUCE-2649. Handling of finished applications in RM. Contributed by Thomas Graves.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1160521
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java
          • /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
          • /hadoop/common/trunk/hadoop-mapreduce/hadoop-mr-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
          • /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml
          • /hadoop/common/trunk/hadoop-mapreduce/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManagerEvent.java
          • /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java
          • /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMConfig.java
          • /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java
          • /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
          • /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestRMAppTransitions.java
          • /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManagerEventType.java
          • /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java
          • /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManagerSubmitEvent.java
          • /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/MockRMApp.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #763 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/763/ ) MAPREDUCE-2649 . Handling of finished applications in RM. Contributed by Thomas Graves. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1160521 Files : /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java /hadoop/common/trunk/hadoop-mapreduce/hadoop-mr-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml /hadoop/common/trunk/hadoop-mapreduce/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManagerEvent.java /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMConfig.java /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestRMAppTransitions.java /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManagerEventType.java /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManagerSubmitEvent.java /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/MockRMApp.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #771 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/771/)
          MAPREDUCE-2649. Handling of finished applications in RM. Contributed by Thomas Graves.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1160521
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java
          • /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
          • /hadoop/common/trunk/hadoop-mapreduce/hadoop-mr-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
          • /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml
          • /hadoop/common/trunk/hadoop-mapreduce/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManagerEvent.java
          • /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java
          • /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMConfig.java
          • /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java
          • /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
          • /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestRMAppTransitions.java
          • /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManagerEventType.java
          • /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java
          • /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManagerSubmitEvent.java
          • /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/MockRMApp.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #771 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/771/ ) MAPREDUCE-2649 . Handling of finished applications in RM. Contributed by Thomas Graves. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1160521 Files : /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java /hadoop/common/trunk/hadoop-mapreduce/hadoop-mr-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml /hadoop/common/trunk/hadoop-mapreduce/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManagerEvent.java /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMConfig.java /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestRMAppTransitions.java /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManagerEventType.java /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManagerSubmitEvent.java /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/MockRMApp.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #848 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/848/)
          MAPREDUCE-2649. Handling of finished applications in RM. Contributed by Thomas Graves.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1160521
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java
          • /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
          • /hadoop/common/trunk/hadoop-mapreduce/hadoop-mr-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
          • /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml
          • /hadoop/common/trunk/hadoop-mapreduce/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManagerEvent.java
          • /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java
          • /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMConfig.java
          • /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java
          • /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
          • /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestRMAppTransitions.java
          • /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManagerEventType.java
          • /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java
          • /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManagerSubmitEvent.java
          • /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/MockRMApp.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #848 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/848/ ) MAPREDUCE-2649 . Handling of finished applications in RM. Contributed by Thomas Graves. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1160521 Files : /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java /hadoop/common/trunk/hadoop-mapreduce/hadoop-mr-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml /hadoop/common/trunk/hadoop-mapreduce/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManagerEvent.java /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMConfig.java /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestRMAppTransitions.java /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManagerEventType.java /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManagerSubmitEvent.java /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/MockRMApp.java
          Arun C Murthy made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          Arun C Murthy added a comment -

          I just committed this. Thanks Thomas!

          Show
          Arun C Murthy added a comment - I just committed this. Thanks Thomas!
          Hide
          Arun C Murthy added a comment -

          Good point, sorry I missed it.

          I'll commit this asap. Thanks.

          Show
          Arun C Murthy added a comment - Good point, sorry I missed it. I'll commit this asap. Thanks.
          Hide
          Thomas Graves added a comment -

          To be more clear its in the patch already..

          + ApplicationId applicationId = submissionContext.getApplicationId();
          + if (rmContext.getRMApps().get(applicationId) != null)

          { throw new IOException("Application with id " + applicationId + " is already present! Cannot add a duplicate!"); }
          Show
          Thomas Graves added a comment - To be more clear its in the patch already.. + ApplicationId applicationId = submissionContext.getApplicationId(); + if (rmContext.getRMApps().get(applicationId) != null) { throw new IOException("Application with id " + applicationId + " is already present! Cannot add a duplicate!"); }
          Hide
          Thomas Graves added a comment -

          Actually I left that check in ClientRMService already..

          Show
          Thomas Graves added a comment - Actually I left that check in ClientRMService already..
          Hide
          Arun C Murthy added a comment -

          I see no good way around this.

          One option is to do this check in ClientRMService to shorten the window...

          Show
          Arun C Murthy added a comment - I see no good way around this. One option is to do this check in ClientRMService to shorten the window...
          Hide
          Arun C Murthy added a comment -

          Thomas the patch looks fine, I fixed a minor package name problem for TestAppManager and ran all the tests.

          One minor issue: looks like there is no way for the job-client to understand that it submitted the same app twice since the error is silently lost in RMAppManager with:

          
                if (rmContext.getRMApps().putIfAbsent(applicationId, application) != null) {
                  LOG.info("Application with id " + applicationId + 
                      " is already present! Cannot add a duplicate!");
                  // don't send event through dispatcher as it will be handled by app already
                  // present with this id.
                  application.handle(new RMAppRejectedEvent(applicationId,
                      "Application with this id is already present! Cannot add a duplicate!"));
                  } else {
                    // ...
          

          Previously it was caught by the ClientRMService who then threw an exception...

          I see no good way around this.

          Thoughts?

          Show
          Arun C Murthy added a comment - Thomas the patch looks fine, I fixed a minor package name problem for TestAppManager and ran all the tests. One minor issue: looks like there is no way for the job-client to understand that it submitted the same app twice since the error is silently lost in RMAppManager with: if (rmContext.getRMApps().putIfAbsent(applicationId, application) != null ) { LOG.info( "Application with id " + applicationId + " is already present! Cannot add a duplicate!" ); // don't send event through dispatcher as it will be handled by app already // present with this id. application.handle( new RMAppRejectedEvent(applicationId, "Application with this id is already present! Cannot add a duplicate!" )); } else { // ... Previously it was caught by the ClientRMService who then threw an exception... I see no good way around this. Thoughts?
          Hide
          Thomas Graves added a comment -

          Since there isn't any test-patch script yet I ran all the tests. They all passed:

          Ran:
          mvn clean install assembly:assembly

          [INFO] ------------------------------------------------------------------------
          [INFO] Reactor Summary:[INFO]
          [INFO] hadoop-yarn-api ................................... SUCCESS [21.917s]
          [INFO] hadoop-yarn-common ................................ SUCCESS [9.999s]
          [INFO] hadoop-yarn-server-common ......................... SUCCESS [7.904s]
          [INFO] hadoop-yarn-server-nodemanager .................... SUCCESS [35.353s]
          [INFO] hadoop-yarn-server-resourcemanager ................ SUCCESS [38.603s]
          [INFO] hadoop-yarn-server-tests .......................... SUCCESS [7.361s]
          [INFO] hadoop-yarn-server ................................ SUCCESS [0.069s]
          [INFO] hadoop-yarn ....................................... SUCCESS [0.079s]
          [INFO] hadoop-mapreduce-client-core ...................... SUCCESS [6.682s]
          [INFO] hadoop-mapreduce-client-common .................... SUCCESS [5.401s]
          [INFO] hadoop-mapreduce-client-shuffle ................... SUCCESS [2.287s]
          [INFO] hadoop-mapreduce-client-app ....................... SUCCESS [1:39.490s]
          [INFO] hadoop-mapreduce-client-hs ........................ SUCCESS [5.548s]
          [INFO] hadoop-mapreduce-client-jobclient ................. SUCCESS [3:24.055s]
          [INFO] hadoop-mapreduce-client ........................... SUCCESS [0.052s]
          [INFO] hadoop-mapreduce .................................. SUCCESS [58.003s]
          [INFO] ------------------------------------------------------------------------
          [INFO] BUILD SUCCESS
          [INFO] ------------------------------------------------------------------------
          [INFO] Total time: 8:23.395s
          [INFO] Finished at: Mon Aug 22 19:05:14 UTC 2011
          [INFO] Final Memory: 41M/297M
          [INFO] ------------------------------------------------------------------------

          Show
          Thomas Graves added a comment - Since there isn't any test-patch script yet I ran all the tests. They all passed: Ran: mvn clean install assembly:assembly [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] hadoop-yarn-api ................................... SUCCESS [21.917s] [INFO] hadoop-yarn-common ................................ SUCCESS [9.999s] [INFO] hadoop-yarn-server-common ......................... SUCCESS [7.904s] [INFO] hadoop-yarn-server-nodemanager .................... SUCCESS [35.353s] [INFO] hadoop-yarn-server-resourcemanager ................ SUCCESS [38.603s] [INFO] hadoop-yarn-server-tests .......................... SUCCESS [7.361s] [INFO] hadoop-yarn-server ................................ SUCCESS [0.069s] [INFO] hadoop-yarn ....................................... SUCCESS [0.079s] [INFO] hadoop-mapreduce-client-core ...................... SUCCESS [6.682s] [INFO] hadoop-mapreduce-client-common .................... SUCCESS [5.401s] [INFO] hadoop-mapreduce-client-shuffle ................... SUCCESS [2.287s] [INFO] hadoop-mapreduce-client-app ....................... SUCCESS [1:39.490s] [INFO] hadoop-mapreduce-client-hs ........................ SUCCESS [5.548s] [INFO] hadoop-mapreduce-client-jobclient ................. SUCCESS [3:24.055s] [INFO] hadoop-mapreduce-client ........................... SUCCESS [0.052s] [INFO] hadoop-mapreduce .................................. SUCCESS [58.003s] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 8:23.395s [INFO] Finished at: Mon Aug 22 19:05:14 UTC 2011 [INFO] Final Memory: 41M/297M [INFO] ------------------------------------------------------------------------
          Hide
          Arun C Murthy added a comment -

          Is this better in MR2 because the state on the RM is much smaller? Or is there still per-container state being retained?

          It's better in MR2 since state-per-app is constant and very small, unlike in MR1 where state-per-app is unpredictable (depends on #tasks, Counters etc.)

          Show
          Arun C Murthy added a comment - Is this better in MR2 because the state on the RM is much smaller? Or is there still per-container state being retained? It's better in MR2 since state-per-app is constant and very small, unlike in MR1 where state-per-app is unpredictable (depends on #tasks, Counters etc.)
          Hide
          Todd Lipcon added a comment -

          Coming late to see this patch. One question from an MR2 n00b: in MR1, these configs were very error prone since different jobs took up wildly variant amounts of RAM. So a value of 100 retained jobs would work fine for small jobs but fail miserably with OOME if bigger jobs were submitted. Then we added all of the limits to try to curtail this, but it's a bit of a stopgap rather than a real solution.

          Is this better in MR2 because the state on the RM is much smaller? Or is there still per-container state being retained?

          One thing we should consider is to adopt an interface like HBase's "HeapSize" – objects that are going to be retained/cached need to implement it, and then the limit scan be set based on memory usage rather than strict counts.

          Show
          Todd Lipcon added a comment - Coming late to see this patch. One question from an MR2 n00b: in MR1, these configs were very error prone since different jobs took up wildly variant amounts of RAM. So a value of 100 retained jobs would work fine for small jobs but fail miserably with OOME if bigger jobs were submitted. Then we added all of the limits to try to curtail this, but it's a bit of a stopgap rather than a real solution. Is this better in MR2 because the state on the RM is much smaller? Or is there still per-container state being retained? One thing we should consider is to adopt an interface like HBase's "HeapSize" – objects that are going to be retained/cached need to implement it, and then the limit scan be set based on memory usage rather than strict counts.
          Thomas Graves made changes -
          Attachment MAPREDUCE-2649-v7.patch [ 12490962 ]
          Hide
          Thomas Graves added a comment -

          add test for new state transition in RMAppImpl.java

          Show
          Thomas Graves added a comment - add test for new state transition in RMAppImpl.java
          Thomas Graves made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Thomas Graves made changes -
          Attachment MAPREDUCE-2649-v6.patch [ 12490960 ]
          Hide
          Thomas Graves added a comment -

          Updates to have RMAppManager.

          Show
          Thomas Graves added a comment - Updates to have RMAppManager.
          Thomas Graves made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Thomas Graves added a comment -

          making changes to have common RMAppManager.

          Show
          Thomas Graves added a comment - making changes to have common RMAppManager.
          Hide
          Thomas Graves added a comment -

          Thanks for the review, sorry for my delay responding, I was out.

          I agree, it makes sense to have an RMAppManager entity doing all the manipulations. I'll make the changes here.

          Show
          Thomas Graves added a comment - Thanks for the review, sorry for my delay responding, I was out. I agree, it makes sense to have an RMAppManager entity doing all the manipulations. I'll make the changes here.
          Hide
          Arun C Murthy added a comment -

          Thomas, the patch looks good, thanks!

          One thing that bothers me, nothing related to your patch of course, is that RMContextImpl.applications is manipulated in ClientRMService (submitApplication adds an RMApp to the list) and now, with your patch, RMAppExpirer removes it from the same list.

          Should we consider renaming RMAppExpirer to something like RMAppManager which does both the add and delete? Thus ClientRMService should send an event to RMAppManager which then creates the RMApp, adds to list and sends other notifications. Similarly APP_COMPLETED comes over to RMAppManager which then deletes RMApps as in your patch.
          Thoughts? I'm fine doing this in another jira too... but it might improve the overall system rightaway. Thanks.

          Show
          Arun C Murthy added a comment - Thomas, the patch looks good, thanks! One thing that bothers me, nothing related to your patch of course, is that RMContextImpl.applications is manipulated in ClientRMService (submitApplication adds an RMApp to the list) and now, with your patch, RMAppExpirer removes it from the same list. Should we consider renaming RMAppExpirer to something like RMAppManager which does both the add and delete? Thus ClientRMService should send an event to RMAppManager which then creates the RMApp, adds to list and sends other notifications. Similarly APP_COMPLETED comes over to RMAppManager which then deletes RMApps as in your patch. Thoughts? I'm fine doing this in another jira too... but it might improve the overall system rightaway. Thanks.
          Thomas Graves made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Thomas Graves made changes -
          Attachment MAPREDUCE-2649-v5.patch [ 12490239 ]
          Hide
          Thomas Graves added a comment -

          removed some uneeded code.

          Show
          Thomas Graves added a comment - removed some uneeded code.
          Thomas Graves made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Thomas Graves made changes -
          Attachment MAPREDUCE-2649-v4.patch [ 12490175 ]
          Hide
          Thomas Graves added a comment -

          updated the yarn-default.xml to include new config.

          Show
          Thomas Graves added a comment - updated the yarn-default.xml to include new config.
          Hide
          Mahadev konar added a comment -

          Thomas, could you please add the property to yarn-default.xml as well. We need to populate our yarn-default.xml which currently is missing quite a few of the config knobs.

          Show
          Mahadev konar added a comment - Thomas, could you please add the property to yarn-default.xml as well. We need to populate our yarn-default.xml which currently is missing quite a few of the config knobs.
          Hide
          Thomas Graves added a comment -

          Reworked the patch to send an event when the RMapp completes and have the expirer handle it. It expires apps based on a maximum number of completed jobs.

          Show
          Thomas Graves added a comment - Reworked the patch to send an event when the RMapp completes and have the expirer handle it. It expires apps based on a maximum number of completed jobs.
          Thomas Graves made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Release Note The new configs added are:

            // time(in ms) between when the expire applications thread checks <name>yarn.server.resourcemanager.expire.applications.monitor.interval</name>
           
            // the length of time(in ms) the RM keeps a completed application
          <name>yarn.server.resourcemanager.expire.applications.interval</name>

             // the maximum number of completed applications per user RM keeps <name>yarn.server.resourcemanager.expire.applications.user.completed.max</name>
          New config added:

             // the maximum number of completed applications the RM keeps <name>yarn.server.resourcemanager.expire.applications.completed.max</name>
          Thomas Graves made changes -
          Attachment MAPREDUCE-2649-v3.patch [ 12490171 ]
          Thomas Graves made changes -
          Link This issue depends on MAPREDUCE-2781 [ MAPREDUCE-2781 ]
          Hide
          Thomas Graves added a comment -

          I'm fine with the simpler model. This algorithm was based on how the jobtracker does it now. It will simplify things a lot and we won't need a separate thread.

          If no one else has objects I'll make the changes.

          Show
          Thomas Graves added a comment - I'm fine with the simpler model. This algorithm was based on how the jobtracker does it now. It will simplify things a lot and we won't need a separate thread. If no one else has objects I'll make the changes.
          Arun C Murthy made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Arun C Murthy added a comment -

          Thomas, sorry to come in late.

          As you put it, it's much easier to do with the RMApp state-machine now, could you please use that? This way we don't need to write a lot of redundant code.

          Also, seems to me that we could do with a much simpler model:
          Just define a maximum number of in-memory completed jobs (say 10k) and not bother with jobs-per-user etc.

          Thoughts? Do you see any issues with the simpler model?

          Show
          Arun C Murthy added a comment - Thomas, sorry to come in late. As you put it, it's much easier to do with the RMApp state-machine now, could you please use that? This way we don't need to write a lot of redundant code. Also, seems to me that we could do with a much simpler model: Just define a maximum number of in-memory completed jobs (say 10k) and not bother with jobs-per-user etc. Thoughts? Do you see any issues with the simpler model?
          Hide
          Thomas Graves added a comment -

          It might be a nice enhancement to have this more event driven - which appears easier now with RM reworked. When app finishes event could be sent to the expire monitor and it handles it. I think that can happen as a follow on change though.

          Show
          Thomas Graves added a comment - It might be a nice enhancement to have this more event driven - which appears easier now with RM reworked. When app finishes event could be sent to the expire monitor and it handles it. I think that can happen as a follow on change though.
          Thomas Graves made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Release Note The new configs added are:

          + // time(in ms) between when the retire applications thread checks
          <name>yarn.server.resourcemanager.retireApps.check.interval</name>

          + // the length of time(in ms) the RM keeps a completed application
          <name>yarn.server.resourcemanager.retireApps.keep.period</name>

          + // the maximum number of completed applications per user RM keeps
          <name>yarn.server.resourcemanager.retireApps.user.completed.max</name>
          The new configs added are:

            // time(in ms) between when the expire applications thread checks <name>yarn.server.resourcemanager.expire.applications.monitor.interval</name>
           
            // the length of time(in ms) the RM keeps a completed application
          <name>yarn.server.resourcemanager.expire.applications.interval</name>

             // the maximum number of completed applications per user RM keeps <name>yarn.server.resourcemanager.expire.applications.user.completed.max</name>
          Thomas Graves made changes -
          Attachment MAPREDUCE-2649-v2.patch [ 12489851 ]
          Hide
          Thomas Graves added a comment -

          patch reworked.

          Show
          Thomas Graves added a comment - patch reworked.
          Hide
          Thomas Graves added a comment -

          2781 fixes the setting of application start and finish time which this uses to expire applications.

          Show
          Thomas Graves added a comment - 2781 fixes the setting of application start and finish time which this uses to expire applications.
          Thomas Graves made changes -
          Link This issue depends on MAPREDUCE-2781 [ MAPREDUCE-2781 ]
          Thomas Graves made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Thomas Graves added a comment -

          reworking patch since RM got changed a lot.

          Show
          Thomas Graves added a comment - reworking patch since RM got changed a lot.
          Hide
          Robert Joseph Evans added a comment -

          Just a few comments about the patch.

          This is very minor but it would be good to make APPLICATION_FINISHTIME_COMPARATOR a static inner class of ApplicationsManagerImpl instead of an anonymous class. It reduces the amount of unused data in the class, even though it is only instantiated once.

          There is no error handling on RetireApplicationsThread. If an Error or RuntimeException occurs then the thread will die silently. However I cannot find an example of how it should be handled in the YARN code, so it might be fine the way it is.

          Show
          Robert Joseph Evans added a comment - Just a few comments about the patch. This is very minor but it would be good to make APPLICATION_FINISHTIME_COMPARATOR a static inner class of ApplicationsManagerImpl instead of an anonymous class. It reduces the amount of unused data in the class, even though it is only instantiated once. There is no error handling on RetireApplicationsThread. If an Error or RuntimeException occurs then the thread will die silently. However I cannot find an example of how it should be handled in the YARN code, so it might be fine the way it is.
          Thomas Graves made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Release Note The new configs added are:

          + // time(in ms) between when the retire applications thread checks
          <name>yarn.server.resourcemanager.retireApps.check.interval</name>

          + // the length of time(in ms) the RM keeps a completed application
          <name>yarn.server.resourcemanager.retireApps.keep.period</name>

          + // the maximum number of completed applications per user RM keeps
          <name>yarn.server.resourcemanager.retireApps.user.completed.max</name>
          Thomas Graves made changes -
          Fix Version/s 0.23.0 [ 12315570 ]
          Hide
          Thomas Graves added a comment -

          I've added a thread to the application manager that will retire the applications either when you hit a configurable time limit after app has finished or when the hit a max number per user. How often the thread runs to check is also configurable. The applications are simply removed from the RM application map and thus removed from the UI.

          The new configs added are:

          + // time(in ms) between when the retire applications thread checks
          <name>yarn.server.resourcemanager.retireApps.check.interval</name>

          + // the length of time(in ms) the RM keeps a completed application
          <name>yarn.server.resourcemanager.retireApps.keep.period</name>

          + // the maximum number of completed applications per user RM keeps
          <name>yarn.server.resourcemanager.retireApps.user.completed.max</name>

          Show
          Thomas Graves added a comment - I've added a thread to the application manager that will retire the applications either when you hit a configurable time limit after app has finished or when the hit a max number per user. How often the thread runs to check is also configurable. The applications are simply removed from the RM application map and thus removed from the UI. The new configs added are: + // time(in ms) between when the retire applications thread checks <name>yarn.server.resourcemanager.retireApps.check.interval</name> + // the length of time(in ms) the RM keeps a completed application <name>yarn.server.resourcemanager.retireApps.keep.period</name> + // the maximum number of completed applications per user RM keeps <name>yarn.server.resourcemanager.retireApps.user.completed.max</name>
          Thomas Graves made changes -
          Field Original Value New Value
          Attachment MAPREDUCE-2649-patch-mr279.txt [ 12488105 ]
          Thomas Graves created issue -

            People

            • Assignee:
              Thomas Graves
              Reporter:
              Thomas Graves
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development