Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-5009

NMLeveldbStateStoreService database can grow substantially leading to longer recovery times

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Similar to the RM case in YARN-5008, I have seen state stores for nodemanagers with high container churn become significantly larger than they should be due to lack of sufficient database compaction.

      1. YARN-5009-branch-2.6.002.patch
        8 kB
        Jason Lowe
      2. YARN-5009.002.patch
        9 kB
        Jason Lowe
      3. YARN-5009.001.patch
        9 kB
        Jason Lowe

        Activity

        Hide
        jlowe Jason Lowe added a comment -

        Patch to add periodic manual compactions every hour by default. Compaction cycle interval can be configured or disabled by setting an interval of zero.

        Show
        jlowe Jason Lowe added a comment - Patch to add periodic manual compactions every hour by default. Compaction cycle interval can be configured or disabled by setting an interval of zero.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 10s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 10s Maven dependency ordering for branch
        +1 mvninstall 6m 27s trunk passed
        +1 compile 1m 43s trunk passed with JDK v1.8.0_92
        +1 compile 2m 2s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 35s trunk passed
        +1 mvnsite 1m 24s trunk passed
        +1 mvneclipse 0m 36s trunk passed
        -1 findbugs 1m 6s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in trunk has 1 extant Findbugs warnings.
        +1 javadoc 1m 19s trunk passed with JDK v1.8.0_92
        +1 javadoc 3m 48s trunk passed with JDK v1.7.0_95
        0 mvndep 0m 10s Maven dependency ordering for patch
        +1 mvninstall 1m 13s the patch passed
        +1 compile 1m 49s the patch passed with JDK v1.8.0_92
        +1 javac 1m 49s the patch passed
        +1 compile 2m 5s the patch passed with JDK v1.7.0_95
        +1 javac 2m 5s the patch passed
        +1 checkstyle 0m 33s the patch passed
        +1 mvnsite 1m 19s the patch passed
        +1 mvneclipse 0m 32s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 findbugs 3m 41s the patch passed
        +1 javadoc 1m 15s the patch passed with JDK v1.8.0_92
        +1 javadoc 3m 43s the patch passed with JDK v1.7.0_95
        +1 unit 0m 20s hadoop-yarn-api in the patch passed with JDK v1.8.0_92.
        +1 unit 1m 55s hadoop-yarn-common in the patch passed with JDK v1.8.0_92.
        +1 unit 11m 10s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_92.
        +1 unit 0m 22s hadoop-yarn-api in the patch passed with JDK v1.7.0_95.
        +1 unit 2m 14s hadoop-yarn-common in the patch passed with JDK v1.7.0_95.
        +1 unit 11m 34s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_95.
        +1 asflicense 0m 19s Patch does not generate ASF License warnings.
        66m 51s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:7b1c37a
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12801260/YARN-5009.001.patch
        JIRA Issue YARN-5009
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux c802b6bbd0d8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 6f26b66
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_92 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        findbugs https://builds.apache.org/job/PreCommit-YARN-Build/11262/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-warnings.html
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11262/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/11262/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 10s Maven dependency ordering for branch +1 mvninstall 6m 27s trunk passed +1 compile 1m 43s trunk passed with JDK v1.8.0_92 +1 compile 2m 2s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 35s trunk passed +1 mvnsite 1m 24s trunk passed +1 mvneclipse 0m 36s trunk passed -1 findbugs 1m 6s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in trunk has 1 extant Findbugs warnings. +1 javadoc 1m 19s trunk passed with JDK v1.8.0_92 +1 javadoc 3m 48s trunk passed with JDK v1.7.0_95 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 1m 13s the patch passed +1 compile 1m 49s the patch passed with JDK v1.8.0_92 +1 javac 1m 49s the patch passed +1 compile 2m 5s the patch passed with JDK v1.7.0_95 +1 javac 2m 5s the patch passed +1 checkstyle 0m 33s the patch passed +1 mvnsite 1m 19s the patch passed +1 mvneclipse 0m 32s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 3m 41s the patch passed +1 javadoc 1m 15s the patch passed with JDK v1.8.0_92 +1 javadoc 3m 43s the patch passed with JDK v1.7.0_95 +1 unit 0m 20s hadoop-yarn-api in the patch passed with JDK v1.8.0_92. +1 unit 1m 55s hadoop-yarn-common in the patch passed with JDK v1.8.0_92. +1 unit 11m 10s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_92. +1 unit 0m 22s hadoop-yarn-api in the patch passed with JDK v1.7.0_95. +1 unit 2m 14s hadoop-yarn-common in the patch passed with JDK v1.7.0_95. +1 unit 11m 34s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 19s Patch does not generate ASF License warnings. 66m 51s Subsystem Report/Notes Docker Image:yetus/hadoop:7b1c37a JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12801260/YARN-5009.001.patch JIRA Issue YARN-5009 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux c802b6bbd0d8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6f26b66 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_92 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/11262/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-warnings.html JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11262/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/11262/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        jianhe Jian He added a comment -

        minor nit: here the intervalMsec is using int, YARN-5008 is also using long. should we use long ?

        +    int intervalMsec = conf.getInt(
        +        YarnConfiguration.NM_RECOVERY_COMPACTION_INTERVAL_SECS,
        +        YarnConfiguration.DEFAULT_NM_RECOVERY_COMPACTION_INTERVAL_SECS)
        
        Show
        jianhe Jian He added a comment - minor nit: here the intervalMsec is using int, YARN-5008 is also using long. should we use long ? + int intervalMsec = conf.getInt( + YarnConfiguration.NM_RECOVERY_COMPACTION_INTERVAL_SECS, + YarnConfiguration.DEFAULT_NM_RECOVERY_COMPACTION_INTERVAL_SECS)
        Hide
        jlowe Jason Lowe added a comment -

        Nice catch! Yes, it should be using a long to avoid overflow in case someone wants to configure a cycle that's on the order of months. I updated the patch accordingly.

        Show
        jlowe Jason Lowe added a comment - Nice catch! Yes, it should be using a long to avoid overflow in case someone wants to configure a cycle that's on the order of months. I updated the patch accordingly.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 16s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 33s Maven dependency ordering for branch
        +1 mvninstall 6m 55s trunk passed
        +1 compile 1m 50s trunk passed with JDK v1.8.0_92
        +1 compile 2m 5s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 37s trunk passed
        +1 mvnsite 1m 28s trunk passed
        +1 mvneclipse 0m 39s trunk passed
        -1 findbugs 1m 8s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in trunk has 1 extant Findbugs warnings.
        +1 javadoc 1m 25s trunk passed with JDK v1.8.0_92
        +1 javadoc 3m 46s trunk passed with JDK v1.7.0_95
        0 mvndep 0m 10s Maven dependency ordering for patch
        +1 mvninstall 1m 14s the patch passed
        +1 compile 1m 46s the patch passed with JDK v1.8.0_92
        +1 javac 1m 46s the patch passed
        +1 compile 2m 5s the patch passed with JDK v1.7.0_95
        +1 javac 2m 5s the patch passed
        +1 checkstyle 0m 33s the patch passed
        +1 mvnsite 1m 21s the patch passed
        +1 mvneclipse 0m 32s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 findbugs 3m 47s the patch passed
        +1 javadoc 1m 19s the patch passed with JDK v1.8.0_92
        +1 javadoc 3m 45s the patch passed with JDK v1.7.0_95
        +1 unit 0m 20s hadoop-yarn-api in the patch passed with JDK v1.8.0_92.
        +1 unit 2m 5s hadoop-yarn-common in the patch passed with JDK v1.8.0_92.
        +1 unit 11m 21s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_92.
        +1 unit 0m 24s hadoop-yarn-api in the patch passed with JDK v1.7.0_95.
        +1 unit 2m 21s hadoop-yarn-common in the patch passed with JDK v1.7.0_95.
        +1 unit 11m 49s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_95.
        +1 asflicense 0m 22s Patch does not generate ASF License warnings.
        69m 24s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:cf2ee45
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12801319/YARN-5009.002.patch
        JIRA Issue YARN-5009
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux aa254343873a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 6243eab
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_92 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        findbugs https://builds.apache.org/job/PreCommit-YARN-Build/11272/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-warnings.html
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11272/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/11272/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 33s Maven dependency ordering for branch +1 mvninstall 6m 55s trunk passed +1 compile 1m 50s trunk passed with JDK v1.8.0_92 +1 compile 2m 5s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 37s trunk passed +1 mvnsite 1m 28s trunk passed +1 mvneclipse 0m 39s trunk passed -1 findbugs 1m 8s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in trunk has 1 extant Findbugs warnings. +1 javadoc 1m 25s trunk passed with JDK v1.8.0_92 +1 javadoc 3m 46s trunk passed with JDK v1.7.0_95 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 1m 14s the patch passed +1 compile 1m 46s the patch passed with JDK v1.8.0_92 +1 javac 1m 46s the patch passed +1 compile 2m 5s the patch passed with JDK v1.7.0_95 +1 javac 2m 5s the patch passed +1 checkstyle 0m 33s the patch passed +1 mvnsite 1m 21s the patch passed +1 mvneclipse 0m 32s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 3m 47s the patch passed +1 javadoc 1m 19s the patch passed with JDK v1.8.0_92 +1 javadoc 3m 45s the patch passed with JDK v1.7.0_95 +1 unit 0m 20s hadoop-yarn-api in the patch passed with JDK v1.8.0_92. +1 unit 2m 5s hadoop-yarn-common in the patch passed with JDK v1.8.0_92. +1 unit 11m 21s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_92. +1 unit 0m 24s hadoop-yarn-api in the patch passed with JDK v1.7.0_95. +1 unit 2m 21s hadoop-yarn-common in the patch passed with JDK v1.7.0_95. +1 unit 11m 49s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 22s Patch does not generate ASF License warnings. 69m 24s Subsystem Report/Notes Docker Image:yetus/hadoop:cf2ee45 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12801319/YARN-5009.002.patch JIRA Issue YARN-5009 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux aa254343873a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6243eab Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_92 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/11272/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-warnings.html JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11272/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/11272/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        jianhe Jian He added a comment -

        Committed to trunk, branch-2, branch-2.8, branch-2.7
        thanks Jason !

        Show
        jianhe Jian He added a comment - Committed to trunk, branch-2, branch-2.8, branch-2.7 thanks Jason !
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #9690 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9690/)
        YARN-5009. NMLeveldbStateStoreService database can grow substantially (jianhe: rev 4a8508501bc753858693dacdafba61d604702f71)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/recovery/NMLeveldbStateStoreService.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/recovery/TestNMLeveldbStateStoreService.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9690 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9690/ ) YARN-5009 . NMLeveldbStateStoreService database can grow substantially (jianhe: rev 4a8508501bc753858693dacdafba61d604702f71) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/recovery/NMLeveldbStateStoreService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/recovery/TestNMLeveldbStateStoreService.java
        Hide
        jlowe Jason Lowe added a comment -

        Thanks for the review and commit, Jian! Note that the commit to branch-2.7 had a bad conflict resolution in yarn-default.xml, so I reverted and recommitted to correct it.

        Show
        jlowe Jason Lowe added a comment - Thanks for the review and commit, Jian! Note that the commit to branch-2.7 had a bad conflict resolution in yarn-default.xml, so I reverted and recommitted to correct it.
        Hide
        jianhe Jian He added a comment -

        ah, thanks for the correction !

        Show
        jianhe Jian He added a comment - ah, thanks for the correction !
        Hide
        eepayne Eric Payne added a comment -

        Changing Fix Version to 2.7.3 since branch 2.7.3 has not yet been created.

        Show
        eepayne Eric Payne added a comment - Changing Fix Version to 2.7.3 since branch 2.7.3 has not yet been created.
        Hide
        ctrezzo Chris Trezzo added a comment -

        Adding 2.6.5 to the target versions with the intention of backporting this to branch-2.6. Please let me know if you think otherwise. Thanks!

        Show
        ctrezzo Chris Trezzo added a comment - Adding 2.6.5 to the target versions with the intention of backporting this to branch-2.6. Please let me know if you think otherwise. Thanks!
        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        Closing the JIRA as part of 2.7.3 release.

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - Closing the JIRA as part of 2.7.3 release.
        Hide
        jlowe Jason Lowe added a comment -

        The merge conflicts for branch-2.6 were trivial context differences. Attaching the branch-2.6 patch for reference, and I will commit this to branch-2.6 later today.

        Show
        jlowe Jason Lowe added a comment - The merge conflicts for branch-2.6 were trivial context differences. Attaching the branch-2.6 patch for reference, and I will commit this to branch-2.6 later today.
        Hide
        jlowe Jason Lowe added a comment -

        I committed this to branch-2.6.

        Show
        jlowe Jason Lowe added a comment - I committed this to branch-2.6.
        Hide
        ctrezzo Chris Trezzo added a comment -

        Thanks Jason Lowe!

        Show
        ctrezzo Chris Trezzo added a comment - Thanks Jason Lowe !

          People

          • Assignee:
            jlowe Jason Lowe
            Reporter:
            jlowe Jason Lowe
          • Votes:
            0 Vote for this issue
            Watchers:
            12 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development