Similar to the RM case in YARN-5008, I have seen state stores for nodemanagers with high container churn become significantly larger than they should be due to lack of sufficient database compaction.
Patch to add periodic manual compactions every hour by default. Compaction cycle interval can be configured or disabled by setting an interval of zero.
This message was automatically generated.
minor nit: here the intervalMsec is using int, YARN-5008 is also using long. should we use long ?
+ int intervalMsec = conf.getInt(
Nice catch! Yes, it should be using a long to avoid overflow in case someone wants to configure a cycle that's on the order of months. I updated the patch accordingly.
Committed to trunk, branch-2, branch-2.8, branch-2.7
thanks Jason !
FAILURE: Integrated in Hadoop-trunk-Commit #9690 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9690/)
YARN-5009. NMLeveldbStateStoreService database can grow substantially (jianhe: rev 4a8508501bc753858693dacdafba61d604702f71)
Thanks for the review and commit, Jian! Note that the commit to branch-2.7 had a bad conflict resolution in yarn-default.xml, so I reverted and recommitted to correct it.
ah, thanks for the correction !
Changing Fix Version to 2.7.3 since branch 2.7.3 has not yet been created.
Adding 2.6.5 to the target versions with the intention of backporting this to branch-2.6. Please let me know if you think otherwise. Thanks!
Closing the JIRA as part of 2.7.3 release.
The merge conflicts for branch-2.6 were trivial context differences. Attaching the branch-2.6 patch for reference, and I will commit this to branch-2.6 later today.
I committed this to branch-2.6.
Thanks Jason Lowe!