[OAK-10739] Provide Support for Full Garbage Collection in Mongo Document Store - ASF JIRA

Attach files

Attach Screenshot

Add vote

Voters

Watch issue

Watchers

Create sub-task

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Epic
Status: In Progress
Priority: Major
Resolution: Unresolved
Affects Version/s: None
Fix Version/s: None
Component/s: documentmk
Labels:
- DetailedGC

Epic Name:
Support for Full Garbage Collection

Description

We need to provide the support to collect & remove the full garbage for DocumentNodeStore.

At the time of creating this epic garbage includes orphaned nodes, deleted properties, unmerged branch commits, and old revisions.

This list can be updated in case a new type of garbage is found.

Attachments

org.apache.jackrabbit.oak.plugins.document.BranchCommitGCTest.txt
29/May/24 12:11
4 kB
Manfred Baedke
org.apache.jackrabbit.oak.plugins.document.VersionGarbageCollectorIT.txt
29/May/24 11:42
68 kB
Manfred Baedke

Issue Links

Add Link

relates to

OAK-10845 store-document: occasional test failure: BranchCommitGCTest.unmergedMergedRemoveChild

Open

Delete this link

links to

GitHub Pull Request #1454

Delete this link

GitHub Pull Request #1459

Delete this link

GitHub Pull Request #1462

Delete this link

GitHub Pull Request #1467

Delete this link

GitHub Pull Request #1482

Delete this link

(1 links to)

Issues in epic

quick-create-issue-for-epic-label

OAK-8646	Clean up changes from orphaned branch commits	Resolved	Rishabh Daim	Actions
OAK-10193	Garbage collect deleted properties	Resolved	Rishabh Daim	Actions
OAK-10199	Skeleton of an additional, extendable "detail" garbage collector based on only "_modified"	Resolved	Rishabh Daim	Actions
OAK-10213	Add new api in DocumentStore to handle bulk findAndUpdate operations	Closed	Rishabh Daim	Actions
OAK-10370	Dry-run mode for full GC	Resolved	Rishabh Daim	Actions
OAK-10378	Add metrics for detailed GC	Resolved	Rishabh Daim	Actions
OAK-10382	oak-run support for flatfile	Closed	Stefan Egli	Actions
OAK-10535	Clean up old revisions in a document	Resolved	José Andrés Cordero Benítez	Actions
OAK-10570	oak-run support for fullgc	Closed	Stefan Egli	Actions
OAK-10583	repeat detailedGC also if provided scope not fully processed	Resolved	Stefan Egli	Actions
OAK-10586	DetailedGC hardening	Open	Unassigned	Actions
OAK-10597	embedded verification for detailedGC	Resolved	Stefan Egli	Actions
OAK-10632	Make Embedded DetailedGC Configurable for dryRun mode	Resolved	Rishabh Daim	Actions
OAK-10633	Make Embedded DetailedGC Configurable in detailedGC	Resolved	Rishabh Daim	Actions
OAK-10659	Remove orphaned nodes/documents	Closed	Stefan Egli	Actions
OAK-10676	Consider late-writes while removing deleted properties during detailedGC	Resolved	Rishabh Daim	Actions
OAK-10688	Keep only traversed state, remove all other revisions	Resolved	Stefan Egli	Actions
OAK-10689	Extend oak-run revisions command with "detail" garbage collection	Resolved	José Andrés Cordero Benítez	Actions
OAK-10710	Reset detailedGC settings after running the detailedGC cycle	Resolved	Rishabh Daim	Actions
OAK-10714	DGC : enable embedded verification for tests by default	Resolved	Stefan Egli	Actions
OAK-10715	embedded verification should use traversed nodeState	Resolved	Stefan Egli	Actions
OAK-10724	Introduce detailed gc mode that only deletes orphan nodes and deleted properties	Resolved	Stefan Egli	Actions
OAK-10726	Fix BranchCommitGCTest and make it parameterized by gcType (also for VersionGarbageCollectorIT)	Resolved	Stefan Egli	Actions
OAK-10727	log revisionDetailedGcType	Resolved	Stefan Egli	Actions
OAK-10728	embedded verification fails if id is from long path	Resolved	Stefan Egli	Actions
OAK-10734	DetailedGC must keep entries in "_revisions" for non branch commits, unless older than sweep	Resolved	Stefan Egli	Actions
OAK-10736	Collect DetailedGC Stats for DryRun mode	Resolved	Rishabh Daim	Actions
OAK-10740	Collect Orphan nodes deletion metrics	Resolved	Rishabh Daim	Actions
OAK-10741	Improve logging for detailedGC	Closed	Rishabh Daim	Actions
OAK-10742	Introduce include/exclude lists for detailedGC	Closed	Stefan Egli	Actions
OAK-10743	Split orphaned gc mode into two : with-gap, without-gap	Resolved	Stefan Egli	Actions
OAK-10744	OSGi config to control the fullGC mode	Reopened	Rishabh Daim	Actions
OAK-10745	Osgi Config for a delayFactor & batch-size while removing fullGC	Closed	Rishabh Daim	Actions
OAK-10746	Provide support to reset dry-run mode.	Reopened	Rishabh Daim	Actions
OAK-10747	Provide support to reset fullGC	Resolved	Unassigned	Actions
OAK-10748	Improve statistics to collect which type of garbage is present/deleted	Closed	José Andrés Cordero Benítez	Actions
OAK-10749	Perform a dry-run just once in a running cluster	Open	Unassigned	Actions
OAK-10750	Dynamically evaluate FT for detailedGC & embedded verification mode	Open	Unassigned	Actions
OAK-10751	Add compaction option via oak-run	Resolved	Rishabh Daim	Actions
OAK-10753	Add audit logger for DetailedGC	Resolved	Rishabh Daim	Actions
OAK-10758	Avoid or mitigate Cursor Timeout in Mongo when running detailedGC	In Progress	José Andrés Cordero Benítez	Actions
OAK-10761	gap orphans improvement : ignore greatest existing ancestors	Resolved	Stefan Egli	Actions
OAK-10763	DetailedGC is skipping deletion of deleted props for orphan nodes	Resolved	Rishabh Daim	Actions
OAK-10764	gap orphans improvement : only lookup greatest existing ancestor, then cache	Resolved	Stefan Egli	Actions
OAK-10765	Fetch only id while checking for GAP orphan during verification	Resolved	Rishabh Daim	Actions
OAK-10792	Rename DetailedGC to FullGC	Closed	Rishabh Daim	Actions
OAK-10843	Flaky fullgc tests	Closed	Unassigned	Actions
OAK-10844	speed up fullgc tests	Open	Unassigned	Actions
OAK-10853	VersionGarbageCollectorIT.testDeletedPropsAndUnmergedBCWithCollision failure	Closed	Unassigned	Actions
OAK-10869	testDeletedPropsAndUnmergedBCWithCollisionWithDryRunMode test failure	Open	Stefan Egli	Actions
OAK-10883	document mk: disable flaky test VersionGarbageCollectorIT.testBundledPropUnmergedBCGC for mode ORPHANS_EMPTYPROPS_UNMERGED_BC	Closed	Julian Reschke	Actions
OAK-10891	document mk: disable flaky test BranchCommitGCTest.testDeletedPropsAndUnmergedBCfor mode ORPHANS_EMPTYPROPS_KEEP_ONE_ALL_PROPS	Closed	José Andrés Cordero Benítez	Actions
OAK-10896	Add osgi config for removal of deleted properties and orphaned nodes	Closed	Unassigned	Actions
OAK-10912	Make include/exclude paths for fullgc configurable via OSGI Config	Closed	Rishabh Daim	Actions
OAK-10914	fullGC exclude paths should use _modified_id index	Closed	Stefan Egli	Actions
OAK-10916	Provide support for include/exclude paths for fullGC in oak-run	Closed	Rishabh Daim	Actions
OAK-10918	Embed and log an explain for the fullgc mongo query	Closed	Stefan Egli	Actions
OAK-10919	Fix DocumentNodeStoreConfigTest unit case	Closed	Rishabh Daim	Actions
OAK-10921	Fix race condition while resetting fullGC variables from oak-run	Closed	Rishabh Daim	Actions
OAK-10948	Speed up VersionGarbageCollectorIT	Open	Unassigned	Actions
OAK-10974	testDeletedPropsAndUnmergedBCWithoutCollision flakyness	Open	Unassigned	Actions
OAK-10986	MissingLastRevSeekerTest.getNonSplitDocs[RDBFixture: RDB-H2(file)] failure	Open	Unassigned	Actions
OAK-10987	VersionGCSupportTest.findModifiedDocsWhenOldestDocIsAbsent[RDBFixture: RDB-H2(file)] failure	Open	Unassigned	Actions
OAK-11002	VersionGCWithSplitTest.gcWithConcurrentSplit[MongoFixture: MongoDB] - teardown fails with NPE	Closed	Julian Reschke	Actions
OAK-11014	Write test case to prove fullGC working correctly with bundled property nodes	Closed	Rishabh Daim	Actions
OAK-11015	Create fullGC Mode for empty properties only	Closed	Rishabh Daim	Actions
OAK-11203	Add delayFactor and progress/batchSize for fullGC in Oak-Run Command	Closed	Rishabh Daim	Actions
OAK-11218	FullGC Exclude list parsing should ignore an Array of empty String	Closed	José Andrés Cordero Benítez	Actions
OAK-11231	FullGC iterates same documents on certain cases	Closed	José Andrés Cordero Benítez	Actions
OAK-11251	Improve warning in collectFullGC	Resolved	José Andrés Cordero Benítez	Actions
OAK-11261	Flaky test preventing Oak release: BranchCommitGCTest.unmergedAddThenMergedAddAndRemoveChildren	Closed	José Andrés Cordero Benítez	Actions
OAK-11263	VersionGarbageCollectorIT.testDeletedPropsAndUnmergedBCWithCollisionWithDryRunMode seems flaky	Open	Unassigned	Actions