Uploaded image for project: 'Jackrabbit Oak'
  1. Jackrabbit Oak
  2. OAK-4577 BlobGC performance improvements
  3. OAK-4200

[BlobGC] Improve collection times of blobs available

    XMLWordPrintableJSON

Details

    • Technical task
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 1.5.6, 1.6.0
    • None
    • None

    Description

      The blob collection phase (Identifying all the blobs available in the data store) is quite an expensive part of the whole GC process, taking up a few hours sometimes on large repositories, due to iteration of the sub-folders in the data store.

      In an offline discussion with tmueller and chetanm, the idea came up that this phase can be faster if

      • Blobs ids are tracked when the blobs are added for e.g. in a simple file in the datastore per cluster node.
      • GC then consolidates this file from all the cluster nodes and uses it to get the candidates for GC.
      • This variant of the MarkSweepGC can be triggered more frequently. It would be ok to miss blob id additions to this file during a crash etc., as these blobs can be cleaned up in the regular MarkSweepGC cycles triggered occasionally.

      We also may be able to track other metadata along with the blob ids like paths, timestamps etc. for auditing/analytics, in-conjunction with OAK-3140.

      Attachments

        Issue Links

          Blocked

          New Feature - A new feature of the product, which has yet to be developed. OAK-4492 FileDataStore with configurable path based structure

          • Major - Major loss of function.
          • Open
          is related to

          Bug - A problem which impairs or prevents the functions of the product. OAK-4961 Default repository.home in DocumentNodeStoreService hides framework property

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. OAK-5461 [BlobGC] BlobIdTracker remove() should merge generations

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. OAK-5231 Proper resource cleanup in BlobTrackerTest

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          New Feature - A new feature of the product, which has yet to be developed. OAK-3140 DataStore / BlobStore: add a method to pass a "type" when writing

          • Major - Major loss of function.
          • Open

          Documentation - Documentation or Website OAK-5014 Minor description change for OSGi blobTrackSnapshotIntervalInSecs property

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. OAK-4429 [oak-blob-cloud] S3Backend#getAllIdentifiers should not store all elements in memory

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. OAK-4430 DataStoreBlobStore#getAllChunkIds fetches DataRecord when not needed

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. OAK-2808 Active deletion of 'deleted' Lucene index files from DataStore without relying on full scale Blob GC

          • Major - Major loss of function.
          • Closed

          Activity

            People

              amitjain Amit Jain
              amitjain Amit Jain
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: