Uploaded image for project: 'Ignite'
  1. Ignite
  2. IGNITE-11704

Write tombstones during rebalance to get rid of deferred delete buffer

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Reopened
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
    • Release Note:
      Introduced tombstone objects writing during rebalance to avoid possible OOM of deferred delete buffer
    • Ignite Flags:
      Release Notes Required

      Description

      Currently Ignite relies on deferred delete buffer in order to handle write-remove conflicts during rebalance. Given the limit size of the buffer, this approach is fundamentally flawed, especially in case when persistence is enabled.

      I suggest to extend the logic of data storage to be able to store key tombstones - to keep version for deleted entries. The tombstones will be stored when rebalance is in progress and should be cleaned up when rebalance is completed.

      Later this approach may be used to implement fast partition rebalance based on merkle trees (in this case, tombstones should be written on an incomplete baseline).

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                agoncharuk Alexey Goncharuk
                Reviewer:
                Alexey Scherbakov
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 0.5h
                  0.5h