Uploaded image for project: 'Ignite'
  1. Ignite
  2. IGNITE-13143

Persistent store defragmentation

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: persistence
    • Labels:
    • Ignite Flags:
      Docs Required, Release Notes Required

      Description

      Persistent store enables users to store data of their caches in a durable fashion on disk still benefiting from in-memory nature of Apache Ignite. Data of caches is stored in files created for every primary or backup partition assigned to that node and in an additional file for all user indexes.

      Files in filesystem are allocated lazily (only if some data is actually stored to particular partition) and grow automatically when more data is added to the cache. But the problem is that files cannot shrink even if all data is removed.

      This umbrella ticket covers all other tasks needed to implement simple yet effective approach to defragmentation. Detailed discussion could be found in IEP-47 and in corresponding dev-list discussion but core ideas are as follows:

      1. Defragmentation is performed in a special maintenance mode when node starts, provides access to some APIs like metrics or JMX management but doesn't join the cluster.
      2. It is performed by copying all data from all partitions on node to new files with automatic compaction. After successful copy old partition files are deleted.
      3. Metrics on progress of the operation are provided to the user.
      4. Operation is fault-tolerant and in case of node failure proceeds after node restart.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              sergey-chugunov Sergey Chugunov
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:

                Time Tracking

                Estimated:
                Original Estimate - 48h Original Estimate - 48h
                48h
                Remaining:
                Time Spent - 33.5h Remaining Estimate - 39h 50m
                39h 50m
                Logged:
                Time Spent - 33.5h Remaining Estimate - 39h 50m
                33.5h