Uploaded image for project: 'Ignite'
  1. Ignite
  2. IGNITE-11252

Docs: Index corruption recovery procedure

    XMLWordPrintableJSON

    Details

    • Type: Task
    • Status: Open
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: 2.7
    • Fix Version/s: 2.8
    • Component/s: documentation
    • Labels:
      None
    • Ignite Flags:
      Docs Required

      Description

      We need to document a recovery procedure if an index corruption happens. Refer to this thread for details and examples of the exception dumped to the logs if the issue occurs:
      http://apache-ignite-developers.2346864.n4.nabble.com/Ignite-index-corruption-issue-gt-unrecoverable-cluster-td39869.html

      1. Recovering from an index corruption
        1. Applicable if
          It is known that an index of a cache is corrupted, but the main data (partition files and WAL) is fine. Show code snippets of possible examples. Find via the references shared in the dev list discussion.
        1. Steps to recover
          1. Stop the node
          2. Delete index.bin of the affected caches (path is db/<consistent_id>/cache-<cache_name>/index.bin)
          3. Start the node
      • Note: At this point the node is active in the cluster but don’t have indexes.
        It means that it serves SQL queries but their performance can be low.
        Avoid running SQL queries on large tables at this point
        4. Wait for message “Finished indexes rebuilding for cache <cache_name>” in the Ignite log
      1. Recovering from a persistent storage corruption
        1. Applicable if
          A part of the persistent storage (partition files, checkpoint markers or WAL) was corrupted
          and there is no other way to recover it, but there are healthy copies of all data on other nodes.
        1. Steps to recover
          1. Stop the node
          2. Delete all persistence files of the node (best to clear Ignite working directory, storage directory, WAL and WAL archive directories)
          3. Make sure consistentId is explicitly set in the configuration of the node
      • If it isn’t, lookup the generated consistentId using control.sh and set it explicitly in the config or via IGNITE_CONSISTENT_ID (2.8+ only)
        4. Start the node
        5. Wait for messages <Finished rebalancing cache> for all caches

        Attachments

          Activity

            People

            • Assignee:
              pgarg Prachi Garg
              Reporter:
              dmagda Denis A. Magda
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: