Lucene - Core
  1. Lucene - Core
  2. LUCENE-5580

Always verify stored fields' checksum on merge

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.8
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      I have seen a couple of index corruptions over the last months, and most of them happened on stored fields. The explanation might just be that since stored fields are usually most of the index size, they are just more likely to be corrupted due to a hardware/operating-system failure, but it might be as well a sneaky bug on our side.

      Lucene recently added checksums to index files, and you can enable integrity verification upon merge, but this comes with a cost since you need to read all index files twice instead of once. If you are merging a very large segment and your merges are I/O-bound, this might be noticeable.

      I would like to implement integrity checks for stored fields on merges on the fly, so that the stored fields files need to be read only once.

        Issue Links

          Activity

          Adrien Grand created issue -
          Adrien Grand made changes -
          Field Original Value New Value
          Attachment LUCENE-5580.patch [ 12639164 ]
          Adrien Grand made changes -
          Link This issue depends upon LUCENE-5578 [ LUCENE-5578 ]
          Adrien Grand made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Uwe Schindler made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Adrien Grand
              Reporter:
              Adrien Grand
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development