[LUCENE-5580] Always verify stored fields' checksum on merge

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.8
• Lucene Fields: New
• Assignee: Adrien Grand
• Reporter: Adrien Grand
• Created: 08/Apr/14 09:16
• Updated: 28/Aug/22 14:04
• Resolved: 09/Apr/14 09:11

Description

I have seen a couple of index corruptions over the last months, and most of them happened on stored fields. The explanation might just be that since stored fields are usually most of the index size, they are just more likely to be corrupted due to a hardware/operating-system failure, but it might be as well a sneaky bug on our side.

Lucene recently added checksums to index files, and you can enable integrity verification upon merge, but this comes with a cost since you need to read all index files twice instead of once. If you are merging a very large segment and your merges are I/O-bound, this might be noticeable.

I would like to implement integrity checks for stored fields on merges on the fly, so that the stored fields files need to be read only once.

Issue Links

• depends upon: LUCENE-5578 Stored fields might accumulate checksums on merges