[LUCENE-7113] OfflineSorter and BKD should verify checksums in their temp files

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 6.0, 6.1, 7.0
• Lucene Fields: New
• Assignee: Michael McCandless
• Reporter: Michael McCandless
• Created: 17/Mar/16 14:47
• Updated: 28/Aug/22 14:52
• Resolved: 19/Mar/16 09:38

Description

I am trying to index all 3.2 B points from the latest OpenStreetMaps export.

My SSDs were not up to this, so I added a spinning magnets disk to beast2.

But then I was hitting scary bug-like exceptions (ArrayIndexOutOfBoundsException) when indexing the first 2B points, and I finally checked dmesg and saw that my hard drive is dying.

I think it's important that our temp file usages also validate checksums (like we do for all our index files, either at reader open or at merge or CheckIndex), so we can hopefully easily differentiate a bit-flipping IO system from a possible Lucene bug, in the future.