Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-7113

OfflineSorter and BKD should verify checksums in their temp files

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 6.0, 6.1, 7.0
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      I am trying to index all 3.2 B points from the latest OpenStreetMaps export.

      My SSDs were not up to this, so I added a spinning magnets disk to beast2.

      But then I was hitting scary bug-like exceptions (ArrayIndexOutOfBoundsException) when indexing the first 2B points, and I finally checked dmesg and saw that my hard drive is dying.

      I think it's important that our temp file usages also validate checksums (like we do for all our index files, either at reader open or at merge or CheckIndex), so we can hopefully easily differentiate a bit-flipping IO system from a possible Lucene bug, in the future.

        Attachments

        1. LUCENE-7113.patch
          53 kB
          Michael McCandless
        2. LUCENE-7113.patch
          53 kB
          Michael McCandless

          Activity

            People

            • Assignee:
              mikemccand Michael McCandless
              Reporter:
              mikemccand Michael McCandless
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: