Lucene - Core
  1. Lucene - Core
  2. LUCENE-2975

hotspot bug in readvint gives wrong results

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 3.1
    • Fix Version/s: 3.1, 3.2, 4.0-ALPHA
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New, Patch Available

      Description

      When testing the 3.1-RC1 made by Yonik on the PANGAEA (www.pangaea.de) productive system I figured out that suddenly on a large segment (about 5 GiB) some stored fiels suddenly produce a strange deflate decompression problem (CompressionTools) although the stored fields are no longer pre-3.0 compressed. It seems that the header of the stored field is read incorrectly at the buffer boundary in MultiMMapDir and then FieldsReader just incorrectly detects a deflate-compressed field (CompressionTools).

      The error occurs reproducible on CheckIndex with MMapDirectory, but not with NIODir or SimpleDir. The FDT file of that segment is 2.6 GiB, on Solaris the chunk size is Integer.MAX_VALUE, so we have 2 MultiMMap IndexInputs.

      Robert and me have the index ready as a tar file, we will do tests on our local machines and hopefully solve the bug, maybe introduced by Robert's recent changes to MMap.

      1. LUCENE-2975.patch
        2 kB
        Uwe Schindler
      2. LUCENE-2975.patch
        2 kB
        Uwe Schindler
      3. LUCENE-2975.patch
        2 kB
        Uwe Schindler
      4. LUCENE-2975.patch
        7 kB
        Uwe Schindler
      5. perf.png
        30 kB
        Michael McCandless
      6. LUCENE-2975.patch
        8 kB
        Uwe Schindler

        Issue Links

          Activity

          Uwe Schindler created issue -
          Hoss Man made changes -
          Field Original Value New Value
          Fix Version/s 3.1 [ 12314822 ]
          Uwe Schindler made changes -
          Attachment LUCENE-2975.patch [ 12474051 ]
          Uwe Schindler made changes -
          Attachment LUCENE-2975.patch [ 12474053 ]
          Robert Muir made changes -
          Summary MMapDirectory on chunk size boundaries broken hotspot bug in readvint gives wrong results
          Uwe Schindler made changes -
          Attachment LUCENE-2975.patch [ 12474066 ]
          Uwe Schindler made changes -
          Attachment LUCENE-2975.patch [ 12474067 ]
          Uwe Schindler made changes -
          Attachment LUCENE-2975.patch [ 12474066 ]
          Uwe Schindler made changes -
          Attachment LUCENE-2975.patch [ 12474069 ]
          Michael McCandless made changes -
          Attachment perf.png [ 12474071 ]
          Uwe Schindler made changes -
          Attachment LUCENE-2975.patch [ 12474073 ]
          Uwe Schindler made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Lucene Fields [New] [New, Patch Available]
          Assignee Uwe Schindler [ thetaphi ]
          Fix Version/s 3.2 [ 12316070 ]
          Fix Version/s 4.0 [ 12314025 ]
          Resolution Fixed [ 1 ]
          Grant Ingersoll made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Uwe Schindler made changes -
          Link This issue is related to LUCENE-3335 [ LUCENE-3335 ]
          Uwe Schindler made changes -
          Link This issue is related to LUCENE-3346 [ LUCENE-3346 ]

            People

            • Assignee:
              Uwe Schindler
              Reporter:
              Uwe Schindler
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development