Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-7409

Look into making mmapdirectory's unmap safer

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 6.2, 7.0
    • Component/s: core/store
    • Labels:
    • Lucene Fields:
      New

      Description

      I have seen a few bugs around this recently: of course its a bug in application code but a JVM crash is not good.

      I think we should see if we can prevent the crashes better than the current weak map, e.g. make it a safer option.

      I made an ugly prototype here: https://github.com/apache/lucene-solr/compare/master...rmuir:ace?expand=1

      It has a test that crashes the JVM without the patch but passes with.

      Hacky patch only implements readBytes() but has no problems with the luceneutil benchmark (1M):

      Report after iter 19:
                          Task    QPS base      StdDev   QPS patch      StdDev                Pct diff
                        IntNRQ      105.23     (17.6%)      100.42     (10.1%)   -4.6% ( -27% -   28%)
                       Respell      128.35     (13.2%)      125.88      (7.4%)   -1.9% ( -19% -   21%)
                        Fuzzy1      110.14     (17.2%)      108.28     (13.2%)   -1.7% ( -27% -   34%)
                     LowPhrase      337.02     (13.0%)      333.72      (9.3%)   -1.0% ( -20% -   24%)
                     MedPhrase      146.44     (12.9%)      145.55      (8.0%)   -0.6% ( -19% -   23%)
                   MedSpanNear       96.85     (13.1%)       96.57      (7.8%)   -0.3% ( -18% -   23%)
                  HighSpanNear       95.85     (13.9%)       96.33      (8.2%)    0.5% ( -18% -   26%)
                    HighPhrase      146.84     (13.6%)      148.40      (8.4%)    1.1% ( -18% -   26%)
                      HighTerm      295.15     (15.8%)      298.77      (9.5%)    1.2% ( -20% -   31%)
                   LowSpanNear      268.80     (12.4%)      272.16      (7.9%)    1.2% ( -16% -   24%)
                      Wildcard      284.09     (11.7%)      290.91      (8.9%)    2.4% ( -16% -   25%)
                       Prefix3      212.50     (15.4%)      217.76     (10.0%)    2.5% ( -19% -   32%)
                     OrHighLow      358.65     (15.0%)      368.93     (10.7%)    2.9% ( -19% -   33%)
                    AndHighMed      799.65     (13.2%)      834.74      (7.8%)    4.4% ( -14% -   29%)
               MedSloppyPhrase      229.36     (15.9%)      239.95      (9.8%)    4.6% ( -18% -   36%)
                        Fuzzy2       69.58     (14.6%)       72.82     (14.5%)    4.7% ( -21% -   39%)
                   AndHighHigh      426.98     (12.8%)      451.77      (7.3%)    5.8% ( -12% -   29%)
                       MedTerm     1361.11     (14.5%)     1450.90      (9.2%)    6.6% ( -14% -   35%)
                      PKLookup      266.61     (13.4%)      284.28      (8.4%)    6.6% ( -13% -   32%)
              HighSloppyPhrase      251.22     (16.9%)      268.32     (10.7%)    6.8% ( -17% -   41%)
                     OrHighMed      235.92     (17.2%)      253.12     (12.8%)    7.3% ( -19% -   45%)
                    OrHighHigh      186.79     (13.5%)      201.15      (9.7%)    7.7% ( -13% -   35%)
               LowSloppyPhrase      395.23     (15.9%)      425.93      (9.3%)    7.8% ( -15% -   39%)
                    AndHighLow     1128.28     (14.9%)     1242.11      (8.2%)   10.1% ( -11% -   38%)
                       LowTerm     3024.62     (12.9%)     3367.65      (9.7%)   11.3% (  -9% -   39%)
      

      We should do more testing. Maybe its totally the wrong tradeoff, maybe we only need handles for getters and everything inlines correctly, rather than needing a ton for every getXYZ() method...

        Attachments

        1. LUCENE-7409.patch
          23 kB
          Uwe Schindler
        2. StoreStore.java
          2 kB
          Dawid Weiss

          Issue Links

            Activity

              People

              • Assignee:
                thetaphi Uwe Schindler
                Reporter:
                rcmuir Robert Muir
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: