[LUCENE-7409] Look into making mmapdirectory's unmap safer

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 6.2, 7.0
• Component/s: core/store
• Labels: crush performance
• Lucene Fields: New
• Assignee: Uwe Schindler
• Reporter: Robert Muir
• Created: 09/Aug/16 06:48
• Updated: 28/Aug/22 15:01
• Resolved: 18/Aug/16 13:55

Description

I have seen a few bugs around this recently: of course its a bug in application code but a JVM crash is not good.

I think we should see if we can prevent the crashes better than the current weak map, e.g. make it a safer option.

I made an ugly prototype here: https://github.com/apache/lucene-solr/compare/master...rmuir:ace?expand=1

It has a test that crashes the JVM without the patch but passes with.

Hacky patch only implements readBytes() but has no problems with the luceneutil benchmark (1M):

Report after iter 19:
                    Task    QPS base      StdDev   QPS patch      StdDev                Pct diff
                  IntNRQ      105.23     (17.6%)      100.42     (10.1%)   -4.6% ( -27% -   28%)
                 Respell      128.35     (13.2%)      125.88      (7.4%)   -1.9% ( -19% -   21%)
                  Fuzzy1      110.14     (17.2%)      108.28     (13.2%)   -1.7% ( -27% -   34%)
               LowPhrase      337.02     (13.0%)      333.72      (9.3%)   -1.0% ( -20% -   24%)
               MedPhrase      146.44     (12.9%)      145.55      (8.0%)   -0.6% ( -19% -   23%)
             MedSpanNear       96.85     (13.1%)       96.57      (7.8%)   -0.3% ( -18% -   23%)
            HighSpanNear       95.85     (13.9%)       96.33      (8.2%)    0.5% ( -18% -   26%)
              HighPhrase      146.84     (13.6%)      148.40      (8.4%)    1.1% ( -18% -   26%)
                HighTerm      295.15     (15.8%)      298.77      (9.5%)    1.2% ( -20% -   31%)
             LowSpanNear      268.80     (12.4%)      272.16      (7.9%)    1.2% ( -16% -   24%)
                Wildcard      284.09     (11.7%)      290.91      (8.9%)    2.4% ( -16% -   25%)
                 Prefix3      212.50     (15.4%)      217.76     (10.0%)    2.5% ( -19% -   32%)
               OrHighLow      358.65     (15.0%)      368.93     (10.7%)    2.9% ( -19% -   33%)
              AndHighMed      799.65     (13.2%)      834.74      (7.8%)    4.4% ( -14% -   29%)
         MedSloppyPhrase      229.36     (15.9%)      239.95      (9.8%)    4.6% ( -18% -   36%)
                  Fuzzy2       69.58     (14.6%)       72.82     (14.5%)    4.7% ( -21% -   39%)
             AndHighHigh      426.98     (12.8%)      451.77      (7.3%)    5.8% ( -12% -   29%)
                 MedTerm     1361.11     (14.5%)     1450.90      (9.2%)    6.6% ( -14% -   35%)
                PKLookup      266.61     (13.4%)      284.28      (8.4%)    6.6% ( -13% -   32%)
        HighSloppyPhrase      251.22     (16.9%)      268.32     (10.7%)    6.8% ( -17% -   41%)
               OrHighMed      235.92     (17.2%)      253.12     (12.8%)    7.3% ( -19% -   45%)
              OrHighHigh      186.79     (13.5%)      201.15      (9.7%)    7.7% ( -13% -   35%)
         LowSloppyPhrase      395.23     (15.9%)      425.93      (9.3%)    7.8% ( -15% -   39%)
              AndHighLow     1128.28     (14.9%)     1242.11      (8.2%)   10.1% ( -11% -   38%)
                 LowTerm     3024.62     (12.9%)     3367.65      (9.7%)   11.3% (  -9% -   39%)

We should do more testing. Maybe its totally the wrong tradeoff, maybe we only need handles for getters and everything inlines correctly, rather than needing a ton for every getXYZ() method...

Issue Links

• is duplicated by: LUCENE-6943 Jvm Crashes occassionaly with Lucene 4.6.1