Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.6, 4.0-ALPHA
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      in LUCENE-3601, i worked up a change where we intentionally crash() all un-fsynced files
      in tests to ensure that we are calling sync on files when we should.

      I think this would be nice to do always (and with some fixes all tests pass).

      But this is super-slow sometimes because when we corrupt the unsynced segments.gen, it causes
      SIS.read to take 500ms each time (and in checkindex for some reason we do this twice, which seems wrong).

      I can workaround this for now for tests (just do a partial crash that avoids corrupting the segments.gen),
      but I wanted to create this issue for discussion about the sleeping/non-fsyncing of segments.gen, just
      because i guess its possible someone could hit this slowness.

      1. LUCENE-3605.patch
        7 kB
        Michael McCandless

        Activity

        Hide
        Michael McCandless added a comment -

        Patch: I removed the sleep+retry loop in loading the segments.gen
        file; I don't think SIS should do any sleeping (it's risky; an app
        could in theory hit the net 0.5 second sleep here).

        I also changed SegmentInfos.finishCommit to sync this file, if it
        was written successfully, but to suppress exceptions (but then try to
        delete the file) since a reader could have it open as we try to commit
        (which is fine; the file is optional since we use dir listing to
        locate segments_N).

        Show
        Michael McCandless added a comment - Patch: I removed the sleep+retry loop in loading the segments.gen file; I don't think SIS should do any sleeping (it's risky; an app could in theory hit the net 0.5 second sleep here). I also changed SegmentInfos.finishCommit to sync this file, if it was written successfully, but to suppress exceptions (but then try to delete the file) since a reader could have it open as we try to commit (which is fine; the file is optional since we use dir listing to locate segments_N).
        Hide
        Robert Muir added a comment -

        I agree... i think we can also remove the now unused defaultGenFileRetryPauseMsec and its setter too?

        Show
        Robert Muir added a comment - I agree... i think we can also remove the now unused defaultGenFileRetryPauseMsec and its setter too?
        Hide
        Robert Muir added a comment -

        I meant getter... you already nuked the setter

        Show
        Robert Muir added a comment - I meant getter... you already nuked the setter
        Hide
        Michael McCandless added a comment -

        Woops – I'll nuke the getter!

        Show
        Michael McCandless added a comment - Woops – I'll nuke the getter!

          People

          • Assignee:
            Michael McCandless
            Reporter:
            Robert Muir
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development