Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 4.0-ALPHA
    • Fix Version/s: 4.0, 6.0
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      While looking at LUCENE-4123, i thought about this:

      I don't like how mmap creates a separate mapping for each CFS slice, to me this is way too many mmapings.

      Instead I think its slicer should map the .CFS file, and then when asked for an offset+length slice of that, it should be using .duplicate()d buffers of that single master mapping.

      then when you close the .CFS it closes that one mapping.

      this is probably too scary for 4.0, we should take our time, but I think we should do it.

      1. LUCENE-4364.patch
        42 kB
        Robert Muir
      2. LUCENE-4364.patch
        42 kB
        Robert Muir
      3. LUCENE-4364.patch
        38 kB
        Robert Muir
      4. LUCENE-4364.patch
        37 kB
        Robert Muir
      5. LUCENE-4364.patch
        30 kB
        Uwe Schindler
      6. LUCENE-4364.patch
        30 kB
        Robert Muir
      7. LUCENE-4364.patch
        30 kB
        Robert Muir
      8. LUCENE-4364.patch
        23 kB
        Robert Muir
      9. LUCENE-4364.patch
        23 kB
        Robert Muir

        Activity

        Hide
        Robert Muir added a comment -

        patch: I brought LUCENE-4124 up to speed as i needed it to solve this.

        there are some TODOs and some things that should be cleaned up, but all tests pass.

        Show
        Robert Muir added a comment - patch: I brought LUCENE-4124 up to speed as i needed it to solve this. there are some TODOs and some things that should be cleaned up, but all tests pass.
        Hide
        Robert Muir added a comment -

        updated with some forgotten safety. since all slices are clones, they have to be added to the clones map in createSlice:

                MMapSliceIndexInput input = new MMapSliceIndexInput(description, slices, offset, length, chunkSizePower, clones);
                clones.put(input, true); // the sliced input is itself a clone, only the slicer holds the "real" mappings.
                return input;
        
        Show
        Robert Muir added a comment - updated with some forgotten safety. since all slices are clones, they have to be added to the clones map in createSlice: MMapSliceIndexInput input = new MMapSliceIndexInput(description, slices, offset, length, chunkSizePower, clones); clones.put(input, true); // the sliced input is itself a clone, only the slicer holds the "real" mappings. return input;
        Hide
        Uwe Schindler added a comment -

        Hey, I did not yet fully review the patch, but with ByteBuffers we can do the EOF case very easy (the problem Mike had in his CachedDirectory). ByteBuffer internally checks limit() in all cases, so you cannot read beyond the limit. So when we create a slicer, the last buffer could be created not by a simple duplicate(), but instead by ByteBuffer.slice() [which is also a duplicate, but with other start [in our case still 0], but with a limited length (the modulo of (startOffset + length) % bufsize). As ByteBuffer is doing the checks in all cases, a duplicate buffer with a shorter length makes reading behind EOF impossible. The checks are done in all cases, by explicitely giving a limit for the last buffer we can correctly support EOF with IndexOutofBoundsException.

        Show
        Uwe Schindler added a comment - Hey, I did not yet fully review the patch, but with ByteBuffers we can do the EOF case very easy (the problem Mike had in his CachedDirectory). ByteBuffer internally checks limit() in all cases, so you cannot read beyond the limit. So when we create a slicer, the last buffer could be created not by a simple duplicate(), but instead by ByteBuffer.slice() [which is also a duplicate, but with other start [in our case still 0] , but with a limited length (the modulo of (startOffset + length) % bufsize). As ByteBuffer is doing the checks in all cases, a duplicate buffer with a shorter length makes reading behind EOF impossible. The checks are done in all cases, by explicitely giving a limit for the last buffer we can correctly support EOF with IndexOutofBoundsException.
        Hide
        Robert Muir added a comment -

        Uwe: this is how the whole patch works already. See the slice method.

        slices[slices.length-1].limit((int) (sliceEnd & chunkSizeMask));
        

        Please apply the patch to review, and you can see that there are no added checks here.

        Show
        Robert Muir added a comment - Uwe: this is how the whole patch works already. See the slice method. slices[slices.length-1].limit(( int ) (sliceEnd & chunkSizeMask)); Please apply the patch to review, and you can see that there are no added checks here.
        Hide
        Uwe Schindler added a comment -

        Sorry, i did not see that, your patch already does that!

        Show
        Uwe Schindler added a comment - Sorry, i did not see that, your patch already does that!
        Hide
        Robert Muir added a comment -

        Further refactoring, i think this is much cleaner.

        This whole slicing thing imo, is just really a specialized clone() with offset and length.

        So i pushed it into the generic ByteBufferIndexInput itself (it uses super.clone, etc as normal).

        This makes the CFS code here so dead simple:

          /** Maps the cfs once, and returns sliced clones on request */
          @Override
          public IndexInputSlicer createSlicer(String name, IOContext context) throws IOException {
            final MMapIndexInput cfs = (MMapIndexInput) openInput(name, context);
            return new IndexInputSlicer() {
              
              @Override
              public IndexInput openSlice(String sliceDescription, long offset, long length) throws IOException {
                ensureOpen();
                return cfs.slice(sliceDescription, offset, length);
              }
        
              @Override
              public IndexInput openFullSlice() throws IOException {
                ensureOpen();
                return cfs.clone();
              }
              
              @Override
              public void close() throws IOException {
                cfs.close();
              }
            };
          }
        
        Show
        Robert Muir added a comment - Further refactoring, i think this is much cleaner. This whole slicing thing imo, is just really a specialized clone() with offset and length. So i pushed it into the generic ByteBufferIndexInput itself (it uses super.clone, etc as normal). This makes the CFS code here so dead simple: /** Maps the cfs once, and returns sliced clones on request */ @Override public IndexInputSlicer createSlicer( String name, IOContext context) throws IOException { final MMapIndexInput cfs = (MMapIndexInput) openInput(name, context); return new IndexInputSlicer() { @Override public IndexInput openSlice( String sliceDescription, long offset, long length) throws IOException { ensureOpen(); return cfs.slice(sliceDescription, offset, length); } @Override public IndexInput openFullSlice() throws IOException { ensureOpen(); return cfs.clone(); } @Override public void close() throws IOException { cfs.close(); } }; }
        Hide
        Robert Muir added a comment -

        note we should add the bounds checks for offset/length to this .slice() method just as a sanity check (i had them in the original patch, sorry).

        Show
        Robert Muir added a comment - note we should add the bounds checks for offset/length to this .slice() method just as a sanity check (i had them in the original patch, sorry).
        Hide
        Robert Muir added a comment -

        just some minor cleanups and adding back those bounds checks i forgot.

        Show
        Robert Muir added a comment - just some minor cleanups and adding back those bounds checks i forgot.
        Hide
        Uwe Schindler added a comment -

        Hi, thats a great improvement. Very simple! In my opinion, we should maybe check, if we can remove the whole Slicer in all Indexinputs? Just make the slice(...) method return the current BufferedIndexInput-based one. This could be another issue, once this is in.

        About the patch:

        • With the IndexInputSlicer which has a private "mastercopy" of the originalIndexInput, the openFullSlice method is garanteed to have 0L as file pointer. We should maybe assert this (because this is important, as in contrast to MMapIndexInput.slice(), clone() returns a clone with the same filepointer as the original). So add a assert 0L == cloned.getFilePointer().
        • I think we could minimize the code duplication in clone() and slice(). Just use slice() as only implementation and let clone() look like this (not tested):
        public MMapIndexInput clone() {
          final MMapIndexInput clone = this.slice(this.description, 0L, this.length);
          try {
            clone.seek(this.getFilePointer());
          } catch (IOException ioe) {
            throw new RuntimException("Cannot happen...");
          }
          return clone;
        }
        

        By that the code is not duplicated. The only downside is that it seeks two times on clone, but that should not be a problem.

        Show
        Uwe Schindler added a comment - Hi, thats a great improvement. Very simple! In my opinion, we should maybe check, if we can remove the whole Slicer in all Indexinputs? Just make the slice(...) method return the current BufferedIndexInput-based one. This could be another issue, once this is in. About the patch: With the IndexInputSlicer which has a private "mastercopy" of the originalIndexInput, the openFullSlice method is garanteed to have 0L as file pointer. We should maybe assert this (because this is important, as in contrast to MMapIndexInput.slice(), clone() returns a clone with the same filepointer as the original). So add a assert 0L == cloned.getFilePointer(). I think we could minimize the code duplication in clone() and slice(). Just use slice() as only implementation and let clone() look like this (not tested): public MMapIndexInput clone() { final MMapIndexInput clone = this .slice( this .description, 0L, this .length); try { clone.seek( this .getFilePointer()); } catch (IOException ioe) { throw new RuntimException( "Cannot happen..." ); } return clone; } By that the code is not duplicated. The only downside is that it seeks two times on clone, but that should not be a problem.
        Hide
        Robert Muir added a comment -

        Uwe: seems like good ideas i think. I'm going to sleep so feel free to play. I just wanted to get some progress made tonight,
        but I feel better about it.

        Additionally for a directory like Mike's (with array-backed bytebuffer) or maybe a future RAMDir or whatever this would be reusable stuff I think.

        Show
        Robert Muir added a comment - Uwe: seems like good ideas i think. I'm going to sleep so feel free to play. I just wanted to get some progress made tonight, but I feel better about it. Additionally for a directory like Mike's (with array-backed bytebuffer) or maybe a future RAMDir or whatever this would be reusable stuff I think.
        Hide
        Uwe Schindler added a comment -

        I combined the slicing and clone methods to be one. When doing this it now also supports slicing clones or slicing slices (as offset is added). I commented out the IllegalStateException, but we can reenable it to disallow.
        I also made in the abstract class all methods reading from buffers final, to help Hotspot (MMapIndexInput is already final, but this makes it more reuseable).
        I also removed the class name from some Exception messages like AlreadyClosed, because the toString() method already contains type.

        Show
        Uwe Schindler added a comment - I combined the slicing and clone methods to be one. When doing this it now also supports slicing clones or slicing slices (as offset is added). I commented out the IllegalStateException, but we can reenable it to disallow. I also made in the abstract class all methods reading from buffers final, to help Hotspot (MMapIndexInput is already final, but this makes it more reuseable). I also removed the class name from some Exception messages like AlreadyClosed, because the toString() method already contains type.
        Hide
        Uwe Schindler added a comment -

        One question: Why did you change the chunkSize to be passed in ctor? For this patch its unrelated, or do I miss something. We now pass everything through ctor, but then its inconsequent to make the useUnmapHack to be a setter (unless we make it a static for all new MMapDirectories)?

        Show
        Uwe Schindler added a comment - One question: Why did you change the chunkSize to be passed in ctor? For this patch its unrelated, or do I miss something. We now pass everything through ctor, but then its inconsequent to make the useUnmapHack to be a setter (unless we make it a static for all new MMapDirectories)?
        Hide
        Robert Muir added a comment -

        it was an intermediate step. unlike unmaphack, its a little more broken for chunkSize to be a setter.

        with one of the previous patches, i noticed that if you were to change this at certain times it could be bad news. so I wanted to prevent the possibility of problems... probably not an isssue now.

        Show
        Robert Muir added a comment - it was an intermediate step. unlike unmaphack, its a little more broken for chunkSize to be a setter. with one of the previous patches, i noticed that if you were to change this at certain times it could be bad news. so I wanted to prevent the possibility of problems... probably not an isssue now.
        Hide
        Robert Muir added a comment -

        We should put the IllegalStateException back. there is no sense in pretending to support slices from clones, its totally untested and not useful.

        If its useful somehow in the future and we have good tests thats different, but currently its useless and untested.

        Show
        Robert Muir added a comment - We should put the IllegalStateException back. there is no sense in pretending to support slices from clones, its totally untested and not useful. If its useful somehow in the future and we have good tests thats different, but currently its useless and untested.
        Hide
        Uwe Schindler added a comment - - edited

        I have nothing against that, but it is not completely untested, look at the code When you clone a slice, the offset of the original must be applied by the code when calling the private slice method - and that is done now. This was necessary to unify the clone and slice method to one common code. cloning is now only slice with 0..length.

        But I agree we should not support this until we remove the IndexInputSlicer class at all and make every IndexInput sliceable (in a later issue). There is no chance that user code can call this method, as it is private to ByteBufferIndexInput.

        Show
        Uwe Schindler added a comment - - edited I have nothing against that, but it is not completely untested, look at the code When you clone a slice, the offset of the original must be applied by the code when calling the private slice method - and that is done now. This was necessary to unify the clone and slice method to one common code. cloning is now only slice with 0..length. But I agree we should not support this until we remove the IndexInputSlicer class at all and make every IndexInput sliceable (in a later issue). There is no chance that user code can call this method, as it is private to ByteBufferIndexInput.
        Hide
        Uwe Schindler added a comment -

        I forgot to mention: My patch also fixes a "small bug" in MMapIndexInput: If you closed a clone, this did not change it's state at all, the close() method of clones was completely ignored. I changed that to at least unset the buffers for this clone/slice, so later calls throw AlreadyClosed. Clones of that one (previously created) are not affected. The top-level cleanup also unsets all clones of course again, but thats fine. Theoretically, "closed" clones could be removed from the map, but that is unneeded, because they will diappear automatically (as soon as they are unreferenced).

        Show
        Uwe Schindler added a comment - I forgot to mention: My patch also fixes a "small bug" in MMapIndexInput: If you closed a clone, this did not change it's state at all, the close() method of clones was completely ignored. I changed that to at least unset the buffers for this clone/slice, so later calls throw AlreadyClosed. Clones of that one (previously created) are not affected. The top-level cleanup also unsets all clones of course again, but thats fine. Theoretically, "closed" clones could be removed from the map, but that is unneeded, because they will diappear automatically (as soon as they are unreferenced).
        Hide
        Robert Muir added a comment -

        its completely untested except with offset=0 and length=length, thats my problem (that doesnt count).

        its also totally unnecessary to support: there is absolutely, positively, zero use-case.
        ill change the patch to add it back. ill also remove the assert getFilePointer() == 0L (nothing depends on that).

        Show
        Robert Muir added a comment - its completely untested except with offset=0 and length=length, thats my problem (that doesnt count). its also totally unnecessary to support: there is absolutely, positively, zero use-case. ill change the patch to add it back. ill also remove the assert getFilePointer() == 0L (nothing depends on that).
        Hide
        Robert Muir added a comment -

        updated patch: I removed the assert 0L (its a contract of the method slice) from the openSlice method, and added back the IAE if you try to slice from a clone.

        as far as the openFullSlice, the assert is correct (for 4.x maybe), but in trunk we don't even need this method: its a back compat shim for reading older 3.x formatted .cfs files, which trunk no longer supports. this is just dead code: I removed it: we should deprecate it from 4.x

        Show
        Robert Muir added a comment - updated patch: I removed the assert 0L (its a contract of the method slice) from the openSlice method, and added back the IAE if you try to slice from a clone. as far as the openFullSlice, the assert is correct (for 4.x maybe), but in trunk we don't even need this method: its a back compat shim for reading older 3.x formatted .cfs files, which trunk no longer supports. this is just dead code: I removed it: we should deprecate it from 4.x
        Hide
        Robert Muir added a comment -

        while reviewing the CFS reading code i see the classic assert-should-be-check-put-return-value:

        assert !mapping.containsKey(id): "id=" + id + " was written multiple times in the CFS";
                mapping.put(id, fileEntry);
        

        Ill fix this and re-sync up the patch.

        Show
        Robert Muir added a comment - while reviewing the CFS reading code i see the classic assert-should-be-check-put-return-value: assert !mapping.containsKey(id): "id=" + id + " was written multiple times in the CFS"; mapping.put(id, fileEntry); Ill fix this and re-sync up the patch.
        Hide
        Robert Muir added a comment -

        re-synced to trunk after changing the assert to a hard check.

        Show
        Robert Muir added a comment - re-synced to trunk after changing the assert to a hard check.
        Hide
        Michael McCandless added a comment -

        This patch looks great! MMapDir is so simple now ... and I love how slice is a method on BBII. Very nice to nuke openFullSlice too.

        Show
        Michael McCandless added a comment - This patch looks great! MMapDir is so simple now ... and I love how slice is a method on BBII. Very nice to nuke openFullSlice too.
        Hide
        Robert Muir added a comment -

        updated patch: I added explicit CFS versions of all the mmap tests for better testing (so we dont rely upon the random tests or other tests).

        also fixed the checks in seek to properly catch negative positions. previously if you seeked a slice that had offset > 0 (the start offset into the first mapped buffer), and passed a negative pos but pos was < -offset, we wouldnt catch the negative access, instead positioning ourselves to a negative getFilePointer and pointed at a previous file's bytes.

        I feel good about this patch. I'd like to commit to trunk soon if there are no objections.

        Show
        Robert Muir added a comment - updated patch: I added explicit CFS versions of all the mmap tests for better testing (so we dont rely upon the random tests or other tests). also fixed the checks in seek to properly catch negative positions. previously if you seeked a slice that had offset > 0 (the start offset into the first mapped buffer), and passed a negative pos but pos was < -offset, we wouldnt catch the negative access, instead positioning ourselves to a negative getFilePointer and pointed at a previous file's bytes. I feel good about this patch. I'd like to commit to trunk soon if there are no objections.
        Hide
        Robert Muir added a comment -

        just a tweak adding a comment to explain the previous pos >= -offset thing.

        additionally i changed the shift value back to >> in case pos+offset overflows (this is more clear to me that we will always get exception), but the handling is right, its always seek past EOF here.

        Show
        Robert Muir added a comment - just a tweak adding a comment to explain the previous pos >= -offset thing. additionally i changed the shift value back to >> in case pos+offset overflows (this is more clear to me that we will always get exception), but the handling is right, its always seek past EOF here.
        Hide
        Uwe Schindler added a comment -

        Hi Robert, patch with new tests look great! +1 to commit and also backport.

        Show
        Uwe Schindler added a comment - Hi Robert, patch with new tests look great! +1 to commit and also backport.
        Hide
        Robert Muir added a comment -

        Hi Uwe, thanks for the vote of confidence and reviews and help. I'll commit to trunk tonight, give hudson a weekend at it, and backport monday if there are no issues.

        I don't expect any problems though, I think this is just as simple as before and we can move forwards to simplify CFS access.

        Show
        Robert Muir added a comment - Hi Uwe, thanks for the vote of confidence and reviews and help. I'll commit to trunk tonight, give hudson a weekend at it, and backport monday if there are no issues. I don't expect any problems though, I think this is just as simple as before and we can move forwards to simplify CFS access.
        Hide
        Robert Muir added a comment -

        Committed to trunk only. lets bake it over the weekend.

        Show
        Robert Muir added a comment - Committed to trunk only. lets bake it over the weekend.
        Hide
        Commit Tag Bot added a comment -

        [branch_4x commit] Robert Muir
        http://svn.apache.org/viewvc?view=revision&revision=1382800

        LUCENE-4364: MMapDirectory makes too many maps for CFS

        Show
        Commit Tag Bot added a comment - [branch_4x commit] Robert Muir http://svn.apache.org/viewvc?view=revision&revision=1382800 LUCENE-4364 : MMapDirectory makes too many maps for CFS
        Hide
        Uwe Schindler added a comment -

        Closed after release.

        Show
        Uwe Schindler added a comment - Closed after release.

          People

          • Assignee:
            Unassigned
            Reporter:
            Robert Muir
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development