Lucene - Core
  1. Lucene - Core
  2. LUCENE-6932

Seek past EOF with RAMDirectory should throw EOFException

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 6.0
    • Fix Version/s: 6.0, 5.x, 5.4.2
    • Component/s: core/index
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      In the JUnit test case from the attached file, I call "IndexInput.seek()" on a position past
      EOF. However, there is no EOFException that is thrown.

      To reproduce the error, please use the seed test: -Dtests.seed=8273A81C129D35E2

      1. issue6932.patch
        2 kB
        Stéphane Campinas
      2. LUCENE-6932.patch
        9 kB
        Michael McCandless
      3. LUCENE-6932.patch
        7 kB
        Michael McCandless
      4. LUCENE-6932.patch
        3 kB
        Michael McCandless
      5. testcase.txt
        0.9 kB
        Stéphane Campinas

        Issue Links

          Activity

          Hide
          Stéphane Campinas added a comment - - edited

          A possible solution for this bug is in the attached file issue6932.patch.
          The problem is that the "bufferPosition" variable is overwritten in the "seek" method, although it was set to BUFFER_SIZE since EOF should be thrown.

          Show
          Stéphane Campinas added a comment - - edited A possible solution for this bug is in the attached file issue6932.patch. The problem is that the "bufferPosition" variable is overwritten in the "seek" method, although it was set to BUFFER_SIZE since EOF should be thrown.
          Hide
          Michael McCandless added a comment -

          Thanks Stephane Campinas, I merged your two patches together into one, and changed the approach a bit to avoid adding a new enforceEOF member to RAMInputStream ... does it look OK?

          Show
          Michael McCandless added a comment - Thanks Stephane Campinas , I merged your two patches together into one, and changed the approach a bit to avoid adding a new enforceEOF member to RAMInputStream ... does it look OK?
          Hide
          Stéphane Campinas added a comment -

          Works for me Michael McCandless, and it is much cleaner that my proposed fix!

          Show
          Stéphane Campinas added a comment - Works for me Michael McCandless , and it is much cleaner that my proposed fix!
          Hide
          Michael McCandless added a comment -

          Thanks Stephane Campinas, I'll commit soon...

          Show
          Michael McCandless added a comment - Thanks Stephane Campinas , I'll commit soon...
          Hide
          ASF subversion and git services added a comment -

          Commit 1725111 from Michael McCandless in branch 'dev/trunk'
          [ https://svn.apache.org/r1725111 ]

          LUCENE-6932: RAMDirectory's IndexInput should always throw EOFE if you seek beyond the end of the file and then try to read

          Show
          ASF subversion and git services added a comment - Commit 1725111 from Michael McCandless in branch 'dev/trunk' [ https://svn.apache.org/r1725111 ] LUCENE-6932 : RAMDirectory's IndexInput should always throw EOFE if you seek beyond the end of the file and then try to read
          Hide
          ASF subversion and git services added a comment -

          Commit 1725112 from Michael McCandless in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1725112 ]

          LUCENE-6932: RAMDirectory's IndexInput should always throw EOFE if you seek beyond the end of the file and then try to read

          Show
          ASF subversion and git services added a comment - Commit 1725112 from Michael McCandless in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1725112 ] LUCENE-6932 : RAMDirectory's IndexInput should always throw EOFE if you seek beyond the end of the file and then try to read
          Hide
          Michael McCandless added a comment -
          Show
          Michael McCandless added a comment - Thanks Stephane Campinas
          Hide
          Adrien Grand added a comment -

          Apparently these changes caused this test failure: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/15594/. Here is a test case that demonstrates the issue:

            public void testSeek() throws IOException {
              try (Directory dir = newDirectory()) {
                try (IndexOutput out = dir.createOutput("a", IOContext.DEFAULT)) {
                  for (int i = 0; i < 1024; ++i) {
                    out.writeByte((byte) 0);
                  }
                }
                try (IndexInput in = dir.openInput("a", IOContext.DEFAULT)) {
                  in.seek(100);
                  assertEquals(100, in.getFilePointer());
                  in.seek(1024);
                  assertEquals(1024, in.getFilePointer());
                }
              }
            }
          

          The second assert fails with: expected <1024> but got <2048>.

          Show
          Adrien Grand added a comment - Apparently these changes caused this test failure: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/15594/ . Here is a test case that demonstrates the issue: public void testSeek() throws IOException { try (Directory dir = newDirectory()) { try (IndexOutput out = dir.createOutput( "a" , IOContext.DEFAULT)) { for ( int i = 0; i < 1024; ++i) { out.writeByte(( byte ) 0); } } try (IndexInput in = dir.openInput( "a" , IOContext.DEFAULT)) { in.seek(100); assertEquals(100, in.getFilePointer()); in.seek(1024); assertEquals(1024, in.getFilePointer()); } } } The second assert fails with: expected <1024> but got <2048>.
          Hide
          Michael McCandless added a comment -

          Argh

          Show
          Michael McCandless added a comment - Argh
          Hide
          Michael McCandless added a comment -

          OK here's a new patch, folding in Adrien Grand's new test (thank you!), and
          simplifying how RAMInputStream tracks its current buffer: I
          removed bufferStart and removed the boolean enforceEOF param
          to switchCurrentBuffer and removed lazy setting the buffer in the
          ctor. I renamed switchCurrentBuffer to setCurrentBuffer and
          simplified it so that it's no longer responsible for enforcing EOF,
          and added a new private nextBuffer, called when we read bytes,
          where we enforce EOF.

          I think net/net it's simpler, and tests pass once!

          Show
          Michael McCandless added a comment - OK here's a new patch, folding in Adrien Grand 's new test (thank you!), and simplifying how RAMInputStream tracks its current buffer: I removed bufferStart and removed the boolean enforceEOF param to switchCurrentBuffer and removed lazy setting the buffer in the ctor. I renamed switchCurrentBuffer to setCurrentBuffer and simplified it so that it's no longer responsible for enforcing EOF, and added a new private nextBuffer , called when we read bytes, where we enforce EOF. I think net/net it's simpler, and tests pass once!
          Hide
          Michael McCandless added a comment -

          New patch, with an idea from Adrien Grand that RAMInputStream.seek should throw EOFException immediately if you try to seek beyond the end of the file, rather than delaying it until the next read, which adds confusing code now.

          So I did, simplifying seek, added a test case, and fixed seek javadocs explaining this change...

          Show
          Michael McCandless added a comment - New patch, with an idea from Adrien Grand that RAMInputStream.seek should throw EOFException immediately if you try to seek beyond the end of the file, rather than delaying it until the next read, which adds confusing code now. So I did, simplifying seek , added a test case, and fixed seek javadocs explaining this change...
          Hide
          Adrien Grand added a comment -

          +1!

          Show
          Adrien Grand added a comment - +1!
          Hide
          Robert Muir added a comment -

          +1

          Show
          Robert Muir added a comment - +1
          Hide
          ASF subversion and git services added a comment -

          Commit 1726039 from Michael McCandless in branch 'dev/trunk'
          [ https://svn.apache.org/r1726039 ]

          LUCENE-6932: RAMInputStream now throws EOFException if you seek beyond the end of the file

          Show
          ASF subversion and git services added a comment - Commit 1726039 from Michael McCandless in branch 'dev/trunk' [ https://svn.apache.org/r1726039 ] LUCENE-6932 : RAMInputStream now throws EOFException if you seek beyond the end of the file
          Hide
          ASF subversion and git services added a comment -

          Commit 1726056 from Michael McCandless in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1726056 ]

          LUCENE-6932: RAMInputStream now throws EOFException if you seek beyond the end of the file

          Show
          ASF subversion and git services added a comment - Commit 1726056 from Michael McCandless in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1726056 ] LUCENE-6932 : RAMInputStream now throws EOFException if you seek beyond the end of the file
          Hide
          ASF subversion and git services added a comment -

          Commit 1726227 from Michael McCandless in branch 'dev/trunk'
          [ https://svn.apache.org/r1726227 ]

          LUCENE-6932: also fix NIOFSIndexInput to throw EOFE if you seek beyond end of file

          Show
          ASF subversion and git services added a comment - Commit 1726227 from Michael McCandless in branch 'dev/trunk' [ https://svn.apache.org/r1726227 ] LUCENE-6932 : also fix NIOFSIndexInput to throw EOFE if you seek beyond end of file
          Hide
          ASF subversion and git services added a comment -

          Commit 1726229 from Michael McCandless in branch 'dev/trunk'
          [ https://svn.apache.org/r1726229 ]

          LUCENE-6932: also fix NIOFSIndexInput to throw EOFE if you seek beyond end of file

          Show
          ASF subversion and git services added a comment - Commit 1726229 from Michael McCandless in branch 'dev/trunk' [ https://svn.apache.org/r1726229 ] LUCENE-6932 : also fix NIOFSIndexInput to throw EOFE if you seek beyond end of file
          Hide
          ASF subversion and git services added a comment -

          Commit 1726231 from Michael McCandless in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1726231 ]

          LUCENE-6932: also fix NIOFSIndexInput to throw EOFE if you seek beyond end of file

          Show
          ASF subversion and git services added a comment - Commit 1726231 from Michael McCandless in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1726231 ] LUCENE-6932 : also fix NIOFSIndexInput to throw EOFE if you seek beyond end of file
          Hide
          ASF subversion and git services added a comment -

          Commit 1726264 from Michael McCandless in branch 'dev/trunk'
          [ https://svn.apache.org/r1726264 ]

          LUCENE-6932: fix test bug that was not always using the dir impl being tested

          Show
          ASF subversion and git services added a comment - Commit 1726264 from Michael McCandless in branch 'dev/trunk' [ https://svn.apache.org/r1726264 ] LUCENE-6932 : fix test bug that was not always using the dir impl being tested
          Hide
          ASF subversion and git services added a comment -

          Commit 1726266 from Michael McCandless in branch 'dev/trunk'
          [ https://svn.apache.org/r1726266 ]

          LUCENE-6932: fix SimpleFSIndexInput to throw EOFException if you seek beyond end of file

          Show
          ASF subversion and git services added a comment - Commit 1726266 from Michael McCandless in branch 'dev/trunk' [ https://svn.apache.org/r1726266 ] LUCENE-6932 : fix SimpleFSIndexInput to throw EOFException if you seek beyond end of file
          Hide
          ASF subversion and git services added a comment -

          Commit 1726277 from Michael McCandless in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1726277 ]

          LUCENE-6932: fix test bug that was not always using the dir impl being tested; fix SimpleFSIndexInput to throw EOFException if you seek beyond end of file

          Show
          ASF subversion and git services added a comment - Commit 1726277 from Michael McCandless in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1726277 ] LUCENE-6932 : fix test bug that was not always using the dir impl being tested; fix SimpleFSIndexInput to throw EOFException if you seek beyond end of file
          Hide
          ASF subversion and git services added a comment -

          Commit 1726289 from Michael McCandless in branch 'dev/trunk'
          [ https://svn.apache.org/r1726289 ]

          LUCENE-6932: also fix RAFIndexInput to throw EOFE if you seek beyond end of file

          Show
          ASF subversion and git services added a comment - Commit 1726289 from Michael McCandless in branch 'dev/trunk' [ https://svn.apache.org/r1726289 ] LUCENE-6932 : also fix RAFIndexInput to throw EOFE if you seek beyond end of file
          Hide
          ASF subversion and git services added a comment -

          Commit 1726290 from Michael McCandless in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1726290 ]

          LUCENE-6932: also fix RAFIndexInput to throw EOFE if you seek beyond end of file

          Show
          ASF subversion and git services added a comment - Commit 1726290 from Michael McCandless in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1726290 ] LUCENE-6932 : also fix RAFIndexInput to throw EOFE if you seek beyond end of file
          Hide
          Michael McCandless added a comment -

          Reopen to backport to 5.4.2.

          Show
          Michael McCandless added a comment - Reopen to backport to 5.4.2.
          Hide
          ASF subversion and git services added a comment -

          Commit ad2c18cd72751cd80c11b4916980fb510eaf8f9f in lucene-solr's branch refs/heads/branch_5_4 from Michael McCandless
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=ad2c18c ]

          LUCENE-6932: RAMDirectory's IndexInput should always throw EOFE if you seek beyond the end of the file and then try to read

          git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/branches/branch_5x@1725112 13f79535-47bb-0310-9956-ffa450edef68

          Conflicts:
          lucene/CHANGES.txt

          Show
          ASF subversion and git services added a comment - Commit ad2c18cd72751cd80c11b4916980fb510eaf8f9f in lucene-solr's branch refs/heads/branch_5_4 from Michael McCandless [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=ad2c18c ] LUCENE-6932 : RAMDirectory's IndexInput should always throw EOFE if you seek beyond the end of the file and then try to read git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/branches/branch_5x@1725112 13f79535-47bb-0310-9956-ffa450edef68 Conflicts: lucene/CHANGES.txt
          Hide
          ASF subversion and git services added a comment -

          Commit 9041c1cfe3a7162b77ba2aeb8ba58985ec167528 in lucene-solr's branch refs/heads/branch_5_4 from Michael McCandless
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=9041c1c ]

          LUCENE-6932: RAMInputStream now throws EOFException if you seek beyond the end of the file

          git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/branches/branch_5x@1726056 13f79535-47bb-0310-9956-ffa450edef68

          Conflicts:
          lucene/CHANGES.txt

          Show
          ASF subversion and git services added a comment - Commit 9041c1cfe3a7162b77ba2aeb8ba58985ec167528 in lucene-solr's branch refs/heads/branch_5_4 from Michael McCandless [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=9041c1c ] LUCENE-6932 : RAMInputStream now throws EOFException if you seek beyond the end of the file git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/branches/branch_5x@1726056 13f79535-47bb-0310-9956-ffa450edef68 Conflicts: lucene/CHANGES.txt
          Hide
          ASF subversion and git services added a comment -

          Commit b4fa82b0772718d84db3d177f8ce7450be3c51ac in lucene-solr's branch refs/heads/branch_5_4 from Michael McCandless
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=b4fa82b ]

          LUCENE-6932: also fix NIOFSIndexInput to throw EOFE if you seek beyond end of file

          git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/branches/branch_5x@1726231 13f79535-47bb-0310-9956-ffa450edef68

          Show
          ASF subversion and git services added a comment - Commit b4fa82b0772718d84db3d177f8ce7450be3c51ac in lucene-solr's branch refs/heads/branch_5_4 from Michael McCandless [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=b4fa82b ] LUCENE-6932 : also fix NIOFSIndexInput to throw EOFE if you seek beyond end of file git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/branches/branch_5x@1726231 13f79535-47bb-0310-9956-ffa450edef68
          Hide
          ASF subversion and git services added a comment -

          Commit 041cd9483ec082bc3848cd400c62d50092fc5016 in lucene-solr's branch refs/heads/branch_5_4 from Michael McCandless
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=041cd94 ]

          LUCENE-6932: fix test bug that was not always using the dir impl being tested; fix SimpleFSIndexInput to throw EOFException if you seek beyond end of file

          git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/branches/branch_5x@1726277 13f79535-47bb-0310-9956-ffa450edef68

          Show
          ASF subversion and git services added a comment - Commit 041cd9483ec082bc3848cd400c62d50092fc5016 in lucene-solr's branch refs/heads/branch_5_4 from Michael McCandless [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=041cd94 ] LUCENE-6932 : fix test bug that was not always using the dir impl being tested; fix SimpleFSIndexInput to throw EOFException if you seek beyond end of file git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/branches/branch_5x@1726277 13f79535-47bb-0310-9956-ffa450edef68
          Hide
          ASF subversion and git services added a comment -

          Commit 2512ab6c1f3089cb8fe534532f0676c3358a5cd4 in lucene-solr's branch refs/heads/branch_5_4 from Michael McCandless
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=2512ab6 ]

          LUCENE-6932: also fix RAFIndexInput to throw EOFE if you seek beyond end of file

          git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/branches/branch_5x@1726290 13f79535-47bb-0310-9956-ffa450edef68

          Show
          ASF subversion and git services added a comment - Commit 2512ab6c1f3089cb8fe534532f0676c3358a5cd4 in lucene-solr's branch refs/heads/branch_5_4 from Michael McCandless [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=2512ab6 ] LUCENE-6932 : also fix RAFIndexInput to throw EOFE if you seek beyond end of file git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/branches/branch_5x@1726290 13f79535-47bb-0310-9956-ffa450edef68
          Hide
          ASF subversion and git services added a comment -

          Commit 3100f1b187ffaeee35dfbad1d26b5c44e5e4c1f7 in lucene-solr's branch refs/heads/branch_5_4 from Michael McCandless
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=3100f1b ]

          LUCENE-6932: improve exception messages; rename length parameter to sliceLength, and return it as the length, for clarity

          Show
          ASF subversion and git services added a comment - Commit 3100f1b187ffaeee35dfbad1d26b5c44e5e4c1f7 in lucene-solr's branch refs/heads/branch_5_4 from Michael McCandless [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=3100f1b ] LUCENE-6932 : improve exception messages; rename length parameter to sliceLength, and return it as the length, for clarity

            People

            • Assignee:
              Unassigned
              Reporter:
              Stéphane Campinas
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development