Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-825

NullPointerException from SegmentInfos.FindSegmentsFile.run() if FSDirectory.list() returns NULL

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.1
    • Fix Version/s: 2.2
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New, Patch Available

      Description

      Found this bug while running unit tests to verify an upgrade of our system from 1.4.3 to 2.1.0. This bug did not occur during 1.4.3, it is new to 2.x (I'm pretty sure it's 2.1-only)

      If the index directory gets deleted out from under Lucene after the FSDirectory has been created, then attempts to open an IndexWriter or IndexReader will result in an NPE. Lucene should be throwing an IOException in this case.

      Repro:
      1) Create an FSDirectory pointing somewhere in the filesystem (e.g. /foo/index/1)
      2) rm -rf the parent dir (rm -rf /foo/index)
      3) Try to open an IndexReader

      Result: NullPointerException on line "for(int i=0;i<files.length;i++) { " – 'files' is NULL.

      Expect: IOException

      ....

      This is happening because of a missing NULL check in SegmentInfos$FindSegmentsFile.run():

      if (0 == method) {
      if (directory != null)

      { files = directory.list(); }

      else

      { files = fileDirectory.list(); }

      gen = getCurrentSegmentGeneration(files);

      if (gen == -1) {
      String s = "";
      for(int i=0;i<files.length;i++)

      { s += " " + files[i]; }

      throw new FileNotFoundException("no segments* file found: files:" + s);
      }
      }

      The FSDirectory constructor will make sure the index dir exists, but if it is for some reason deleted out from underneath Lucene after the FSDirectory is instantiated, then java.io.File.list() will return NULL. Probably better to fix FSDirectory.list() to just check for null and return a 0-length array:

      (in org/apache/lucene/store/FSDirectory.java)
      314c314,317
      < return directory.list(IndexFileNameFilter.getFilter());

      > String[] toRet = directory.list(IndexFileNameFilter.getFilter());
      > if (toRet == null)
      > return new String[]{};
      > return toRet;

      1. LUCENE-825.patch
        5 kB
        Michael McCandless

        Activity

        Hide
        mikemccand Michael McCandless added a comment -

        Whoa, thanks for catching this!

        I would rather leave FSDirectory.list() returning null when the
        directory does not exist because that would be an API change that
        makes me a little nervous.

        I'll instead fix it in SegmentInfos and add a unit test showing the
        bug.

        Index: src/java/org/apache/lucene/index/SegmentInfos.java
        ===================================================================
        — src/java/org/apache/lucene/index/SegmentInfos.java (revision 515317)
        +++ src/java/org/apache/lucene/index/SegmentInfos.java (working copy)
        @@ -481,6 +481,10 @@
        files = fileDirectory.list();
        }

        + if (files == null)

        { + throw new FileNotFoundException("no segments* file found in directory " + directory + ": list() returned null"); + }

        +
        gen = getCurrentSegmentGeneration(files);

        if (gen == -1) {

        Show
        mikemccand Michael McCandless added a comment - Whoa, thanks for catching this! I would rather leave FSDirectory.list() returning null when the directory does not exist because that would be an API change that makes me a little nervous. I'll instead fix it in SegmentInfos and add a unit test showing the bug. Index: src/java/org/apache/lucene/index/SegmentInfos.java =================================================================== — src/java/org/apache/lucene/index/SegmentInfos.java (revision 515317) +++ src/java/org/apache/lucene/index/SegmentInfos.java (working copy) @@ -481,6 +481,10 @@ files = fileDirectory.list(); } + if (files == null) { + throw new FileNotFoundException("no segments* file found in directory " + directory + ": list() returned null"); + } + gen = getCurrentSegmentGeneration(files); if (gen == -1) {
        Hide
        mikemccand Michael McCandless added a comment -

        Thanks Tim. Keep the patches coming!

        Show
        mikemccand Michael McCandless added a comment - Thanks Tim. Keep the patches coming!
        Hide
        timbrennan Tim Brennan added a comment -

        The reason I suggested changing FSDirectory is that a quick look through the callers of Directory.list() show that pretty much none of the callsites expect a null return from this function.

        Check out:

        1) SegmentInfos.java - boolean hasSeparateNorms():
        String[] result = dir.list();
        String pattern;
        pattern = name + ".s";
        int patternLength = pattern.length();
        for(int i = 0; i < result.length; i++){
        if(result[i].startsWith(pattern) && Character.isDigit(result[i].charAt(patternLength)))
        (accesses result.length w/o null check on result)

        2) IndexFileDeleter.java - void findDeletableFiles():
        String[] files = directory.list();

        for (int i = 0; i < files.length; i++) {
        (access to files.length w/o null check)

        3) Directory.java - static void copy(...)
        ....same thing...

        Should probably fix all those callsites as well, if we really expect Directory.list() to return null...

        Show
        timbrennan Tim Brennan added a comment - The reason I suggested changing FSDirectory is that a quick look through the callers of Directory.list() show that pretty much none of the callsites expect a null return from this function. Check out: 1) SegmentInfos.java - boolean hasSeparateNorms(): String[] result = dir.list(); String pattern; pattern = name + ".s"; int patternLength = pattern.length(); for(int i = 0; i < result.length; i++){ if(result [i] .startsWith(pattern) && Character.isDigit(result [i] .charAt(patternLength))) (accesses result.length w/o null check on result) 2) IndexFileDeleter.java - void findDeletableFiles(): String[] files = directory.list(); for (int i = 0; i < files.length; i++) { (access to files.length w/o null check) 3) Directory.java - static void copy(...) ....same thing... Should probably fix all those callsites as well, if we really expect Directory.list() to return null...
        Hide
        mikemccand Michael McCandless added a comment -

        Excellent point. I will do that.

        Show
        mikemccand Michael McCandless added a comment - Excellent point. I will do that.
        Hide
        mikemccand Michael McCandless added a comment -

        Attached patch to check all places internal to Lucene where we call Directory.list(). I also updated the javadoc of that method to state that it may return null.

        Show
        mikemccand Michael McCandless added a comment - Attached patch to check all places internal to Lucene where we call Directory.list(). I also updated the javadoc of that method to state that it may return null.

          People

          • Assignee:
            mikemccand Michael McCandless
            Reporter:
            timbrennan Tim Brennan
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development