Solr
  1. Solr
  2. SOLR-455

Better handling when index runs out of disk space

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 1.2, 1.3
    • Fix Version/s: None
    • Component/s: update
    • Labels:
      None
    • Environment:

      Linux/Debian etch

      Description

      We had an index run out of disk space. Queries work fine but commits return
      <h1>500 doc counts differ for segment _18lu: fieldsReader shows 104 but segmentInfo shows 212
      org.apache.lucene.index.CorruptIndexException: doc counts differ for segment _18lu: fieldsReader shows 104 but segmentInfo shows 212
      at org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:191)
      I've made room, restarted resin, and now solr won't start. No useful messages in the startup, just a

      [21:01:49.105] Could not start SOLR. Check solr/home property
      [21:01:49.105] java.lang.NullPointerException
      [21:01:49.105] at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:100)

      Solr should warn the user and/or refuse commits when the index nears the end of disk space

        Issue Links

          Activity

          Hide
          Grant Ingersoll added a comment -

          Just to note there are some Lucene issues also being addressed in regards to full disks.

          Show
          Grant Ingersoll added a comment - Just to note there are some Lucene issues also being addressed in regards to full disks.
          Hide
          Hoss Man added a comment -

          Yeah, Lucene Java 2.3 should do a much better job of dealing with disk full situations.

          As far as warning and/or refusing to add/commit when disk is "nearly full" ... Is this even possible? does Java have any APIs for letting applications know metrics like this (ie: total disk size - space used) ???

          Show
          Hoss Man added a comment - Yeah, Lucene Java 2.3 should do a much better job of dealing with disk full situations. As far as warning and/or refusing to add/commit when disk is "nearly full" ... Is this even possible? does Java have any APIs for letting applications know metrics like this (ie: total disk size - space used) ???
          Hide
          Otis Gospodnetic added a comment -

          My guess would be that this is really out of scope for Solr and should be (is in 2.3!) done on the Lucene level.

          As for Java being able to get info about disk usage, I believe Java 7 will have that.

          Show
          Otis Gospodnetic added a comment - My guess would be that this is really out of scope for Solr and should be (is in 2.3!) done on the Lucene level. As for Java being able to get info about disk usage, I believe Java 7 will have that.
          Hide
          Hoss Man added a comment -

          preventing corruption would definitely be on the Lucene-Java side of the fence, but reporting warnings about almost full disks would definitely be a solr scope thing (assuming it's possible)

          Show
          Hoss Man added a comment - preventing corruption would definitely be on the Lucene-Java side of the fence, but reporting warnings about almost full disks would definitely be a solr scope thing (assuming it's possible)
          Show
          Shalin Shekhar Mangar added a comment - Isn't it already there in Java 6? http://java.sun.com/javase/6/docs/api/java/io/File.html#getFreeSpace( ) http://java.sun.com/javase/6/docs/api/java/io/File.html#getTotalSpace( ) http://java.sun.com/javase/6/docs/api/java/io/File.html#getUsableSpace( )
          Hide
          Jan Høydahl added a comment -

          This issue has been inactive for more than 4 years. Please close if it's no longer relevant/needed, or bring it up to date if you intend to work on it. SPRING_CLEANING_2013

          Show
          Jan Høydahl added a comment - This issue has been inactive for more than 4 years. Please close if it's no longer relevant/needed, or bring it up to date if you intend to work on it. SPRING_CLEANING_2013

            People

            • Assignee:
              Unassigned
              Reporter:
              Brian Whitman
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:

                Development