Details

    • Type: New Feature New Feature
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 4.0-ALPHA
    • Fix Version/s: None
    • Component/s: core/search
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      In order to offer user's near realtime search, without incurring
      an indexing performance penalty, we can implement search on
      IndexWriter's RAM buffer. This is the buffer that is filled in
      RAM as documents are indexed. Currently the RAM buffer is
      flushed to the underlying directory (usually disk) before being
      made searchable.

      Todays Lucene based NRT systems must incur the cost of merging
      segments, which can slow indexing.

      Michael Busch has good suggestions regarding how to handle deletes using max doc ids.
      https://issues.apache.org/jira/browse/LUCENE-2293?focusedCommentId=12841923&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12841923

      The area that isn't fully fleshed out is the terms dictionary,
      which needs to be sorted prior to queries executing. Currently
      IW implements a specialized hash table. Michael B has a
      suggestion here:
      https://issues.apache.org/jira/browse/LUCENE-2293?focusedCommentId=12841915&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12841915

      1. LUCENE-2312.patch
        149 kB
        Jason Rutherglen
      2. LUCENE-2312.patch
        102 kB
        Jason Rutherglen
      3. LUCENE-2312.patch
        71 kB
        Jason Rutherglen
      4. LUCENE-2312-FC.patch
        4 kB
        Jason Rutherglen

        Issue Links

          Activity

          Jason Rutherglen created issue -
          Jason Rutherglen made changes -
          Field Original Value New Value
          Link This issue is blocked by LUCENE-2293 [ LUCENE-2293 ]
          Michael Busch made changes -
          Assignee Michael Busch [ michaelbusch ]
          Michael Busch made changes -
          Fix Version/s 3.1 [ 12314025 ]
          Fix Version/s 3.0.2 [ 12314798 ]
          Michael Busch made changes -
          Link This issue is blocked by LUCENE-2324 [ LUCENE-2324 ]
          Jason Rutherglen made changes -
          Comment [ I'm hitting a weird error where after executing a ram buf term docs iteration, adding some docs, then closing the the DWs and the writer, there's an exception which indicates some unknown (to me) state was modified because of the term docs iteration. Or maybe it's obvious? :)

          {code}
          org.apache.lucene.index.CorruptIndexException: docs out of order (-2147483648 <= 2147483647 )
          at org.apache.lucene.index.FormatPostingsDocsWriter.addDoc(FormatPostingsDocsWriter.java:76)
          at org.apache.lucene.index.FreqProxTermsWriter.appendPostings(FreqProxTermsWriter.java:209)
          at org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:127)
          at org.apache.lucene.index.TermsHash.flush(TermsHash.java:145)
          at org.apache.lucene.index.DocInverter.flush(DocInverter.java:72)
          at org.apache.lucene.index.DocFieldProcessor.flush(DocFieldProcessor.java:64)
          at org.apache.lucene.index.DocumentsWriter.flush(DocumentsWriter.java:1185)
          at org.apache.lucene.index.IndexWriter.doFlushInternal(IndexWriter.java:3824)
          at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3733)
          at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3712)
          at org.apache.lucene.index.IndexWriter.closeInternal(IndexWriter.java:1807)
          {code} ]
          Michael Busch made changes -
          Link This issue is blocked by LUCENE-2346 [ LUCENE-2346 ]
          Michael Busch made changes -
          Link This issue is blocked by LUCENE-2329 [ LUCENE-2329 ]
          Michael Busch made changes -
          Fix Version/s Realtime Branch [ 12315213 ]
          Fix Version/s 4.0 [ 12314025 ]
          Jason Rutherglen made changes -
          Link This issue is related to LUCENE-2567 [ LUCENE-2567 ]
          Jason Rutherglen made changes -
          Link This issue is related to LUCENE-2558 [ LUCENE-2558 ]
          Jason Rutherglen made changes -
          Link This issue is blocked by LUCENE-2575 [ LUCENE-2575 ]
          Jason Rutherglen made changes -
          Affects Version/s Realtime Branch [ 12315213 ]
          Affects Version/s 3.0.1 [ 12314401 ]
          Jason Rutherglen made changes -
          Link This issue is blocked by LUCENE-2346 [ LUCENE-2346 ]
          Jason Rutherglen made changes -
          Link This issue relates to LUCENE-2346 [ LUCENE-2346 ]
          Jason Rutherglen made changes -
          Link This issue is related to LUCENE-2558 [ LUCENE-2558 ]
          Jason Rutherglen made changes -
          Link This issue is blocked by LUCENE-2558 [ LUCENE-2558 ]
          Jason Rutherglen made changes -
          Link This issue is related to LUCENE-2567 [ LUCENE-2567 ]
          Jason Rutherglen made changes -
          Link This issue is blocked by LUCENE-2567 [ LUCENE-2567 ]
          Jason Rutherglen made changes -
          Attachment LUCENE-2312.patch [ 12457086 ]
          Jason Rutherglen made changes -
          Link This issue is blocked by LUCENE-2567 [ LUCENE-2567 ]
          Jason Rutherglen made changes -
          Link This issue is blocked by LUCENE-2324 [ LUCENE-2324 ]
          Jason Rutherglen made changes -
          Attachment LUCENE-2312-FC.patch [ 12466663 ]
          Jason Rutherglen made changes -
          Link This issue is blocked by LUCENE-2324 [ LUCENE-2324 ]
          Mark Thomas made changes -
          Workflow jira [ 12501692 ] Default workflow, editable Closed status [ 12564147 ]
          Mark Thomas made changes -
          Workflow Default workflow, editable Closed status [ 12564147 ] jira [ 12583907 ]
          Jason Rutherglen made changes -
          Link This issue is part of HBASE-3529 [ HBASE-3529 ]
          Jason Rutherglen made changes -
          Link This issue is blocked by LUCENE-3199 [ LUCENE-3199 ]
          Jason Rutherglen made changes -
          Link This issue is part of HBASE-3529 [ HBASE-3529 ]
          Jason Rutherglen made changes -
          Link This issue is blocked by LUCENE-3245 [ LUCENE-3245 ]
          Jason Rutherglen made changes -
          Link This issue is related to CASSANDRA-2915 [ CASSANDRA-2915 ]
          Jason Rutherglen made changes -
          Link This issue is blocked by LUCENE-3399 [ LUCENE-3399 ]
          Jason Rutherglen made changes -
          Attachment LUCENE-2312.patch [ 12491564 ]
          Jason Rutherglen made changes -
          Link This issue is blocked by LUCENE-3199 [ LUCENE-3199 ]
          Jason Rutherglen made changes -
          Link This issue is blocked by LUCENE-2329 [ LUCENE-2329 ]
          Jason Rutherglen made changes -
          Link This issue is blocked by LUCENE-3245 [ LUCENE-3245 ]
          Jason Rutherglen made changes -
          Link This issue is blocked by LUCENE-2558 [ LUCENE-2558 ]
          Jason Rutherglen made changes -
          Link This issue is blocked by LUCENE-2293 [ LUCENE-2293 ]
          Jason Rutherglen made changes -
          Attachment LUCENE-2312.patch [ 12492266 ]
          Jason Rutherglen made changes -
          Affects Version/s 4.0 [ 12314025 ]
          Affects Version/s Realtime Branch [ 12315213 ]
          Fix Version/s Realtime Branch [ 12315213 ]
          Jason Rutherglen made changes -
          Link This issue incorporates LUCENE-3199 [ LUCENE-3199 ]
          Jason Rutherglen made changes -
          Link This issue is blocked by LUCENE-2575 [ LUCENE-2575 ]
          Jason Rutherglen made changes -
          Link This issue incorporates LUCENE-2575 [ LUCENE-2575 ]
          Jason Rutherglen made changes -
          Link This issue is blocked by LUCENE-3399 [ LUCENE-3399 ]
          Jason Rutherglen made changes -
          Link This issue incorporates LUCENE-3399 [ LUCENE-3399 ]

            People

            • Assignee:
              Michael Busch
              Reporter:
              Jason Rutherglen
            • Votes:
              1 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

              • Created:
                Updated:

                Development