Solr
  1. Solr
  2. SOLR-846

Out Of memory doing delta import with fetch size set to -1

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.3
    • Fix Version/s: 1.4
    • Labels:
      None
    • Environment:

      Linux 2.6.18-92.1.13.el5xen, mysql 5.0

      Description

      Database has about 3 million records. Doing full-import there is no problem. However, when a large number of changes occurred 2558057, delta-import throws OutOfMemory error after 1288338 documents processed. The stack trace is below

      Exception in thread "Thread-3" java.lang.OutOfMemoryError: Java heap space
      at org.tartarus.snowball.ext.EnglishStemmer.<init>(EnglishStemmer.java:4
      9)
      at org.apache.solr.analysis.EnglishPorterFilter.<init>(EnglishPorterFilt
      erFactory.java:83)
      at org.apache.solr.analysis.EnglishPorterFilterFactory.create(EnglishPor
      terFilterFactory.java:66)
      at org.apache.solr.analysis.EnglishPorterFilterFactory.create(EnglishPor
      terFilterFactory.java:35)
      at org.apache.solr.analysis.TokenizerChain.tokenStream(TokenizerChain.ja
      va:48)
      at org.apache.solr.schema.IndexSchema$SolrIndexAnalyzer.tokenStream(Inde
      xSchema.java:348)
      at org.apache.lucene.analysis.Analyzer.reusableTokenStream(Analyzer.java
      :44)
      at org.apache.lucene.index.DocInverterPerField.processFields(DocInverter
      PerField.java:117)
      at org.apache.lucene.index.DocFieldConsumersPerField.processFields(DocFi
      eldConsumersPerField.java:36)
      at org.apache.lucene.index.DocFieldProcessorPerThread.processDocument(Do
      cFieldProcessorPerThread.java:234)
      at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWrite
      r.java:765)
      at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWrite
      r.java:748)
      at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:2
      118)
      at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:2
      095)
      at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandle
      r2.java:232)
      at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpd
      ateProcessorFactory.java:59)
      at org.apache.solr.handler.dataimport.SolrWriter.upload(SolrWriter.java:
      69)
      at org.apache.solr.handler.dataimport.DataImportHandler$1.upload(DataImp
      ortHandler.java:288)
      at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilde
      r.java:319)
      at org.apache.solr.handler.dataimport.DocBuilder.doDelta(DocBuilder.java
      :211)
      at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java
      :133)
      at org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImp
      orter.java:359)
      at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.j
      ava:388)
      at org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.ja
      va:377)

      dataSource in data-config.xml has been with the batchSize of "-1".
      <dataSource driver="com.mysql.jdbc.Driver" url="jdbc:mysql://host/dbname"
      user="" password="" batchSize="-1"/>

        Activity

        Ricky Leung created issue -
        Noble Paul made changes -
        Field Original Value New Value
        Attachment SOLR-846.patch [ 12393844 ]
        Shalin Shekhar Mangar made changes -
        Fix Version/s 1.5 [ 12313566 ]
        Hoss Man made changes -
        Fix Version/s Next [ 12315093 ]
        Fix Version/s 1.5 [ 12313566 ]
        Hoss Man made changes -
        Fix Version/s 3.2 [ 12316172 ]
        Fix Version/s Next [ 12315093 ]
        Robert Muir made changes -
        Fix Version/s 3.3 [ 12316471 ]
        Fix Version/s 3.2 [ 12316172 ]
        Robert Muir made changes -
        Fix Version/s 3.4 [ 12316683 ]
        Fix Version/s 4.0 [ 12314992 ]
        Fix Version/s 3.3 [ 12316471 ]
        Robert Muir made changes -
        Fix Version/s 3.5 [ 12317876 ]
        Fix Version/s 3.4 [ 12316683 ]
        Simon Willnauer made changes -
        Fix Version/s 3.6 [ 12319065 ]
        Fix Version/s 3.5 [ 12317876 ]
        Hoss Man made changes -
        Fix Version/s 3.6 [ 12319065 ]
        Hoss Man made changes -
        Fix Version/s 4.0 [ 12322455 ]
        Fix Version/s 4.0-ALPHA [ 12314992 ]
        Robert Muir made changes -
        Fix Version/s 4.0 [ 12322551 ]
        Fix Version/s 4.0-BETA [ 12322455 ]
        Hoss Man made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Fix Version/s 1.4 [ 12313351 ]
        Fix Version/s 4.0 [ 12322551 ]
        Resolution Fixed [ 1 ]
        Uwe Schindler made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Unassigned
            Reporter:
            Ricky Leung
          • Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development