Details

    • Type: Task Task
    • Status: Closed
    • Priority: Trivial Trivial
    • Resolution: Fixed
    • Affects Version/s: 1.3
    • Fix Version/s: 1.5
    • Component/s: indexer
    • Labels:
      None

      Description

      Default document batch size should be decreased to prevent OOMEs.

        Activity

        Hide
        Julien Nioche added a comment -

        Or we could catch the exceptions (OOME or else) and retry the docs in the buffer one by one which will also help identify when a specific document is causing problem

        Show
        Julien Nioche added a comment - Or we could catch the exceptions (OOME or else) and retry the docs in the buffer one by one which will also help identify when a specific document is causing problem
        Hide
        Markus Jelsma added a comment -

        Would that be feasible? It's not easy to restore to a previous state and resume differently and we don't really know where it's thrown from. The Javadocs on Error states that these should not be caught. Also, a failing reducer will retry anyway for a number of times.

        Do you think it's worth the effort?

        Show
        Markus Jelsma added a comment - Would that be feasible? It's not easy to restore to a previous state and resume differently and we don't really know where it's thrown from. The Javadocs on Error states that these should not be caught. Also, a failing reducer will retry anyway for a number of times. Do you think it's worth the effort?
        Hide
        Markus Jelsma added a comment -

        Committed for nutchgora in rev. 1200834. and for trunk in rev. 1200833.

        Show
        Markus Jelsma added a comment - Committed for nutchgora in rev. 1200834. and for trunk in rev. 1200833.
        Hide
        Hudson added a comment -

        Integrated in nutch-trunk-maven #19 (See https://builds.apache.org/job/nutch-trunk-maven/19/)
        NUTCH-1185 Decrease solr.commit.size to 250

        markus : http://svn.apache.org/viewvc/nutch/trunk/viewvc/?view=rev&root=&revision=1200833
        Files :

        • /nutch/trunk/CHANGES.txt
        • /nutch/trunk/conf/nutch-default.xml
        Show
        Hudson added a comment - Integrated in nutch-trunk-maven #19 (See https://builds.apache.org/job/nutch-trunk-maven/19/ ) NUTCH-1185 Decrease solr.commit.size to 250 markus : http://svn.apache.org/viewvc/nutch/trunk/viewvc/?view=rev&root=&revision=1200833 Files : /nutch/trunk/CHANGES.txt /nutch/trunk/conf/nutch-default.xml
        Hide
        Hudson added a comment -

        Integrated in Nutch-trunk #1660 (See https://builds.apache.org/job/Nutch-trunk/1660/)
        NUTCH-1185 Decrease solr.commit.size to 250

        markus : http://svn.apache.org/viewvc/nutch/trunk/viewvc/?view=rev&root=&revision=1200833
        Files :

        • /nutch/trunk/CHANGES.txt
        • /nutch/trunk/conf/nutch-default.xml
        Show
        Hudson added a comment - Integrated in Nutch-trunk #1660 (See https://builds.apache.org/job/Nutch-trunk/1660/ ) NUTCH-1185 Decrease solr.commit.size to 250 markus : http://svn.apache.org/viewvc/nutch/trunk/viewvc/?view=rev&root=&revision=1200833 Files : /nutch/trunk/CHANGES.txt /nutch/trunk/conf/nutch-default.xml
        Hide
        Hudson added a comment -

        Integrated in Nutch-trunk-ant #77 (See https://builds.apache.org/job/Nutch-trunk-ant/77/)
        NUTCH-1185 Decrease solr.commit.size to 250

        markus : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1200833
        Files :

        • /nutch/trunk/CHANGES.txt
        • /nutch/trunk/conf/nutch-default.xml
        Show
        Hudson added a comment - Integrated in Nutch-trunk-ant #77 (See https://builds.apache.org/job/Nutch-trunk-ant/77/ ) NUTCH-1185 Decrease solr.commit.size to 250 markus : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1200833 Files : /nutch/trunk/CHANGES.txt /nutch/trunk/conf/nutch-default.xml
        Hide
        Hudson added a comment -

        Integrated in Nutch-nutchgora #65 (See https://builds.apache.org/job/Nutch-nutchgora/65/)
        NUTCH-1185 Decrease solr.commit.size to 250

        markus : http://svn.apache.org/viewvc/nutch/branches/nutchgora/viewvc/?view=rev&root=&revision=1200834
        Files :

        • /nutch/branches/nutchgora/CHANGES.txt
        • /nutch/branches/nutchgora/conf/nutch-default.xml
        Show
        Hudson added a comment - Integrated in Nutch-nutchgora #65 (See https://builds.apache.org/job/Nutch-nutchgora/65/ ) NUTCH-1185 Decrease solr.commit.size to 250 markus : http://svn.apache.org/viewvc/nutch/branches/nutchgora/viewvc/?view=rev&root=&revision=1200834 Files : /nutch/branches/nutchgora/CHANGES.txt /nutch/branches/nutchgora/conf/nutch-default.xml
        Hide
        Hudson added a comment -

        Integrated in Nutch-nutchgora-ant #19 (See https://builds.apache.org/job/Nutch-nutchgora-ant/19/)
        NUTCH-1185 Decrease solr.commit.size to 250

        markus : http://svn.apache.org/viewvc/nutch/branches/nutchgora/viewvc/?view=rev&root=&revision=1200834
        Files :

        • /nutch/branches/nutchgora/CHANGES.txt
        • /nutch/branches/nutchgora/conf/nutch-default.xml
        Show
        Hudson added a comment - Integrated in Nutch-nutchgora-ant #19 (See https://builds.apache.org/job/Nutch-nutchgora-ant/19/ ) NUTCH-1185 Decrease solr.commit.size to 250 markus : http://svn.apache.org/viewvc/nutch/branches/nutchgora/viewvc/?view=rev&root=&revision=1200834 Files : /nutch/branches/nutchgora/CHANGES.txt /nutch/branches/nutchgora/conf/nutch-default.xml

          People

          • Assignee:
            Markus Jelsma
            Reporter:
            Markus Jelsma
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development