Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 5.1, 6.0
    • Component/s: None
    • Labels:
      None

      Description

      DIH should be able to index hierarchical documents, i.e. it should be able to work with SolrInputDocuments#addChildDocument.

      There was patch in SOLR-3076:
      https://issues.apache.org/jira/secure/attachment/12576960/dih-3076.patch
      But it is not uptodate and far from being complete.

      1. dih-oome-fix.patch
        6 kB
        Vadim Kirilchuk
      2. SOLR-5147.patch
        24 kB
        Thomas Champagne
      3. SOLR-5147.patch
        24 kB
        Thomas Champagne
      4. SOLR-5147-5x.patch
        24 kB
        Shawn Heisey

        Issue Links

          Activity

          Hide
          Uwe Schindler added a comment -

          Move issue to Solr 4.9.

          Show
          Uwe Schindler added a comment - Move issue to Solr 4.9.
          Hide
          Ted Sullivan added a comment -

          Agree with [~rkesamreddy@gmail.com] - I also have an ongoing project that could use this functionality which is even more compelling given the recent enhancements in 4.9 (SOLR-5285). I too will volunteer to get it "uptodate" and "complete" if it can be assigned to someone.

          Show
          Ted Sullivan added a comment - Agree with [~rkesamreddy@gmail.com] - I also have an ongoing project that could use this functionality which is even more compelling given the recent enhancements in 4.9 ( SOLR-5285 ). I too will volunteer to get it "uptodate" and "complete" if it can be assigned to someone.
          Hide
          Thomas Champagne added a comment - - edited

          I have updated the patch. There were only problems in the new unit test TestHierarchicalDocBuilder :

          • adding the field root in the dataimport schema.
          • the parent filter for the query is now a FixedBitSetCachingWrapperFilter.

          You can apply this patch in the trunk.

          All unit tests about dih are OK after applying the patch.

          If you want to test, you can add an attribute child at true in the entity tag in the data config file. The entities taggued with child will be added with the addChildDocument method.
          For example :

          <dataConfig>
            <dataSource type="JdbcDataSource" />
            <document>
              <entity name="PARENT" query="select * from PARENT">
                <field column="id" />
                <field column="desc" />
                <field column="type_s" />
                <entity child="true" name="CHILD" query="select * from CHILD where parent_id='${PARENT.id}'">
                  <field column="id" />
                  <field column="desc" />
                  <field column="type_s" />
                </entity>
              </entity>
            </document>
          </dataConfig>
          
          Show
          Thomas Champagne added a comment - - edited I have updated the patch. There were only problems in the new unit test TestHierarchicalDocBuilder : adding the field root in the dataimport schema. the parent filter for the query is now a FixedBitSetCachingWrapperFilter. You can apply this patch in the trunk. All unit tests about dih are OK after applying the patch. If you want to test, you can add an attribute child at true in the entity tag in the data config file. The entities taggued with child will be added with the addChildDocument method. For example : <dataConfig> <dataSource type= "JdbcDataSource" /> <document> <entity name= "PARENT" query= "select * from PARENT" > <field column= "id" /> <field column= "desc" /> <field column= "type_s" /> <entity child= "true" name= "CHILD" query= "select * from CHILD where parent_id='${PARENT.id}'" > <field column= "id" /> <field column= "desc" /> <field column= "type_s" /> </entity> </entity> </document> </dataConfig>
          Hide
          Andrew Ventura added a comment -

          Ok, this seemed to work well. Just a couple of comments/questions. do I need to add a rootEntity="true" to the root element, I did that, not sure if that was necessary or not. In addition, it seems like anything that is marked as required in the schema is required both in the Parent Doc and the Child Doc, is that correct? Not a big deal, but it does seem a bit odd. Performance seemed great, both in indexing and in searching. I adopted a style of using a field content_type to identify parent and child docs so I can specify what I want returned in my fq.

          Side note, when using DIH via the admin screen and you want to load a small number of records, I notice it does a SET ROWCOUNT. This is great, except that it totally messes up tables that I'm using cacheKeys on as it applies the rowcount to those as well. Ideally if I had a table setup for caching it would pull those first. If this is a legit issue I will figure out how to post it to JIRA, but I'm new to JIRA and SOLR.

          Show
          Andrew Ventura added a comment - Ok, this seemed to work well. Just a couple of comments/questions. do I need to add a rootEntity="true" to the root element, I did that, not sure if that was necessary or not. In addition, it seems like anything that is marked as required in the schema is required both in the Parent Doc and the Child Doc, is that correct? Not a big deal, but it does seem a bit odd. Performance seemed great, both in indexing and in searching. I adopted a style of using a field content_type to identify parent and child docs so I can specify what I want returned in my fq. Side note, when using DIH via the admin screen and you want to load a small number of records, I notice it does a SET ROWCOUNT. This is great, except that it totally messes up tables that I'm using cacheKeys on as it applies the rowcount to those as well. Ideally if I had a table setup for caching it would pull those first. If this is a legit issue I will figure out how to post it to JIRA, but I'm new to JIRA and SOLR.
          Hide
          Thomas Champagne added a comment -

          No, it's not mandatorty to add a rootEntity="true" to the root element because by default the entities falling under the document are root entities.

          About that is marked as required in the schema is required both in the Parent Doc and the Child Doc, the block join documents works like this. So, it is ok.

          And about your last problem, I don't understand it. Give us an exemple.

          Show
          Thomas Champagne added a comment - No, it's not mandatorty to add a rootEntity="true" to the root element because by default the entities falling under the document are root entities. About that is marked as required in the schema is required both in the Parent Doc and the Child Doc, the block join documents works like this. So, it is ok. And about your last problem, I don't understand it. Give us an exemple.
          Hide
          Henry Cleland added a comment -

          Hi Thomas Champagne i just applied the patch to trunk but the test TestHierarchicalDocBuilder failed. According to your comment, i am not clear if this failure is still to be expected after applying the patch or not. Kindly review output below.

          common.compile-test:
          [mkdir] Created dir: ../xx/lucene-solr/solr/build/contrib/solr-dataimporthandler/classes/test
          [javac] Compiling 47 source files to ../xx/lucene-solr/solr/build/contrib/solr-dataimporthandler/classes/test
          [javac] ../xx/lucene-solr/solr/contrib/dataimporthandler/src/test/org/apache/solr/handler/dataimport/TestHierarchicalDocBuilder.java:37: error: cannot find symbol
          [javac] import org.apache.lucene.search.join.FixedBitSetCachingWrapperFilter;
          [javac] ^
          [javac] symbol: class FixedBitSetCachingWrapperFilter
          [javac] location: package org.apache.lucene.search.join
          [javac] ../xx/lucene-solr/solr/contrib/dataimporthandler/src/test/org/apache/solr/handler/dataimport/TestHierarchicalDocBuilder.java:161: error: no suitable constructor found for ToParentBlockJoinQuery(ToParentBlockJoinQuery,Filter,ScoreMode)
          [javac] ToParentBlockJoinQuery blockJoinQuery = new ToParentBlockJoinQuery(childBlockJoinQuery, createParentFilter(parentType), ScoreMode.Avg);
          [javac] ^
          [javac] constructor ToParentBlockJoinQuery.ToParentBlockJoinQuery(Query,Query,BitDocIdSetFilter,ScoreMode) is not applicable
          [javac] (actual and formal argument lists differ in length)
          [javac] constructor ToParentBlockJoinQuery.ToParentBlockJoinQuery(Query,BitDocIdSetFilter,ScoreMode) is not applicable
          [javac] (actual argument Filter cannot be converted to BitDocIdSetFilter by method invocation conversion)
          [javac] ../xx/lucene-solr/solr/contrib/dataimporthandler/src/test/org/apache/solr/handler/dataimport/TestHierarchicalDocBuilder.java:212: error: no suitable constructor found for ToParentBlockJoinQuery(Query,Filter,ScoreMode)
          [javac] ToParentBlockJoinQuery blockJoinQuery = new ToParentBlockJoinQuery(childQuery, createParentFilter(parentType), ScoreMode.Avg);
          [javac] ^
          [javac] constructor ToParentBlockJoinQuery.ToParentBlockJoinQuery(Query,Query,BitDocIdSetFilter,ScoreMode) is not applicable
          [javac] (actual and formal argument lists differ in length)
          [javac] constructor ToParentBlockJoinQuery.ToParentBlockJoinQuery(Query,BitDocIdSetFilter,ScoreMode) is not applicable
          [javac] (actual argument Filter cannot be converted to BitDocIdSetFilter by method invocation conversion)
          [javac] ../xx/lucene-solr/solr/contrib/dataimporthandler/src/test/org/apache/solr/handler/dataimport/TestHierarchicalDocBuilder.java:381: error: cannot find symbol
          [javac] return new FixedBitSetCachingWrapperFilter(new QueryWrapperFilter(parentQuery));
          [javac] ^
          [javac] symbol: class FixedBitSetCachingWrapperFilter
          [javac] location: class TestHierarchicalDocBuilder
          [javac] Note: ../xx/lucene-solr/solr/contrib/dataimporthandler/src/test/org/apache/solr/handler/dataimport/TestJdbcDataSourceConvertType.java uses or overrides a deprecated API.
          [javac] Note: Recompile with -Xlint:deprecation for details.
          [javac] Note: Some input files use unchecked or unsafe operations.
          [javac] Note: Recompile with -Xlint:unchecked for details.
          [javac] 4 errors

          Show
          Henry Cleland added a comment - Hi Thomas Champagne i just applied the patch to trunk but the test TestHierarchicalDocBuilder failed. According to your comment, i am not clear if this failure is still to be expected after applying the patch or not. Kindly review output below. common.compile-test: [mkdir] Created dir: ../xx/lucene-solr/solr/build/contrib/solr-dataimporthandler/classes/test [javac] Compiling 47 source files to ../xx/lucene-solr/solr/build/contrib/solr-dataimporthandler/classes/test [javac] ../xx/lucene-solr/solr/contrib/dataimporthandler/src/test/org/apache/solr/handler/dataimport/TestHierarchicalDocBuilder.java:37: error: cannot find symbol [javac] import org.apache.lucene.search.join.FixedBitSetCachingWrapperFilter; [javac] ^ [javac] symbol: class FixedBitSetCachingWrapperFilter [javac] location: package org.apache.lucene.search.join [javac] ../xx/lucene-solr/solr/contrib/dataimporthandler/src/test/org/apache/solr/handler/dataimport/TestHierarchicalDocBuilder.java:161: error: no suitable constructor found for ToParentBlockJoinQuery(ToParentBlockJoinQuery,Filter,ScoreMode) [javac] ToParentBlockJoinQuery blockJoinQuery = new ToParentBlockJoinQuery(childBlockJoinQuery, createParentFilter(parentType), ScoreMode.Avg); [javac] ^ [javac] constructor ToParentBlockJoinQuery.ToParentBlockJoinQuery(Query,Query,BitDocIdSetFilter,ScoreMode) is not applicable [javac] (actual and formal argument lists differ in length) [javac] constructor ToParentBlockJoinQuery.ToParentBlockJoinQuery(Query,BitDocIdSetFilter,ScoreMode) is not applicable [javac] (actual argument Filter cannot be converted to BitDocIdSetFilter by method invocation conversion) [javac] ../xx/lucene-solr/solr/contrib/dataimporthandler/src/test/org/apache/solr/handler/dataimport/TestHierarchicalDocBuilder.java:212: error: no suitable constructor found for ToParentBlockJoinQuery(Query,Filter,ScoreMode) [javac] ToParentBlockJoinQuery blockJoinQuery = new ToParentBlockJoinQuery(childQuery, createParentFilter(parentType), ScoreMode.Avg); [javac] ^ [javac] constructor ToParentBlockJoinQuery.ToParentBlockJoinQuery(Query,Query,BitDocIdSetFilter,ScoreMode) is not applicable [javac] (actual and formal argument lists differ in length) [javac] constructor ToParentBlockJoinQuery.ToParentBlockJoinQuery(Query,BitDocIdSetFilter,ScoreMode) is not applicable [javac] (actual argument Filter cannot be converted to BitDocIdSetFilter by method invocation conversion) [javac] ../xx/lucene-solr/solr/contrib/dataimporthandler/src/test/org/apache/solr/handler/dataimport/TestHierarchicalDocBuilder.java:381: error: cannot find symbol [javac] return new FixedBitSetCachingWrapperFilter(new QueryWrapperFilter(parentQuery)); [javac] ^ [javac] symbol: class FixedBitSetCachingWrapperFilter [javac] location: class TestHierarchicalDocBuilder [javac] Note: ../xx/lucene-solr/solr/contrib/dataimporthandler/src/test/org/apache/solr/handler/dataimport/TestJdbcDataSourceConvertType.java uses or overrides a deprecated API. [javac] Note: Recompile with -Xlint:deprecation for details. [javac] Note: Some input files use unchecked or unsafe operations. [javac] Note: Recompile with -Xlint:unchecked for details. [javac] 4 errors
          Hide
          Mikhail Khludnev added a comment -

          Henry Cleland yep, trunk moved much in the pieces which are irrelevant to the patch: cached filters and BJQ.

          James Dyer is there a chance for this patch to be committed any time if I revamp it for trunk?

          Show
          Mikhail Khludnev added a comment - Henry Cleland yep, trunk moved much in the pieces which are irrelevant to the patch: cached filters and BJQ. James Dyer is there a chance for this patch to be committed any time if I revamp it for trunk?
          Hide
          Thomas Champagne added a comment -

          I have updated the patch on the current trunk. There is not a lot modification compared to the previous version :

          • org.apache.lucene.search.join.FixedBitSetCachingWrapperFilter has been renamed to org.apache.lucene.search.join.BitDocIdSetCachingWrapperFilter
          • The constructor org.apache.lucene.search.join.ToParentBlockJoinQuery.ToParentBlockJoinQuery(Query, BitDocIdSetFilter, ScoreMode) has changed and wants now a org.apache.lucene.search.join.BitDocIdSetFilter
          Show
          Thomas Champagne added a comment - I have updated the patch on the current trunk. There is not a lot modification compared to the previous version : org.apache.lucene.search.join.FixedBitSetCachingWrapperFilter has been renamed to org.apache.lucene.search.join.BitDocIdSetCachingWrapperFilter The constructor org.apache.lucene.search.join.ToParentBlockJoinQuery.ToParentBlockJoinQuery(Query, BitDocIdSetFilter, ScoreMode) has changed and wants now a org.apache.lucene.search.join.BitDocIdSetFilter
          Hide
          Mikhail Khludnev added a comment -

          Thomas Champagne good job!

          Show
          Mikhail Khludnev added a comment - Thomas Champagne good job!
          Hide
          Henry Cleland added a comment -

          Great!. You think this will be merged in anytime soon.?

          Show
          Henry Cleland added a comment - Great!. You think this will be merged in anytime soon.?
          Hide
          Shawn Heisey added a comment -

          This code cannot be merged into branch_5x in its current state. The new test tries to use a class that does not exist in branch_5x: org.apache.lucene.index.StoredDocument.

          Show
          Shawn Heisey added a comment - This code cannot be merged into branch_5x in its current state. The new test tries to use a class that does not exist in branch_5x: org.apache.lucene.index.StoredDocument.
          Hide
          Shawn Heisey added a comment - - edited

          Found the relevant change in Lucene. It is LUCENE-3312. Replacing StoredDocument with the old class (org.apache.lucene.document.Document) in the new test allows it to compile and pass on branch_5x.

          Show
          Shawn Heisey added a comment - - edited Found the relevant change in Lucene. It is LUCENE-3312 . Replacing StoredDocument with the old class (org.apache.lucene.document.Document) in the new test allows it to compile and pass on branch_5x.
          Hide
          Mikhail Khludnev added a comment -

          Shawn Heisey don't hesitate to attach the refreshed patch! You never know when a committer is attracted by this.

          Show
          Mikhail Khludnev added a comment - Shawn Heisey don't hesitate to attach the refreshed patch! You never know when a committer is attracted by this.
          Hide
          Noble Paul added a comment -

          Is the attached patch the final one? Does it work on both trunk and 5x ?

          Show
          Noble Paul added a comment - Is the attached patch the final one? Does it work on both trunk and 5x ?
          Hide
          Mikhail Khludnev added a comment -

          @noble.paul, nope

          @elyograg, would you mind to share your achievement?

          Show
          Mikhail Khludnev added a comment - @noble.paul, nope @elyograg, would you mind to share your achievement?
          Hide
          Shawn Heisey added a comment -

          Attached is a diff from 5x where I applied the original patch and then made a change in the new test so it would compile and run.

          Show
          Shawn Heisey added a comment - Attached is a diff from 5x where I applied the original patch and then made a change in the new test so it would compile and run.
          Hide
          Shawn Heisey added a comment - - edited

          The change is simple, already described above. Here is a more explicit change description: I changed the StoredDocument import to org.apache.lucene.document.Document, then changed the type of "doc" to Document.

          Show
          Shawn Heisey added a comment - - edited The change is simple, already described above. Here is a more explicit change description: I changed the StoredDocument import to org.apache.lucene.document.Document, then changed the type of "doc" to Document.
          Hide
          Mikhail Khludnev added a comment -

          usage example:
          mind adding child='true' attribute into nesting child entity

          <document>
            <entity name='PARENT' query='select * from PARENT'>
              <field column='id' />
              <field column='desc' />
              <field column='type_s' />
              <entity child='true' name='CHILD' query="select * from CHILD where parent_id='${PARENT.id}'">
                <field column='id' />
                <field column='desc' />
                <field column='type_s' />
            </entity>
          </entity>
          </document>
          
          Show
          Mikhail Khludnev added a comment - usage example: mind adding child='true' attribute into nesting child entity <document> <entity name='PARENT' query='select * from PARENT'> <field column='id' /> <field column='desc' /> <field column='type_s' /> <entity child=' true ' name='CHILD' query= "select * from CHILD where parent_id='${PARENT.id}'" > <field column='id' /> <field column='desc' /> <field column='type_s' /> </entity> </entity> </document>
          Hide
          Noble Paul added a comment -

          hi Mikhail Khludnev Please let me know if this patch is final so that I can review and commit. I don't think it applies well to trunk

          Show
          Noble Paul added a comment - hi Mikhail Khludnev Please let me know if this patch is final so that I can review and commit. I don't think it applies well to trunk
          Hide
          ASF subversion and git services added a comment -

          Commit 1652360 from Noble Paul in branch 'dev/trunk'
          [ https://svn.apache.org/r1652360 ]

          SOLR-5147 Support child documents in DIH

          Show
          ASF subversion and git services added a comment - Commit 1652360 from Noble Paul in branch 'dev/trunk' [ https://svn.apache.org/r1652360 ] SOLR-5147 Support child documents in DIH
          Hide
          ASF subversion and git services added a comment -

          Commit 1652365 from Noble Paul in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1652365 ]

          SOLR-5147 Support child documents in DIH

          Show
          ASF subversion and git services added a comment - Commit 1652365 from Noble Paul in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1652365 ] SOLR-5147 Support child documents in DIH
          Hide
          ASF subversion and git services added a comment -

          Commit 1652370 from Noble Paul in branch 'dev/branches/lucene_solr_5_0'
          [ https://svn.apache.org/r1652370 ]

          SOLR-5147 Support child documents in DIH

          Show
          ASF subversion and git services added a comment - Commit 1652370 from Noble Paul in branch 'dev/branches/lucene_solr_5_0' [ https://svn.apache.org/r1652370 ] SOLR-5147 Support child documents in DIH
          Hide
          Mikhail Khludnev added a comment - - edited

          Noble Paul and everybody, thanks!
          Noble, I'm afraid CHANGES.txt doesn't mention Vadim Kirilchuk who made the original patch, I appreciate if you can include him.

          Show
          Mikhail Khludnev added a comment - - edited Noble Paul and everybody, thanks! Noble, I'm afraid CHANGES.txt doesn't mention Vadim Kirilchuk who made the original patch , I appreciate if you can include him.
          Hide
          Mark Miller added a comment -

          [mvn] [ERROR] Forbidden method invocation:
          java.util.Collections#shuffle(java.util.List) [Use shuffle(List,
          Random) instead so that it can be reproduced]
          [mvn] [ERROR] in
          org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder
          (TestHierarchicalDocBuilder.java:98)
          [mvn] [ERROR] Scanned 85 (and 403 related) class file(s) for
          forbidden API invocations (in 0.14s), 1 error(s).

          Show
          Mark Miller added a comment - [mvn] [ERROR] Forbidden method invocation: java.util.Collections#shuffle(java.util.List) [Use shuffle(List, Random) instead so that it can be reproduced] [mvn] [ERROR] in org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder (TestHierarchicalDocBuilder.java:98) [mvn] [ERROR] Scanned 85 (and 403 related) class file(s) for forbidden API invocations (in 0.14s), 1 error(s).
          Hide
          ASF subversion and git services added a comment -

          Commit 1652425 from Noble Paul in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1652425 ]

          SOLR-5147 remove forbidden API

          Show
          ASF subversion and git services added a comment - Commit 1652425 from Noble Paul in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1652425 ] SOLR-5147 remove forbidden API
          Hide
          Mark Miller added a comment -

          Build: http://jenkins.thetaphi.de/job/Lucene-Solr-5.x-Linux/11456/
          Java: 32bit/jdk1.8.0_40-ea-b20 -server -XX:+UseG1GC

          1 tests failed.
          FAILED: org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.testRandomDepthHierarchy

          Error Message:
          Java heap space

          Stack Trace:
          java.lang.OutOfMemoryError: Java heap space
          at __randomizedtesting.SeedInfo.seed([3EF55A1AECFE9E4:5BBC3484B3EA26DE]:0)
          at java.util.HashMap.resize(HashMap.java:703)
          at java.util.HashMap.putVal(HashMap.java:628)
          at java.util.HashMap.put(HashMap.java:611)

          Show
          Mark Miller added a comment - Build: http://jenkins.thetaphi.de/job/Lucene-Solr-5.x-Linux/11456/ Java: 32bit/jdk1.8.0_40-ea-b20 -server -XX:+UseG1GC 1 tests failed. FAILED: org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.testRandomDepthHierarchy Error Message: Java heap space Stack Trace: java.lang.OutOfMemoryError: Java heap space at __randomizedtesting.SeedInfo.seed( [3EF55A1AECFE9E4:5BBC3484B3EA26DE] :0) at java.util.HashMap.resize(HashMap.java:703) at java.util.HashMap.putVal(HashMap.java:628) at java.util.HashMap.put(HashMap.java:611)
          Hide
          ASF subversion and git services added a comment -

          Commit 1652426 from Noble Paul in branch 'dev/trunk'
          [ https://svn.apache.org/r1652426 ]

          SOLR-5147 remove forbidden API

          Show
          ASF subversion and git services added a comment - Commit 1652426 from Noble Paul in branch 'dev/trunk' [ https://svn.apache.org/r1652426 ] SOLR-5147 remove forbidden API
          Hide
          ASF subversion and git services added a comment -

          Commit 1652427 from Noble Paul in branch 'dev/branches/lucene_solr_5_0'
          [ https://svn.apache.org/r1652427 ]

          remove forbidden API call SOLR-5147

          Show
          ASF subversion and git services added a comment - Commit 1652427 from Noble Paul in branch 'dev/branches/lucene_solr_5_0' [ https://svn.apache.org/r1652427 ] remove forbidden API call SOLR-5147
          Hide
          Mark Miller added a comment -

          The OOM is on a few jenkins fails on the list.

          Show
          Mark Miller added a comment - The OOM is on a few jenkins fails on the list.
          Hide
          Mark Miller added a comment -

          I'm starting to mind less that your commit messages are not consistent with the rest of the project or even usually with themselves. It has become easier to spot commits that break the build or introduce more flakey tests that are not ready.

          Show
          Mark Miller added a comment - I'm starting to mind less that your commit messages are not consistent with the rest of the project or even usually with themselves. It has become easier to spot commits that break the build or introduce more flakey tests that are not ready.
          Hide
          Mikhail Khludnev added a comment -

          I'm able to reproduce on my laptop with nightly=true. I will fix the test on a weekend. Please consider to @Ignore test so far.

          Show
          Mikhail Khludnev added a comment - I'm able to reproduce on my laptop with nightly=true. I will fix the test on a weekend. Please consider to @Ignore test so far.
          Hide
          Vadim Kirilchuk added a comment -

          Reproduced for:
          -Dtests.seed=5AF0AAE9289B3D28
          -Dtests.slow=true
          -Xmx512M

          ant doesn't need -Xmx512M property, I used it to reproduce failure in eclipse.

          Well, the test is exponential, it creates "parentsNum" hierarchies and for each parent we create nested children and for nested children we create nested grandchildren, etc. Maximum level of grand...grandchildren is "depth". The number of children on each level is randomized too.

          For the seed I have parentsNum=6 and depth=4 but overall it is a 404305 entities.

          I think it is the reason of a fail for a small heap, but going to investigate it further.

          Show
          Vadim Kirilchuk added a comment - Reproduced for: -Dtests.seed=5AF0AAE9289B3D28 -Dtests.slow=true -Xmx512M ant doesn't need -Xmx512M property, I used it to reproduce failure in eclipse. Well, the test is exponential, it creates "parentsNum" hierarchies and for each parent we create nested children and for nested children we create nested grandchildren, etc. Maximum level of grand...grandchildren is "depth". The number of children on each level is randomized too. For the seed I have parentsNum=6 and depth=4 but overall it is a 404305 entities. I think it is the reason of a fail for a small heap, but going to investigate it further.
          Hide
          Vadim Kirilchuk added a comment -

          I attaching the patch which uses randomInt instead of atLeast which is affected by multiplier much. It should help with exponential grow of a number of the documents.

          Also I added javadocs.

          P.s. And yes, I would really appreciate if you won't forget my name in changes.txt this time

          Show
          Vadim Kirilchuk added a comment - I attaching the patch which uses randomInt instead of atLeast which is affected by multiplier much. It should help with exponential grow of a number of the documents. Also I added javadocs. P.s. And yes, I would really appreciate if you won't forget my name in changes.txt this time
          Hide
          Noble Paul added a comment -

          Thanks for the patch and sorry for the omission

          Show
          Noble Paul added a comment - Thanks for the patch and sorry for the omission
          Hide
          ASF subversion and git services added a comment -

          Commit 1652618 from Noble Paul in branch 'dev/trunk'
          [ https://svn.apache.org/r1652618 ]

          SOLR-5147 addressing the OOM failures

          Show
          ASF subversion and git services added a comment - Commit 1652618 from Noble Paul in branch 'dev/trunk' [ https://svn.apache.org/r1652618 ] SOLR-5147 addressing the OOM failures
          Hide
          ASF subversion and git services added a comment -

          Commit 1652622 from Noble Paul in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1652622 ]

          SOLR-5147 addressing the OOM failures

          Show
          ASF subversion and git services added a comment - Commit 1652622 from Noble Paul in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1652622 ] SOLR-5147 addressing the OOM failures
          Hide
          ASF subversion and git services added a comment -

          Commit 1652626 from Noble Paul in branch 'dev/branches/lucene_solr_5_0'
          [ https://svn.apache.org/r1652626 ]

          SOLR-5147 addressing the OOM failures

          Show
          ASF subversion and git services added a comment - Commit 1652626 from Noble Paul in branch 'dev/branches/lucene_solr_5_0' [ https://svn.apache.org/r1652626 ] SOLR-5147 addressing the OOM failures
          Hide
          ASF subversion and git services added a comment -

          Commit 1652652 from Noble Paul in branch 'dev/branches/lucene_solr_5_0'
          [ https://svn.apache.org/r1652652 ]

          SOLR-5147 reverting

          Show
          ASF subversion and git services added a comment - Commit 1652652 from Noble Paul in branch 'dev/branches/lucene_solr_5_0' [ https://svn.apache.org/r1652652 ] SOLR-5147 reverting
          Hide
          ASF subversion and git services added a comment -

          Commit 1653357 from shalin@apache.org in branch 'dev/trunk'
          [ https://svn.apache.org/r1653357 ]

          SOLR-5147: Add dependency to blockjoin lucene module in idea config

          Show
          ASF subversion and git services added a comment - Commit 1653357 from shalin@apache.org in branch 'dev/trunk' [ https://svn.apache.org/r1653357 ] SOLR-5147 : Add dependency to blockjoin lucene module in idea config
          Hide
          ASF subversion and git services added a comment -

          Commit 1653358 from shalin@apache.org in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1653358 ]

          SOLR-5147: Add dependency to blockjoin lucene module in idea config

          Show
          ASF subversion and git services added a comment - Commit 1653358 from shalin@apache.org in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1653358 ] SOLR-5147 : Add dependency to blockjoin lucene module in idea config
          Hide
          Mikhail Khludnev added a comment -

          wasn't it committed fully already? I wonder why it wasn't resolved/closed so far?

          Show
          Mikhail Khludnev added a comment - wasn't it committed fully already? I wonder why it wasn't resolved/closed so far?
          Hide
          Harish Raghavan added a comment -

          Does this patch also cover support for FileDataSource/XPathEntityProcessor ?

          Show
          Harish Raghavan added a comment - Does this patch also cover support for FileDataSource/XPathEntityProcessor ?
          Hide
          Timothy Potter added a comment -

          Bulk close after 5.1 release

          Show
          Timothy Potter added a comment - Bulk close after 5.1 release
          Hide
          Mikhail Khludnev added a comment -

          Harish Raghavan it should.

          Show
          Mikhail Khludnev added a comment - Harish Raghavan it should.
          Hide
          Mikhail Khludnev added a comment -

          I see a minor usability issue: nested documents aren't shown when debugging DIH in SolrAdmin, because SolrInputDocument is rendered as Map w/o children. It's a simple issue, raise if you wish it to be addressed.

          Show
          Mikhail Khludnev added a comment - I see a minor usability issue: nested documents aren't shown when debugging DIH in SolrAdmin, because SolrInputDocument is rendered as Map w/o children. It's a simple issue, raise if you wish it to be addressed.
          Hide
          Mikhail Khludnev added a comment -

          follow up with SOLR-8762

          Show
          Mikhail Khludnev added a comment - follow up with SOLR-8762

            People

            • Assignee:
              Noble Paul
              Reporter:
              Vadim Kirilchuk
            • Votes:
              11 Vote for this issue
              Watchers:
              20 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development