Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-9256

Solr 6.x DataImportHandler fails with postgreSQL dataSource with multiple joined entities

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 6.0, 6.0.1
    • 6.0.2, 6.1, 6.1.1, 6.2, 7.0
    • update
    • None
    • Solr 6.0, 6.0.1 Single Instance or SolrCloud with postgreSQL 9.4 Server on Java Version 1.8.0_91 runtime

    Description

      solr-data-config.xml

      <dataConfig>
        <dataSource name="jdbc" driver="org.postgresql.Driver" url="jdbc:postgresql://host:5432/database" user="username" password="password" readOnly="true" autoCommit="false" />
        <document>
          <entity name="outer" dataSource="jdbc" pk="id" query="SELECT id, a FROM table_a ORDER BY id ASC">
      		<field column="id" name="id" />
      		<field column="a" name="a" />
      	
      		<entity name="inner_one" query="SELECT a_id, b FROM table_b"
      				cacheKey="a_id" cacheLookup="outer.id" cacheImpl="SortedMapBackedCache">
      			<field column="b" name="b" />
      		</entity>
      		
      		<entity name="inner_two" query="SELECT a_id, c FROM table_c ORDER BY a_id ASC"
      				cacheKey="a_id" cacheLookup="outer.id" join="zipper">
      			<field column="c" name="c" />
      		</entity>
      
      	</entity>
        </document>
      </dataConfig>
      

      This works up to SOLR 5.5.2 (latest 5.x) but fails in SOLR 6.x.

      Exception:

      org.apache.solr.handler.dataimport.DataImportHandlerException: org.postgresql.util.PSQLException: Dieses ResultSet ist geschlossen.
      at org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:61)
      at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:434)
      at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:350)
      at com.google.common.collect.Iterators$PeekingImpl.hasNext(Iterators.java:1216)
      at org.apache.solr.handler.dataimport.Zipper.supplyNextChild(Zipper.java:65)
      at org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:127)
      at org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:75)
      at org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:244)
      at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:475)
      at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:514)
      at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:414)
      at org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:329)
      at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:232)
      at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:416)
      at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:480)
      at org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(DataImportHandler.java:200)
      at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)
      at org.apache.solr.core.SolrCore.execute(SolrCore.java:2053)
      at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:652)
      at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:460)
      at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:229)
      at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
      at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
      at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
      at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
      at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
      at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
      at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
      at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
      at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
      at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
      at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
      at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
      at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
      at org.eclipse.jetty.server.Server.handle(Server.java:518)
      at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
      at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
      at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
      at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
      at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
      at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
      at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
      at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
      at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
      at java.lang.Thread.run(Thread.java:745)

      Attachments

        1. SOLR-9256.patch
          3 kB
          Kristine Jetzke

        Issue Links

          Activity

            This could be caused by postgres limitations. Postgres only allows one open resultset per connection. Recent changes to JdbcDataSource.java optimized connection reusing and resultset closing.

            benjamin.richter Benjamin Richter added a comment - This could be caused by postgres limitations. Postgres only allows one open resultset per connection. Recent changes to JdbcDataSource.java optimized connection reusing and resultset closing.

            Can you declare same jdbc twice and use the second with the second entity?

            mkhl Mikhail Khludnev added a comment - Can you declare same jdbc twice and use the second with the second entity?

            tine,tinexw, can you comment on it?

            mkhl Mikhail Khludnev added a comment - tine , tinexw , can you comment on it?

            I tried to reproduce the problem with SOLR 6.1 again, but it seems this error does only occur on 6.0 and 6.0.1. Sorry for the confusion.

            benjamin.richter Benjamin Richter added a comment - I tried to reproduce the problem with SOLR 6.1 again, but it seems this error does only occur on 6.0 and 6.0.1. Sorry for the confusion.

            Tried SOLR 6.0.1 with multiple dataSources (jdbc1, jdbc2, ... jdbc22) ... same error.

            benjamin.richter Benjamin Richter added a comment - Tried SOLR 6.0.1 with multiple dataSources (jdbc1, jdbc2, ... jdbc22) ... same error.
            tinexw Kristine Jetzke added a comment - - edited

            I was able to reproduce it in 6.0.1 as well. It also works for me in 6.1.0.
            It fails in 6.0.1 if one of the inner queries returns no result.

            mkhludnev Which branches correspond to those versions? The result set handling in JdbcDataSource.java does not differ in branch_6_0 and branch_6_1. UPDATE: Nevermind, I found the tags. They do differ in the result handling.

            tinexw Kristine Jetzke added a comment - - edited I was able to reproduce it in 6.0.1 as well. It also works for me in 6.1.0. It fails in 6.0.1 if one of the inner queries returns no result. mkhludnev Which branches correspond to those versions? The result set handling in JdbcDataSource.java does not differ in branch_6_0 and branch_6_1 . UPDATE: Nevermind, I found the tags. They do differ in the result handling.
            tinexw Kristine Jetzke added a comment - - edited

            This commit broke it: https://github.com/apache/lucene-solr/commit/13c9912b3c4698595db8d07fcbc09fe062ee5404 (because the hasnext method always uses the initially created result set. This is closed if no more results are returned. Thus, the exception is thrown the next time the hasnext method is called. In the version before and after, the result set field is accessed instead which is set to null if no more results are available. Thus, no methods are called afterwards and no exception is thrown. )
            This commit fixed it: https://github.com/apache/lucene-solr/commit/22e5d31cdc9e94aec8043fd451ae1918b5062528

            benjamin.richter If you can't upgrade to 6.1.0, you could create a copy the current version of org.apache.solr.handler.dataimport.JdbcDataSource in your own source code and reference it in your data config files.

            tinexw Kristine Jetzke added a comment - - edited This commit broke it: https://github.com/apache/lucene-solr/commit/13c9912b3c4698595db8d07fcbc09fe062ee5404 (because the hasnext method always uses the initially created result set. This is closed if no more results are returned. Thus, the exception is thrown the next time the hasnext method is called. In the version before and after, the result set field is accessed instead which is set to null if no more results are available. Thus, no methods are called afterwards and no exception is thrown. ) This commit fixed it: https://github.com/apache/lucene-solr/commit/22e5d31cdc9e94aec8043fd451ae1918b5062528 benjamin.richter If you can't upgrade to 6.1.0, you could create a copy the current version of org.apache.solr.handler.dataimport.JdbcDataSource in your own source code and reference it in your data config files.
            tinexw Kristine Jetzke added a comment - - edited

            mkhludnev I attached a unit test for the master branch (The 6.0 branch would need a slightly different one) . I think there is nothing else to do since it works correctly in all open branches.

            tinexw Kristine Jetzke added a comment - - edited mkhludnev I attached a unit test for the master branch (The 6.0 branch would need a slightly different one) . I think there is nothing else to do since it works correctly in all open branches.

            Thanks, tinexw!

            mkhl Mikhail Khludnev added a comment - Thanks, tinexw !

            Commit 209bfcf02131e6c9196d3f9f6bd69d7ae2a6fc63 in lucene-solr's branch refs/heads/master from mkhludnev
            [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=209bfcf ]

            SOLR-9209,SOLR-9256: extracting
            JdbcDataSource.createResultSetIterator(), adding a test for
            ResultSetIterator.hasNext()

            jira-bot ASF subversion and git services added a comment - Commit 209bfcf02131e6c9196d3f9f6bd69d7ae2a6fc63 in lucene-solr's branch refs/heads/master from mkhludnev [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=209bfcf ] SOLR-9209 , SOLR-9256 : extracting JdbcDataSource.createResultSetIterator(), adding a test for ResultSetIterator.hasNext()

            Commit 77409fd43264e57d1b65f5fee3b9c7277d2f740a in lucene-solr's branch refs/heads/branch_6x from mkhludnev
            [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=77409fd ]

            SOLR-9209,SOLR-9256: extracting
            JdbcDataSource.createResultSetIterator(), adding a test for
            ResultSetIterator.hasNext()

            jira-bot ASF subversion and git services added a comment - Commit 77409fd43264e57d1b65f5fee3b9c7277d2f740a in lucene-solr's branch refs/heads/branch_6x from mkhludnev [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=77409fd ] SOLR-9209 , SOLR-9256 : extracting JdbcDataSource.createResultSetIterator(), adding a test for ResultSetIterator.hasNext()

            People

              Unassigned Unassigned
              benjamin.richter Benjamin Richter
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: