Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-9256

Solr 6.x DataImportHandler fails with postgreSQL dataSource with multiple joined entities

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 6.0, 6.0.1
    • Fix Version/s: 6.0.2, 6.1, 6.1.1, 6.2, 7.0
    • Component/s: update
    • Security Level: Public (Default Security Level. Issues are Public)
    • Labels:
      None
    • Environment:

      Solr 6.0, 6.0.1 Single Instance or SolrCloud with postgreSQL 9.4 Server on Java Version 1.8.0_91 runtime

      Description

      solr-data-config.xml

      <dataConfig>
        <dataSource name="jdbc" driver="org.postgresql.Driver" url="jdbc:postgresql://host:5432/database" user="username" password="password" readOnly="true" autoCommit="false" />
        <document>
          <entity name="outer" dataSource="jdbc" pk="id" query="SELECT id, a FROM table_a ORDER BY id ASC">
      		<field column="id" name="id" />
      		<field column="a" name="a" />
      	
      		<entity name="inner_one" query="SELECT a_id, b FROM table_b"
      				cacheKey="a_id" cacheLookup="outer.id" cacheImpl="SortedMapBackedCache">
      			<field column="b" name="b" />
      		</entity>
      		
      		<entity name="inner_two" query="SELECT a_id, c FROM table_c ORDER BY a_id ASC"
      				cacheKey="a_id" cacheLookup="outer.id" join="zipper">
      			<field column="c" name="c" />
      		</entity>
      
      	</entity>
        </document>
      </dataConfig>
      

      This works up to SOLR 5.5.2 (latest 5.x) but fails in SOLR 6.x.

      Exception:

      org.apache.solr.handler.dataimport.DataImportHandlerException: org.postgresql.util.PSQLException: Dieses ResultSet ist geschlossen.
      at org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:61)
      at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:434)
      at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:350)
      at com.google.common.collect.Iterators$PeekingImpl.hasNext(Iterators.java:1216)
      at org.apache.solr.handler.dataimport.Zipper.supplyNextChild(Zipper.java:65)
      at org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:127)
      at org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:75)
      at org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:244)
      at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:475)
      at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:514)
      at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:414)
      at org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:329)
      at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:232)
      at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:416)
      at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:480)
      at org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(DataImportHandler.java:200)
      at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)
      at org.apache.solr.core.SolrCore.execute(SolrCore.java:2053)
      at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:652)
      at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:460)
      at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:229)
      at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
      at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
      at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
      at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
      at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
      at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
      at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
      at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
      at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
      at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
      at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
      at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
      at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
      at org.eclipse.jetty.server.Server.handle(Server.java:518)
      at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
      at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
      at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
      at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
      at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
      at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
      at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
      at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
      at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
      at java.lang.Thread.run(Thread.java:745)

      1. SOLR-9256.patch
        3 kB
        Kristine Jetzke

        Issue Links

          Activity

          Hide
          benjamin.richter Benjamin Richter added a comment -

          This could be caused by postgres limitations. Postgres only allows one open resultset per connection. Recent changes to JdbcDataSource.java optimized connection reusing and resultset closing.

          Show
          benjamin.richter Benjamin Richter added a comment - This could be caused by postgres limitations. Postgres only allows one open resultset per connection. Recent changes to JdbcDataSource.java optimized connection reusing and resultset closing.
          Hide
          mkhludnev Mikhail Khludnev added a comment -

          Can you declare same jdbc twice and use the second with the second entity?

          Show
          mkhludnev Mikhail Khludnev added a comment - Can you declare same jdbc twice and use the second with the second entity?
          Hide
          mkhludnev Mikhail Khludnev added a comment -

          Kristine Jetzke,Kristine Jetzke, can you comment on it?

          Show
          mkhludnev Mikhail Khludnev added a comment - Kristine Jetzke , Kristine Jetzke , can you comment on it?
          Hide
          benjamin.richter Benjamin Richter added a comment -

          I tried to reproduce the problem with SOLR 6.1 again, but it seems this error does only occur on 6.0 and 6.0.1. Sorry for the confusion.

          Show
          benjamin.richter Benjamin Richter added a comment - I tried to reproduce the problem with SOLR 6.1 again, but it seems this error does only occur on 6.0 and 6.0.1. Sorry for the confusion.
          Hide
          benjamin.richter Benjamin Richter added a comment -

          Tried SOLR 6.0.1 with multiple dataSources (jdbc1, jdbc2, ... jdbc22) ... same error.

          Show
          benjamin.richter Benjamin Richter added a comment - Tried SOLR 6.0.1 with multiple dataSources (jdbc1, jdbc2, ... jdbc22) ... same error.
          Hide
          tinexw Kristine Jetzke added a comment - - edited

          I was able to reproduce it in 6.0.1 as well. It also works for me in 6.1.0.
          It fails in 6.0.1 if one of the inner queries returns no result.

          Mikhail Khludnev Which branches correspond to those versions? The result set handling in JdbcDataSource.java does not differ in branch_6_0 and branch_6_1. UPDATE: Nevermind, I found the tags. They do differ in the result handling.

          Show
          tinexw Kristine Jetzke added a comment - - edited I was able to reproduce it in 6.0.1 as well. It also works for me in 6.1.0. It fails in 6.0.1 if one of the inner queries returns no result. Mikhail Khludnev Which branches correspond to those versions? The result set handling in JdbcDataSource.java does not differ in branch_6_0 and branch_6_1 . UPDATE: Nevermind, I found the tags. They do differ in the result handling.
          Hide
          tinexw Kristine Jetzke added a comment - - edited

          This commit broke it: https://github.com/apache/lucene-solr/commit/13c9912b3c4698595db8d07fcbc09fe062ee5404 (because the hasnext method always uses the initially created result set. This is closed if no more results are returned. Thus, the exception is thrown the next time the hasnext method is called. In the version before and after, the result set field is accessed instead which is set to null if no more results are available. Thus, no methods are called afterwards and no exception is thrown. )
          This commit fixed it: https://github.com/apache/lucene-solr/commit/22e5d31cdc9e94aec8043fd451ae1918b5062528

          Benjamin Richter If you can't upgrade to 6.1.0, you could create a copy the current version of org.apache.solr.handler.dataimport.JdbcDataSource in your own source code and reference it in your data config files.

          Show
          tinexw Kristine Jetzke added a comment - - edited This commit broke it: https://github.com/apache/lucene-solr/commit/13c9912b3c4698595db8d07fcbc09fe062ee5404 (because the hasnext method always uses the initially created result set. This is closed if no more results are returned. Thus, the exception is thrown the next time the hasnext method is called. In the version before and after, the result set field is accessed instead which is set to null if no more results are available. Thus, no methods are called afterwards and no exception is thrown. ) This commit fixed it: https://github.com/apache/lucene-solr/commit/22e5d31cdc9e94aec8043fd451ae1918b5062528 Benjamin Richter If you can't upgrade to 6.1.0, you could create a copy the current version of org.apache.solr.handler.dataimport.JdbcDataSource in your own source code and reference it in your data config files.
          Hide
          tinexw Kristine Jetzke added a comment - - edited

          Mikhail Khludnev I attached a unit test for the master branch (The 6.0 branch would need a slightly different one) . I think there is nothing else to do since it works correctly in all open branches.

          Show
          tinexw Kristine Jetzke added a comment - - edited Mikhail Khludnev I attached a unit test for the master branch (The 6.0 branch would need a slightly different one) . I think there is nothing else to do since it works correctly in all open branches.
          Hide
          mkhludnev Mikhail Khludnev added a comment -

          Thanks, Kristine Jetzke!

          Show
          mkhludnev Mikhail Khludnev added a comment - Thanks, Kristine Jetzke !
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 209bfcf02131e6c9196d3f9f6bd69d7ae2a6fc63 in lucene-solr's branch refs/heads/master from Mikhail Khludnev
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=209bfcf ]

          SOLR-9209,SOLR-9256: extracting
          JdbcDataSource.createResultSetIterator(), adding a test for
          ResultSetIterator.hasNext()

          Show
          jira-bot ASF subversion and git services added a comment - Commit 209bfcf02131e6c9196d3f9f6bd69d7ae2a6fc63 in lucene-solr's branch refs/heads/master from Mikhail Khludnev [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=209bfcf ] SOLR-9209 , SOLR-9256 : extracting JdbcDataSource.createResultSetIterator(), adding a test for ResultSetIterator.hasNext()
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 77409fd43264e57d1b65f5fee3b9c7277d2f740a in lucene-solr's branch refs/heads/branch_6x from Mikhail Khludnev
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=77409fd ]

          SOLR-9209,SOLR-9256: extracting
          JdbcDataSource.createResultSetIterator(), adding a test for
          ResultSetIterator.hasNext()

          Show
          jira-bot ASF subversion and git services added a comment - Commit 77409fd43264e57d1b65f5fee3b9c7277d2f740a in lucene-solr's branch refs/heads/branch_6x from Mikhail Khludnev [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=77409fd ] SOLR-9209 , SOLR-9256 : extracting JdbcDataSource.createResultSetIterator(), adding a test for ResultSetIterator.hasNext()

            People

            • Assignee:
              Unassigned
              Reporter:
              benjamin.richter Benjamin Richter
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development