Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-9246

Errors for Streaming Expressions using JDBC (Oracle) stream source

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 6.0.1
    • Fix Version/s: 6.0.2, 6.1.1, 6.2, 7.0
    • Component/s: None
    • Labels:
      None
    • Environment:

      Windows 7

      Description

      I have Solr 6.0.0 installed on my PC (windows 7), I was experimenting with ‘Streaming Expression’ by using Oracle jdbc as the
      stream source, but got 'null pointer' errors, below is the details on how to reproduce this error:

      1. create a collection 'document6' which only contain long and string data type,

      schema.xml for Solr collection 'document6': (newly created empty collections with 2 shards)
      ===========================================================================================
      <schema name="document6" version="1.1">
      <types>
      <fieldType name="uuid" class="solr.UUIDField" />
      <fieldType name="bigint" class="solr.TrieLongField" />
      <fieldType name="string" class="solr.StrField" sortMissingLast="true" docValues="true" />
      <fieldType name="long" class="solr.TrieLongField" docValues="true" precisionStep="0" positionIncrementGap="0"/>
      <fieldType name="text" class="solr.TextField">
      <analyzer>
      <tokenizer class="solr.StandardTokenizerFactory"/>
      </analyzer>
      </fieldType>
      <fieldType name="boolean" class="solr.BoolField" sortMissingLast="true" omitNorms="true"/>
      </types>
      <fields>
      <field name="version" type="long" indexed="true" stored="true" multiValued="false"/>
      <field name="document_id" type="long" indexed="true" stored="true" docValues="true"/>
      <field name="sender_msg_dest" type="string" indexed="true" stored="true" docValues="true"/>
      <field name="recip_msg_dest" type="string" indexed="true" stored="true" docValues="true"/>
      <field name="document_type" type="string" indexed="true" stored="true" docValues="true"/>
      <field name="document_key" type="string" indexed="true" stored="true" docValues="true"/>
      </fields>
      <defaultSearchField>document_id</defaultSearchField>
      <uniqueKey>document_id</uniqueKey>
      </schema>

      2. create a new Oracle (version 11.2.0.3) table 'document6' that only contain columns whose jdbc type is long and string,

      create table document6
      (document_id number(12) not null,
      sender_msg_dest varchar2(256),
      recip_msg_dest varchar2(256),
      document_type varchar2(20),
      document_key varchar2(100));

      loaded 9 records;

      Oracle table 'document6': (newly created Oracle table with 9 records)
      =====================================================================
      QA_DOCREP@qlgdb1 > desc document6
      Name Null? Type
      ----------------------------------------- -------- ----------------------------
      DOCUMENT_ID NOT NULL NUMBER(12)
      SENDER_MSG_DEST VARCHAR2(256)
      RECIP_MSG_DEST VARCHAR2(256)
      DOCUMENT_TYPE VARCHAR2(20)
      DOCUMENT_KEY VARCHAR2(100)

      3. tried this jdbc streaming expression in my browser, getting the error stack (see below)

      http://localhost:8988/solr/document6/stream?expr=jdbc(connection="jdbc:oracle:thin:qa_docrep/abc123@lit-racq01-scan.qa.gxsonline.net:1521/qlgdb",sql="SELECT document_id,sender_msg_dest,recip_msg_dest,document_type,document_key FROM document6",sort="document_id asc",driver="oracle.jdbc.driver.OracleDriver")

      errors in solr.log
      ==================
      2016-06-23 14:07:02.833 INFO (qtp1389647288-139) [c:document6 s:shard2 r:core_node1 x:document6_shard2_replica1] o.a.s.c.S.Request [document6_shard2_replica1] webapp=/solr path=/stream params=

      {expr=jdbc(connection%3D"jdbc:oracle:thin:qa_docrep/abc123@lit-racq01-scan.qa.gxsonline.net:1521/qlgdb",sql%3D"SELECT+document_id,sender_msg_dest,recip_msg_dest,document_type,document_key+FROM+document6",sort%3D"document_id+asc",driver%3D"oracle.jdbc.driver.OracleDriver")}

      status=0 QTime=1
      2016-06-23 14:07:05.282 ERROR (qtp1389647288-139) [c:document6 s:shard2 r:core_node1 x:document6_shard2_replica1] o.a.s.c.s.i.s.ExceptionStream java.lang.NullPointerException
      at org.apache.solr.client.solrj.io.stream.JDBCStream.read(JDBCStream.java:305)
      at org.apache.solr.client.solrj.io.stream.ExceptionStream.read(ExceptionStream.java:64)
      at org.apache.solr.handler.StreamHandler$TimerStream.read(StreamHandler.java:374)
      at org.apache.solr.response.TextResponseWriter.writeTupleStream(TextResponseWriter.java:305)
      at org.apache.solr.response.TextResponseWriter.writeVal(TextResponseWriter.java:167)
      at org.apache.solr.response.JSONWriter.writeNamedListAsMapWithDups(JSONResponseWriter.java:183)
      at org.apache.solr.response.JSONWriter.writeNamedList(JSONResponseWriter.java:299)
      at org.apache.solr.response.JSONWriter.writeResponse(JSONResponseWriter.java:95)
      at org.apache.solr.response.JSONResponseWriter.write(JSONResponseWriter.java:60)
      at org.apache.solr.response.QueryResponseWriterUtil.writeQueryResponse(QueryResponseWriterUtil.java:65)
      at org.apache.solr.servlet.HttpSolrCall.writeResponse(HttpSolrCall.java:725)
      at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:469)
      at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:229)
      at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
      at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
      at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
      at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
      at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
      at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
      at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
      at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
      at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
      at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
      at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
      at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
      at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
      at org.eclipse.jetty.server.Server.handle(Server.java:518)
      at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
      at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
      at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
      at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
      at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
      at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
      at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
      at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
      at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
      at java.lang.Thread.run(Thread.java:745)

      2016-06-23 14:07:05.285 ERROR (qtp1389647288-139) [c:document6 s:shard2 r:core_node1 x:document6_shard2_replica1] o.a.s.s.HttpSolrCall null:java.lang.RuntimeException: java.lang.AbstractMethodError: Method oracle/jdbc/driver/T4CStatement.isClosed()Z is abstract
      at org.apache.solr.servlet.HttpSolrCall.sendError(HttpSolrCall.java:605)
      at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:475)
      at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:229)
      at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
      at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
      at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
      at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
      at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
      at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
      at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
      at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
      at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
      at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
      at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
      at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
      at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
      at org.eclipse.jetty.server.Server.handle(Server.java:518)
      at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
      at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
      at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
      at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
      at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
      at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
      at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
      at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
      at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
      at java.lang.Thread.run(Thread.java:745)
      Caused by: java.lang.AbstractMethodError: Method oracle/jdbc/driver/T4CStatement.isClosed()Z is abstract
      at oracle.jdbc.driver.T4CStatement.isClosed(T4CStatement.java)
      at org.apache.solr.client.solrj.io.stream.JDBCStream.close(JDBCStream.java:287)
      at org.apache.solr.client.solrj.io.stream.ExceptionStream.close(ExceptionStream.java:79)
      at org.apache.solr.handler.StreamHandler$TimerStream.close(StreamHandler.java:357)
      at org.apache.solr.response.TextResponseWriter.writeTupleStream(TextResponseWriter.java:317)
      at org.apache.solr.response.TextResponseWriter.writeVal(TextResponseWriter.java:167)
      at org.apache.solr.response.JSONWriter.writeNamedListAsMapWithDups(JSONResponseWriter.java:183)
      at org.apache.solr.response.JSONWriter.writeNamedList(JSONResponseWriter.java:299)
      at org.apache.solr.response.JSONWriter.writeResponse(JSONResponseWriter.java:95)
      at org.apache.solr.response.JSONResponseWriter.write(JSONResponseWriter.java:60)
      at org.apache.solr.response.QueryResponseWriterUtil.writeQueryResponse(QueryResponseWriterUtil.java:65)
      at org.apache.solr.servlet.HttpSolrCall.writeResponse(HttpSolrCall.java:725)
      at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:469)
      ... 26 more

      Here is the feedback from Joel Bernstein:
      =======================================
      I think we're going to have to add some debugging into the code to find what's going on. On line 225 in JDBCStream it's getting the class name for each column. It would be good know what the class names are that the Oracles driver is returning.

      https://github.com/apache/lucene-solr/blob/releases/lucene-solr/6.0.0/solr/solrj/src/java/org/apache/solr/client/solrj/io/stream/JDBCStream.java

      We probably need to throw an exception that includes the class name to help users report what different drivers using for the classes.

        Activity

        Hide
        hliu@opentext.com Hui Liu added a comment -

        Attach the original email threads.

        Show
        hliu@opentext.com Hui Liu added a comment - Attach the original email threads.
        Hide
        dpgove Dennis Gove added a comment -

        I'm gonna go ahead and post a patch with the exception.

        Show
        dpgove Dennis Gove added a comment - I'm gonna go ahead and post a patch with the exception.
        Hide
        dpgove Dennis Gove added a comment -

        On an unknown java class name an exception will now be thrown with the message

        Unable to determine the valueSelector for column '<name>' (col #<number>) of java class '<java class name>' and type '<SQL type name>'
        

        For example

        Unable to determine the valueSelector for column 'UNSP' (col #2) of java class '[B' and type 'BINARY'
        

        The weird looking java class name is because there doesn't appear to be a known java class for a BINARY type.

        Due to error handling within the JDBCStream, this exception will be caught and wrapped as the cause of an IOException. The full exception trace will look like this

        java.io.IOException: Failed to generate value selectors for sqlQuery 'select ID,UNSP from UNSUPPORTED_COLUMNS' against JDBC connection 'jdbc:hsqldb:mem:.'
          at   <stacktrace>
        Caused by: java.sql.SQLException: Unable to determine the valueSelector for column 'UNSP' (col #2) of java class '[B' and type 'BINARY'
          at <stacktrace>
        
        Show
        dpgove Dennis Gove added a comment - On an unknown java class name an exception will now be thrown with the message Unable to determine the valueSelector for column '<name>' (col #<number>) of java class '<java class name>' and type '<SQL type name>' For example Unable to determine the valueSelector for column 'UNSP' (col #2) of java class '[B' and type 'BINARY' The weird looking java class name is because there doesn't appear to be a known java class for a BINARY type. Due to error handling within the JDBCStream, this exception will be caught and wrapped as the cause of an IOException. The full exception trace will look like this java.io.IOException: Failed to generate value selectors for sqlQuery 'select ID,UNSP from UNSUPPORTED_COLUMNS' against JDBC connection 'jdbc:hsqldb:mem:.' at <stacktrace> Caused by: java.sql.SQLException: Unable to determine the valueSelector for column 'UNSP' (col #2) of java class '[B' and type 'BINARY' at <stacktrace>
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit d1a9daec8787a3f6869bdcff282dee8b3936cf24 in lucene-solr's branch refs/heads/branch_6x from Dennis Gove
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=d1a9dae ]

        SOLR-9246: If the JDBCStream sees an unknown column type it will now throw a detailed exception

        Show
        jira-bot ASF subversion and git services added a comment - Commit d1a9daec8787a3f6869bdcff282dee8b3936cf24 in lucene-solr's branch refs/heads/branch_6x from Dennis Gove [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=d1a9dae ] SOLR-9246 : If the JDBCStream sees an unknown column type it will now throw a detailed exception
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 86a19829dbd18d7fe38c0d89d7e23c25419bc935 in lucene-solr's branch refs/heads/branch_6_1 from Dennis Gove
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=86a1982 ]

        SOLR-9246: If the JDBCStream sees an unknown column type it will now throw a detailed exception

        Show
        jira-bot ASF subversion and git services added a comment - Commit 86a19829dbd18d7fe38c0d89d7e23c25419bc935 in lucene-solr's branch refs/heads/branch_6_1 from Dennis Gove [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=86a1982 ] SOLR-9246 : If the JDBCStream sees an unknown column type it will now throw a detailed exception
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit a1756f6deb379f99ada6222ddca3dd4a15dad7d3 in lucene-solr's branch refs/heads/branch_6_0 from Dennis Gove
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=a1756f6 ]

        SOLR-9246: If the JDBCStream sees an unknown column type it will now throw a detailed exception

        Show
        jira-bot ASF subversion and git services added a comment - Commit a1756f6deb379f99ada6222ddca3dd4a15dad7d3 in lucene-solr's branch refs/heads/branch_6_0 from Dennis Gove [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=a1756f6 ] SOLR-9246 : If the JDBCStream sees an unknown column type it will now throw a detailed exception
        Hide
        dpgove Dennis Gove added a comment -

        This was also committed to the master branch. I don't know why an auto note isn't being added here. https://github.com/apache/lucene-solr/commit/1ae0d8d6e1394a941b65c940cb449662d7cab5d2

        Show
        dpgove Dennis Gove added a comment - This was also committed to the master branch. I don't know why an auto note isn't being added here. https://github.com/apache/lucene-solr/commit/1ae0d8d6e1394a941b65c940cb449662d7cab5d2
        Hide
        AaronDanielson Aaron Danielson added a comment -

        The 'Caused by: ...' portion of the error message was never printed for me. My column was using type DECIMAL, but JDBC only understands FLOAT or DOUBLE, so this took some serious time to figure out the issue without a more explicit error message. For what it's worth, it would be really nice if the driver could automatically map column types of DECIMAL to FLOAT.

        Show
        AaronDanielson Aaron Danielson added a comment - The 'Caused by: ...' portion of the error message was never printed for me. My column was using type DECIMAL, but JDBC only understands FLOAT or DOUBLE, so this took some serious time to figure out the issue without a more explicit error message. For what it's worth, it would be really nice if the driver could automatically map column types of DECIMAL to FLOAT.

          People

          • Assignee:
            dpgove Dennis Gove
            Reporter:
            hliu@opentext.com Hui Liu
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development