Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-13414

SolrSchema - Avoid NPE if Luke returns field with no type defined

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 7.3, 7.7.1
    • Fix Version/s: 7.7.2, 8.1, master (9.0)
    • Component/s: Parallel SQL
    • Labels:
      None

      Description

      Summary
      If the underlying Lucene index has fields defined but no type, SolrSchema fails with NPE. The index most likely has issues and would be better to delete/recreate the index. This ticket adds a null check to prevent the NPE and won't break on a potentially invalid index.

      Initial Description
      When attempting to create a JDBC sql query against a large collection (400m + records) we get a null error.

      After initial discussion in solr-user I have been asked to open this ticket - The exception thrown does not provide sufficient detail to understand the underlying problem. Its it thought to be an issue with the schema not initialising correctly. 

      Attached is the managed-schema after a downconfig.

      Stack trace from email thread:

      Solr Admin UI Logging

      java.io.IOException: Failed to execute sqlQuery 'select id from document limit 10' against JDBC connection 'jdbc:calcitesolr:'.
      Error while executing SQL "select id from document limit 10": null
              at org.apache.solr.client.solrj.io.stream.JDBCStream.open(JDBCStream.java:271)
              at org.apache.solr.client.solrj.io.stream.ExceptionStream.open(ExceptionStream.java:54)
              at org.apache.solr.handler.StreamHandler$TimerStream.open(StreamHandler.java:394)
              at org.apache.solr.client.solrj.io.stream.TupleStream.writeMap(TupleStream.java:78)
              at org.apache.solr.common.util.JsonTextWriter.writeMap(JsonTextWriter.java:164)
              at org.apache.solr.common.util.TextWriter.writeVal(TextWriter.java:69)
              at org.apache.solr.response.TextResponseWriter.writeVal(TextResponseWriter.java:152)
              at org.apache.solr.common.util.JsonTextWriter.writeNamedListAsMapWithDups(JsonTextWriter.java:386)
              at org.apache.solr.common.util.JsonTextWriter.writeNamedList(JsonTextWriter.java:292)
              at org.apache.solr.response.JSONWriter.writeResponse(JSONWriter.java:73)
              at org.apache.solr.response.JSONResponseWriter.write(JSONResponseWriter.java:66)
              at org.apache.solr.response.QueryResponseWriterUtil.writeQueryResponse(QueryResponseWriterUtil.java:65)
              at org.apache.solr.servlet.HttpSolrCall.writeResponse(HttpSolrCall.java:788)
              at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:525)
              at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:395)
              at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:341)
              at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
              at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
              at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
              at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
              at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
              at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
              at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)
              at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
              at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
              at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
              at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
              at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)
              at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
              at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
              at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
              at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
              at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
              at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
              at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
              at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
              at org.eclipse.jetty.server.Server.handle(Server.java:502)
              at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
              at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
              at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
              at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
              at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
              at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
              at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
              at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
              at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
              at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
              at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
              at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
              at java.lang.Thread.run(Unknown Source)
      Caused by: java.sql.SQLException: Error while executing SQL "select id from document limit 10": null
              at org.apache.calcite.avatica.Helper.createException(Helper.java:56)
              at org.apache.calcite.avatica.Helper.createException(Helper.java:41)
              at org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156)
              at org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:218)
              at org.apache.solr.client.solrj.io.stream.JDBCStream.open(JDBCStream.java:269)
              ... 49 more
      Caused by: java.lang.NullPointerException
              at org.apache.calcite.plan.volcano.VolcanoPlanner.validate(VolcanoPlanner.java:891)
              at org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:866)
              at org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:883)
              at org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:101)
              at org.apache.calcite.rel.AbstractRelNode.onRegister(AbstractRelNode.java:336)
              at org.apache.calcite.plan.volcano.VolcanoPlanner.registerImpl(VolcanoPlanner.java:1496)
              at org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:863)
              at org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:883)
              at org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:101)
              at org.apache.calcite.rel.AbstractRelNode.onRegister(AbstractRelNode.java:336)
              at org.apache.calcite.plan.volcano.VolcanoPlanner.registerImpl(VolcanoPlanner.java:1496)
              at org.apache.calcite.plan.volcano.VolcanoPlanner.setRoot(VolcanoPlanner.java:308)
              at org.apache.calcite.tools.Programs$5.run(Programs.java:309)
              at org.apache.calcite.tools.Programs$SequenceProgram.run(Programs.java:387)
              at org.apache.calcite.prepare.Prepare.optimize(Prepare.java:186)
              at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:319)
              at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:228)
              at org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:784)
              at org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:639)
              at org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:609)
              at org.apache.calcite.jdbc.CalciteConnectionImpl.parseQuery(CalciteConnectionImpl.java:214)
              at org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:603)
              at org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:638)
              at org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:149)
      

      solr.log

      Error while executing SQL "select id from data limit 10": null
      
      at
      org.apache.solr.client.solrj.io.stream.JDBCStream.open(JDBCStream.java:271)
      at
      org.apache.solr.client.solrj.io.stream.ExceptionStream.open(ExceptionStream.java:54)
      at
      org.apache.solr.handler.StreamHandler$TimerStream.open(StreamHandler.java:394)
      at
      org.apache.solr.client.solrj.io.stream.TupleStream.writeMap(TupleStream.java:78)
      at
      org.apache.solr.common.util.JsonTextWriter.writeMap(JsonTextWriter.java:164)
      at org.apache.solr.common.util.TextWriter.writeVal(TextWriter.java:69)
      at
      org.apache.solr.response.TextResponseWriter.writeVal(TextResponseWriter.java:152)
      at
      org.apache.solr.common.util.JsonTextWriter.writeNamedListAsMapWithDups(JsonTextWriter.java:386)
      at
      org.apache.solr.common.util.JsonTextWriter.writeNamedList(JsonTextWriter.java:292)
      at org.apache.solr.response.JSONWriter.writeResponse(JSONWriter.java:73)
      at
      org.apache.solr.response.JSONResponseWriter.write(JSONResponseWriter.java:66)
      at
      org.apache.solr.response.QueryResponseWriterUtil.writeQueryResponse(QueryResponseWriterUtil.java:65)
      at org.apache.solr.servlet.HttpSolrCall.writeResponse(HttpSolrCall.java:788)
      at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:525)
      at
      org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:395)
      at
      org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:341)
      at
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
      at
      org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
      at
      org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
      at
      org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
      at
      org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
      at
      org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
      at
      org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)
      at
      org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
      at
      org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
      at
      org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
      at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
      at
      org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)
      at
      org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
      at
      org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
      at
      org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
      at
      org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
      at
      org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
      at
      org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
      at
      org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
      at
      org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
      at org.eclipse.jetty.server.Server.handle(Server.java:502)
      at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
      at
      org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
      at
      org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
      at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
      at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
      at
      org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
      at
      org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
      at
      org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
      at
      org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
      at
      org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
      at
      org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
      at java.lang.Thread.run(Unknown Source)
      Caused by: java.sql.SQLException: Error while executing SQL "select id from
      data limit 10": null
      at org.apache.calcite.avatica.Helper.createException(Helper.java:56)
      at org.apache.calcite.avatica.Helper.createException(Helper.java:41)
      at
      org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156)
      at
      org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:218)
      at
      org.apache.solr.client.solrj.io.stream.JDBCStream.open(JDBCStream.java:269)
      ... 48 more
      Caused by: java.lang.NullPointerException
      at
      org.apache.solr.handler.sql.SolrSchema.getRelDataType(SolrSchema.java:103)
      at org.apache.solr.handler.sql.SolrTable.getRowType(SolrTable.java:83)
      at org.apache.calcite.sql.validate.EmptyScope.resolve_(EmptyScope.java:159)
      at
      org.apache.calcite.sql.validate.EmptyScope.resolveTable(EmptyScope.java:99)
      at
      org.apache.calcite.sql.validate.DelegatingScope.resolveTable(DelegatingScope.java:202)
      at
      org.apache.calcite.sql.validate.IdentifierNamespace.resolveImpl(IdentifierNamespace.java:104)
      at
      org.apache.calcite.sql.validate.IdentifierNamespace.validateImpl(IdentifierNamespace.java:167)
      at
      org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84)
      at
      org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:939)
      at
      org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:920)
      at
      org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:2954)
      at
      org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:2939)
      at
      org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect(SqlValidatorImpl.java:3181)
      at
      org.apache.calcite.sql.validate.SelectNamespace.validateImpl(SelectNamespace.java:60)
      at
      org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84)
      at
      org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:939)
      at
      org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:920)
      at org.apache.calcite.sql.SqlSelect.validate(SqlSelect.java:220)
      at
      org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:895)
      at
      org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:605)
      at
      org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:550)
      at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:264)
      at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:228)
      at
      org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:784)
      at
      org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:639)
      at
      org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:609)
      at
      org.apache.calcite.jdbc.CalciteConnectionImpl.parseQuery(CalciteConnectionImpl.java:214)
      at
      org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:603)
      at
      org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:638)
      at
      org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:149)
      ... 50 more
      

        Attachments

        1. before_starting_solr.png
          547 kB
          David Barnett
        2. command_prompt.png
          50 kB
          David Barnett
        3. luke_out.xml
          26 kB
          David Barnett
        4. managed-schema
          56 kB
          David Barnett
        5. new_solr.log
          172 kB
          David Barnett
        6. new_solr-8983-console.log
          2 kB
          David Barnett
        7. solr.log
          210 kB
          David Barnett
        8. SOLR-13414.patch
          3 kB
          Kevin Risden
        9. SOLR-13414.patch
          0.9 kB
          Joel Bernstein
        10. solr-8983-console.log
          5 kB
          David Barnett
        11. solr-8983-console.log
          2 kB
          David Barnett
        12. solr-core-7.8.0-SNAPSHOT.jar
          5.27 MB
          Joel Bernstein

          Issue Links

            Activity

              People

              • Assignee:
                krisden Kevin Risden
                Reporter:
                davebarnett David Barnett
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 0.5h
                  0.5h