Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-6165

DataImportHandler writes BigInteger and BigDecimal as-is which causes errors in SolrCloud replication

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 4.8
    • Fix Version/s: 4.10, 6.0
    • Component/s: None
    • Labels:
      None

      Description

      we are trying to migrate to 4.8 from 4.1, after setting up the new solr cloud when we try to do a dataimport using the DataimportHandler we have issues from replication. following is the error we are getting in the log. The field from the db is numeric field and in solr schema file its been declared as double.

      603215 [qtp280884709-15] INFO org.apache.solr.update.processor.LogUpdateProcessor ? [locations] webapp=/solr path=/update params=

      {update.distrib=FROMLEADER&distrib.from=http://servername:8983/solr/locations/&wt=javabin&version=2}

      {} 0 0
      603216 [qtp280884709-15] ERROR org.apache.solr.core.SolrCore ? org.apache.solr.common.SolrException: ERROR: [doc=SALT LAKE CITY-UT-84127] Error adding field 'city_lat'='java.math.BigDecimal:40.7607793000' msg=For input string: "java.math.BigDecimal:40.7607793000"
      at org.apache.solr.update.DocumentBuilder.toDocument(DocumentBuilder.java:167)
      at org.apache.solr.update.AddUpdateCommand.getLuceneDocument(AddUpdateCommand.java:77)
      at org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:234)
      at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:160)
      at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:69)
      at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:51)
      at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:703)
      at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:857)
      at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:556)
      at org.apache.solr.update.processor.LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:100)
      at org.apache.solr.handler.loader.JavabinLoader$1.update(JavabinLoader.java:96)
      at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:166)
      at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readIterator(JavaBinUpdateRequestCodec.java:136)
      at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:225)
      at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readNamedList(JavaBinUpdateRequestCodec.java:121)
      at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:190)
      at org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:116)
      at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.unmarshal(JavaBinUpdateRequestCodec.java:173)
      at org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:106)
      at org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:58)
      at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
      at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
      at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
      at org.apache.solr.core.SolrCore.execute(SolrCore.java:1916)
      at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:780)
      at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:427)
      at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:217)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
      at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
      at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
      at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
      at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
      at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
      at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
      at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
      at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
      at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
      at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
      at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
      at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
      at org.eclipse.jetty.server.Server.handle(Server.java:368)
      at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
      at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
      at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953)
      at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014)
      at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:953)
      at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
      at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
      at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
      at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
      at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
      at java.lang.Thread.run(Thread.java:745)
      Caused by: java.lang.NumberFormatException: For input string: "java.math.BigDecimal:40.7607793000"
      at sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:2043)
      at sun.misc.FloatingDecimal.parseDouble(FloatingDecimal.java:110)
      at java.lang.Double.parseDouble(Double.java:538)
      at org.apache.solr.schema.TrieField.createField(TrieField.java:615)
      at org.apache.solr.schema.TrieField.createFields(TrieField.java:660)
      at org.apache.solr.update.DocumentBuilder.addField(DocumentBuilder.java:47)
      at org.apache.solr.update.DocumentBuilder.toDocument(DocumentBuilder.java:118)
      ... 51 more

        Activity

        Hide
        asengamalai anand sengamalai added a comment -

        our setup works fine with solr 4.1, the same query and schema does not work in 4.8, i get the above mentioned exception in the cloud setup.

        How ever i downloaded the source for solr 4.8 and modified the org.apache.solr.handler.dataimport.JdbcDataSource and modified the fuction private Map<String, Object> getARow() ;
        for BigDecimal and BigInteger i return them as String and its working fine. Just want o submit this as a workaround for people migrating like us.

        if (!convertType) {
        // Use underlying database's type information
        //since dataimport does not want to support bigdecimal and biginteger
        if((resultSet.getObject(colName) instanceof java.math.BigDecimal)||
        (resultSet.getObject(colName) instanceof java.math.BigInteger))

        { result.put(colName,resultSet.getObject(colName).toString()); }

        else

        { result.put(colName, resultSet.getObject(colName)); }

        continue;
        }

        Show
        asengamalai anand sengamalai added a comment - our setup works fine with solr 4.1, the same query and schema does not work in 4.8, i get the above mentioned exception in the cloud setup. How ever i downloaded the source for solr 4.8 and modified the org.apache.solr.handler.dataimport.JdbcDataSource and modified the fuction private Map<String, Object> getARow() ; for BigDecimal and BigInteger i return them as String and its working fine. Just want o submit this as a workaround for people migrating like us. if (!convertType) { // Use underlying database's type information //since dataimport does not want to support bigdecimal and biginteger if((resultSet.getObject(colName) instanceof java.math.BigDecimal)|| (resultSet.getObject(colName) instanceof java.math.BigInteger)) { result.put(colName,resultSet.getObject(colName).toString()); } else { result.put(colName, resultSet.getObject(colName)); } continue; }
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 1604543 from shalin@apache.org in branch 'dev/trunk'
        [ https://svn.apache.org/r1604543 ]

        SOLR-6165: DataImportHandler should write BigInteger and BigDecimal values as strings

        Show
        jira-bot ASF subversion and git services added a comment - Commit 1604543 from shalin@apache.org in branch 'dev/trunk' [ https://svn.apache.org/r1604543 ] SOLR-6165 : DataImportHandler should write BigInteger and BigDecimal values as strings
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 1604544 from shalin@apache.org in branch 'dev/branches/branch_4x'
        [ https://svn.apache.org/r1604544 ]

        SOLR-6165: DataImportHandler should write BigInteger and BigDecimal values as strings

        Show
        jira-bot ASF subversion and git services added a comment - Commit 1604544 from shalin@apache.org in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1604544 ] SOLR-6165 : DataImportHandler should write BigInteger and BigDecimal values as strings
        Hide
        shalinmangar Shalin Shekhar Mangar added a comment -

        Thanks Anand!

        Show
        shalinmangar Shalin Shekhar Mangar added a comment - Thanks Anand!
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 1611985 from Erik Hatcher in branch 'dev/trunk'
        [ https://svn.apache.org/r1611985 ]

        SOLR-6165: Add tests for convertType and BigDecimal implicit conversion

        Show
        jira-bot ASF subversion and git services added a comment - Commit 1611985 from Erik Hatcher in branch 'dev/trunk' [ https://svn.apache.org/r1611985 ] SOLR-6165 : Add tests for convertType and BigDecimal implicit conversion
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 1611987 from Erik Hatcher in branch 'dev/branches/branch_4x'
        [ https://svn.apache.org/r1611987 ]

        SOLR-6165: Add tests for convertType and BigDecimal implicit conversion

        Show
        jira-bot ASF subversion and git services added a comment - Commit 1611987 from Erik Hatcher in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1611987 ] SOLR-6165 : Add tests for convertType and BigDecimal implicit conversion

          People

          • Assignee:
            shalinmangar Shalin Shekhar Mangar
            Reporter:
            asengamalai anand sengamalai
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development