Solr
  1. Solr
  2. SOLR-6165

DataImportHandler writes BigInteger and BigDecimal as-is which causes errors in SolrCloud replication

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 4.8
    • Fix Version/s: 4.10, 6.0
    • Component/s: None
    • Labels:
      None

      Description

      we are trying to migrate to 4.8 from 4.1, after setting up the new solr cloud when we try to do a dataimport using the DataimportHandler we have issues from replication. following is the error we are getting in the log. The field from the db is numeric field and in solr schema file its been declared as double.

      603215 [qtp280884709-15] INFO org.apache.solr.update.processor.LogUpdateProcessor ? [locations] webapp=/solr path=/update params=

      {update.distrib=FROMLEADER&distrib.from=http://servername:8983/solr/locations/&wt=javabin&version=2}

      {} 0 0
      603216 [qtp280884709-15] ERROR org.apache.solr.core.SolrCore ? org.apache.solr.common.SolrException: ERROR: [doc=SALT LAKE CITY-UT-84127] Error adding field 'city_lat'='java.math.BigDecimal:40.7607793000' msg=For input string: "java.math.BigDecimal:40.7607793000"
      at org.apache.solr.update.DocumentBuilder.toDocument(DocumentBuilder.java:167)
      at org.apache.solr.update.AddUpdateCommand.getLuceneDocument(AddUpdateCommand.java:77)
      at org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:234)
      at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:160)
      at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:69)
      at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:51)
      at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:703)
      at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:857)
      at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:556)
      at org.apache.solr.update.processor.LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:100)
      at org.apache.solr.handler.loader.JavabinLoader$1.update(JavabinLoader.java:96)
      at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:166)
      at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readIterator(JavaBinUpdateRequestCodec.java:136)
      at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:225)
      at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readNamedList(JavaBinUpdateRequestCodec.java:121)
      at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:190)
      at org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:116)
      at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.unmarshal(JavaBinUpdateRequestCodec.java:173)
      at org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:106)
      at org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:58)
      at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
      at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
      at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
      at org.apache.solr.core.SolrCore.execute(SolrCore.java:1916)
      at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:780)
      at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:427)
      at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:217)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
      at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
      at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
      at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
      at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
      at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
      at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
      at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
      at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
      at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
      at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
      at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
      at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
      at org.eclipse.jetty.server.Server.handle(Server.java:368)
      at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
      at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
      at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953)
      at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014)
      at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:953)
      at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
      at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
      at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
      at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
      at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
      at java.lang.Thread.run(Thread.java:745)
      Caused by: java.lang.NumberFormatException: For input string: "java.math.BigDecimal:40.7607793000"
      at sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:2043)
      at sun.misc.FloatingDecimal.parseDouble(FloatingDecimal.java:110)
      at java.lang.Double.parseDouble(Double.java:538)
      at org.apache.solr.schema.TrieField.createField(TrieField.java:615)
      at org.apache.solr.schema.TrieField.createFields(TrieField.java:660)
      at org.apache.solr.update.DocumentBuilder.addField(DocumentBuilder.java:47)
      at org.apache.solr.update.DocumentBuilder.toDocument(DocumentBuilder.java:118)
      ... 51 more

        Activity

        Hide
        anand sengamalai added a comment -

        our setup works fine with solr 4.1, the same query and schema does not work in 4.8, i get the above mentioned exception in the cloud setup.

        How ever i downloaded the source for solr 4.8 and modified the org.apache.solr.handler.dataimport.JdbcDataSource and modified the fuction private Map<String, Object> getARow() ;
        for BigDecimal and BigInteger i return them as String and its working fine. Just want o submit this as a workaround for people migrating like us.

        if (!convertType) {
        // Use underlying database's type information
        //since dataimport does not want to support bigdecimal and biginteger
        if((resultSet.getObject(colName) instanceof java.math.BigDecimal)||
        (resultSet.getObject(colName) instanceof java.math.BigInteger))

        { result.put(colName,resultSet.getObject(colName).toString()); }

        else

        { result.put(colName, resultSet.getObject(colName)); }

        continue;
        }

        Show
        anand sengamalai added a comment - our setup works fine with solr 4.1, the same query and schema does not work in 4.8, i get the above mentioned exception in the cloud setup. How ever i downloaded the source for solr 4.8 and modified the org.apache.solr.handler.dataimport.JdbcDataSource and modified the fuction private Map<String, Object> getARow() ; for BigDecimal and BigInteger i return them as String and its working fine. Just want o submit this as a workaround for people migrating like us. if (!convertType) { // Use underlying database's type information //since dataimport does not want to support bigdecimal and biginteger if((resultSet.getObject(colName) instanceof java.math.BigDecimal)|| (resultSet.getObject(colName) instanceof java.math.BigInteger)) { result.put(colName,resultSet.getObject(colName).toString()); } else { result.put(colName, resultSet.getObject(colName)); } continue; }
        Hide
        ASF subversion and git services added a comment -

        Commit 1604543 from shalin@apache.org in branch 'dev/trunk'
        [ https://svn.apache.org/r1604543 ]

        SOLR-6165: DataImportHandler should write BigInteger and BigDecimal values as strings

        Show
        ASF subversion and git services added a comment - Commit 1604543 from shalin@apache.org in branch 'dev/trunk' [ https://svn.apache.org/r1604543 ] SOLR-6165 : DataImportHandler should write BigInteger and BigDecimal values as strings
        Hide
        ASF subversion and git services added a comment -

        Commit 1604544 from shalin@apache.org in branch 'dev/branches/branch_4x'
        [ https://svn.apache.org/r1604544 ]

        SOLR-6165: DataImportHandler should write BigInteger and BigDecimal values as strings

        Show
        ASF subversion and git services added a comment - Commit 1604544 from shalin@apache.org in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1604544 ] SOLR-6165 : DataImportHandler should write BigInteger and BigDecimal values as strings
        Hide
        Shalin Shekhar Mangar added a comment -

        Thanks Anand!

        Show
        Shalin Shekhar Mangar added a comment - Thanks Anand!
        Hide
        ASF subversion and git services added a comment -

        Commit 1611985 from Erik Hatcher in branch 'dev/trunk'
        [ https://svn.apache.org/r1611985 ]

        SOLR-6165: Add tests for convertType and BigDecimal implicit conversion

        Show
        ASF subversion and git services added a comment - Commit 1611985 from Erik Hatcher in branch 'dev/trunk' [ https://svn.apache.org/r1611985 ] SOLR-6165 : Add tests for convertType and BigDecimal implicit conversion
        Hide
        ASF subversion and git services added a comment -

        Commit 1611987 from Erik Hatcher in branch 'dev/branches/branch_4x'
        [ https://svn.apache.org/r1611987 ]

        SOLR-6165: Add tests for convertType and BigDecimal implicit conversion

        Show
        ASF subversion and git services added a comment - Commit 1611987 from Erik Hatcher in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1611987 ] SOLR-6165 : Add tests for convertType and BigDecimal implicit conversion

          People

          • Assignee:
            Shalin Shekhar Mangar
            Reporter:
            anand sengamalai
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development