[SOLR-12283] Unable To Load ZKPropertiesWriter when dih.jar is added as runtimelib BLOB in .system collection - ASF JIRA

Details

Type: Test
Status: Resolved
Priority: Major
Resolution: Won't Fix
Affects Version/s: 6.6.1, 7.3
Fix Version/s: None
Component/s: contrib - DataImportHandler
Labels:
None
Environment:

Debian
SolrCloud

Description

Hello,

It's been 2 weeks that I try to correct this problem with the community user-solr but no success. I seriously wonder if this is not a problem in the code. I do not have the impression that many people use DIH with Solr's cloud version.
On Internet, no similar problem.

For information, the following configuration of DIH comes from DIHs that work in production on a single Solr server. The connections to the databases are therefore correct.

Errors messages:

DataImporter

Full Import failed:org.apache.solr.handler.dataimport.DataImportHandlerException: Unable to PropertyWriter implementation:ZKPropertiesWriter
	at org.apache.solr.handler.dataimport.DataImporter.createPropertyWriter(DataImporter.java:339)
	at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:420)
	at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:483)
	at org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(DataImportHandler.java:183)
	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:195)
	at org.apache.solr.core.SolrCore.execute(SolrCore.java:2503)
	at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:711)
	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:517)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:384)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:330)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1629)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:219)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.server.Server.handle(Server.java:530)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:256)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
	at org.apache.solr.handler.dataimport.DocBuilder.loadClass(DocBuilder.java:935)
	at org.apache.solr.handler.dataimport.DataImporter.createPropertyWriter(DataImporter.java:335)
	... 42 more

RequestHandlerBase

java.lang.NullPointerException
	at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:429)
	at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:483)
	at org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(DataImportHandler.java:183)
	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:195)
	at org.apache.solr.core.SolrCore.execute(SolrCore.java:2503)
	at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:711)
	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:517)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:384)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:330)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1629)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:219)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.server.Server.handle(Server.java:530)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:256)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
	at java.lang.Thread.run(Thread.java:748)

HttpSolrCall

null:java.lang.NullPointerException
	at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:429)
	at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:483)
	at org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(DataImportHandler.java:183)
	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:195)
	at org.apache.solr.core.SolrCore.execute(SolrCore.java:2503)
	at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:711)
	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:517)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:384)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:330)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1629)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:219)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.server.Server.handle(Server.java:530)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:256)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
	at java.lang.Thread.run(Thread.java:748)

Process to reproduce the problem:

Create .system collection for the BLOB

curl http://localhost:8983/solr/admin/collections?action=CREATE&name=.system

Add Data Import Handler JAR file.

curl -X POST -H 'Content-Type: application/octet-stream' --data-binary @solr-dataimporthandler-7.3.0.jar http://localhost:8983/solr/.system/blob/data-import-handler

Add the runtime lib on the configoverlay.json with the API

curl http://localhost:8983/solr/unicorn/config -H 'Content-type:application/json' -d '{"add-runtimelib": { "name":"data-import-handler", "version":1 }}'

Send mysql connector on the blob. I have test with the connector-bin.jar but I have the same error.

curl -X POST -H 'Content-Type: application/octet-stream' --data-binary @mysql-connector-java-5.1.46.jar http://localhost:8983/solr/.system/blob/mysql-connector-java

Add runtime

curl http://localhost:8983/solr/unicorn/config -H 'Content-type:application/json' -d '{"add-runtimelib": { "name":"mysql-connector-java", "version":1 }}'

Personally, I create a DIH folder on the collection config path on the cloud. It's not necessary.

And I put the DIH config file on the cloud with the zkcli (it's a personnal script to overload the zkcli.sh on Solr but it's the same result. My file is on the cloud.)

./zkcli.py putfile /configs/unicorn/DIH/ DIH/indexation_events.xml

Add request handler config:

curl -X POST http://{solr_server}:8983/solr/{collection_name}/config -H 'Content-type:application/json' -d '
{
  "add-requesthandler":
  {
    "name": "/test-dih",
    "class": "org.apache.solr.handler.dataimport.DataImportHandler",
    "runtimeLib": true,
    "version": 1,
    "defaults":
    {
      "config": "DIH/indexation_events.xml"
    }
  }
} '

Reload the config:

curl http://localhost:8983/solr/unicorn/test-dih?command=reload-config&commit=true&debug=true

Import and have error:

curl http://localhost:8983/solr/unicorn/test-dih?command=full-import&commit=true&debug=true

I have test to add on request handler and BLOB this jar files but not change:

data-import-handler-extras
solr-core
solr-solrj

I have join all files used in this process.

Thansk you.

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending

modified-DIH.zip
19/May/18 19:18
218 kB
Shawn Heisey
modified-DIH.zip
27/Apr/18 16:14
218 kB
Shawn Heisey
mysql-connector-java-5.1.46.jar
27/Apr/18 09:19
981 kB
Maxence SAUNIER
mysql-connector-java-5.1.46-bin.jar
27/Apr/18 09:19
981 kB
Maxence SAUNIER
request_handler_config.json
27/Apr/18 09:19
0.2 kB
Maxence SAUNIER
solr.log
05/Jun/18 12:19
89 kB
Maxence SAUNIER
solr.log
30/Apr/18 10:11
78 kB
Maxence SAUNIER
solr.log
27/Apr/18 16:36
7.11 MB
Maxence SAUNIER
solr.log
27/Apr/18 15:02
6.68 MB
Maxence SAUNIER
solr-core-7.3.0.jar
27/Apr/18 09:19
4.96 MB
Maxence SAUNIER
solr-dataimporthandler-7.3.0.jar
27/Apr/18 09:19
223 kB
Maxence SAUNIER
solr-dataimporthandler-extras-7.3.0.jar
27/Apr/18 09:19
39 kB
Maxence SAUNIER
solr-solrj-7.3.0.jar
27/Apr/18 09:19
1.66 MB
Maxence SAUNIER

Issue Links

is superceded by

SOLR-14783 Remove DIH from 9.0

Closed

Unable To Load ZKPropertiesWriter when dih.jar is added as runtimelib BLOB in .system collection

Details

Description

Attachments

Attachments

Issue Links

Activity

People

Dates