Solr
  1. Solr
  2. SOLR-3165

Cannot use DIH in Solrcloud + Zookeeper

    Details

      Description

      There is a problem with configure DIH in Solrcloud + Zookeeper configuration from wiki:
      http://wiki.apache.org/solr/SolrCloud.

      Standard configuration in solrconfig.xml:

       
        <requestHandler name="/dataimport" class="org.apache.solr.handler.dataimport.DataImportHandler">
          <lst name="defaults">
             <str name="config">db-data-config.xml</str>
          </lst>
        </requestHandler>
      

      After starting solr with zookeeper I've got errors:

      Feb 20, 2012 11:30:12 AM org.apache.solr.common.SolrException log
      SEVERE: null:org.apache.solr.common.SolrException
              at org.apache.solr.core.SolrCore.<init>(SolrCore.java:606)
              at org.apache.solr.core.SolrCore.<init>(SolrCore.java:490)
              at org.apache.solr.core.CoreContainer.create(CoreContainer.java:705)
              at org.apache.solr.core.CoreContainer.load(CoreContainer.java:442)
              at org.apache.solr.core.CoreContainer.load(CoreContainer.java:313)
              at org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:262)
              at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:98)
              at org.mortbay.jetty.servlet.FilterHolder.doStart(FilterHolder.java:97)
              at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
              at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:713)
              at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
              at org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1282)
              at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:518)
              at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:499)
              at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
              at org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152)
              at org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156)
              at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
              at org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152)
              at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
              at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
              at org.mortbay.jetty.Server.doStart(Server.java:224)
              at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
              at org.mortbay.xml.XmlConfiguration.main(XmlConfiguration.java:985)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
              at java.lang.reflect.Method.invoke(Method.java:597)
              at org.mortbay.start.Main.invokeMain(Main.java:194)
              at org.mortbay.start.Main.start(Main.java:534)
              at org.mortbay.start.Main.start(Main.java:441)
              at org.mortbay.start.Main.main(Main.java:119)
      Caused by: org.apache.solr.common.SolrException: FATAL: Could not create importer. DataImporter config invalid
              at org.apache.solr.handler.dataimport.DataImportHandler.inform(DataImportHandler.java:120)
              at org.apache.solr.core.SolrResourceLoader.inform(SolrResourceLoader.java:542)
              at org.apache.solr.core.SolrCore.<init>(SolrCore.java:601)
              ... 31 more
      Caused by: org.apache.solr.common.cloud.ZooKeeperException: ZkSolrResourceLoader does not support getConfigDir() - likely, w
              at org.apache.solr.cloud.ZkSolrResourceLoader.getConfigDir(ZkSolrResourceLoader.java:99)
              at org.apache.solr.handler.dataimport.SimplePropertiesWriter.init(SimplePropertiesWriter.java:47)
              at org.apache.solr.handler.dataimport.DataImporter.<init>(DataImporter.java:112)
              at org.apache.solr.handler.dataimport.DataImportHandler.inform(DataImportHandler.java:114)
              ... 33 more
      

      But the db-data-config.xml file exists in Zookeeper:

      [zk: localhost:2181(CONNECTED) 0] ls /configs/conf1 [admin-extra.menu-top.html, dict, solrconfig.xml, dataimport.properties, admin-extra.html, solrconfig.xml.old, solrconfig.xml.new, solrconfig.xml~, xslt, db-data-config.xml, velocity, elevate.xml, admin-extra.menu-bottom.html, solrconfig.xml.dataimport, schema.xml]
      [zk: localhost:2181(CONNECTED) 1]
      

      I downloaded solr from trunk (2012-02-16)

      1. SOLR-3165.patch
        7 kB
        Alexey Serba
      2. SOLR-3165.patch
        7 kB
        Sami Siren

        Activity

        Hide
        Sami Siren added a comment -

        Here's a stab at fixing this issue. I noticed there were two places where DIH wanted to access conf dir: the config file (read) and some kind of state file (read/write) access.

        I added ZKPropertiesWriter that writes the contents of the state file into zk.

        Show
        Sami Siren added a comment - Here's a stab at fixing this issue. I noticed there were two places where DIH wanted to access conf dir: the config file (read) and some kind of state file (read/write) access. I added ZKPropertiesWriter that writes the contents of the state file into zk.
        Hide
        Mark Miller added a comment -

        Nice Sami! I was going to take the same approach.

        Longer term I think we want to add a write api to solrresource loader. If something came from the classpath, it would still be written to the conf area and later that would override a new read...

        Show
        Mark Miller added a comment - Nice Sami! I was going to take the same approach. Longer term I think we want to add a write api to solrresource loader. If something came from the classpath, it would still be written to the conf area and later that would override a new read...
        Hide
        Mark Miller added a comment -

        Took a look at the patch and it looks good to me - only thing I might suggest is using CoreContainer#isZookeeperAware rather than comparing the zkController to null, but pretty minor stylistic thing.

        Show
        Mark Miller added a comment - Took a look at the patch and it looks good to me - only thing I might suggest is using CoreContainer#isZookeeperAware rather than comparing the zkController to null, but pretty minor stylistic thing.
        Hide
        Alexey Serba added a comment -

        I updated Sami's patch to store dataimport property file under collection directory in zk and name it after data import handler (the same way it is stored in local filesystem)

        Show
        Alexey Serba added a comment - I updated Sami's patch to store dataimport property file under collection directory in zk and name it after data import handler (the same way it is stored in local filesystem)
        Hide
        Sami Siren added a comment -

        Thanks Alexey, Mark. I took the latest patch from Alexey, incorporated Marks suggestion and committed this to trunk.

        Show
        Sami Siren added a comment - Thanks Alexey, Mark. I took the latest patch from Alexey, incorporated Marks suggestion and committed this to trunk.

          People

          • Assignee:
            Sami Siren
            Reporter:
            Agnieszka
          • Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development