Solr
  1. Solr
  2. SOLR-4112

Dataimporting with SolrCloud Fails

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 5.0
    • Fix Version/s: 4.1, 5.0
    • Component/s: None
    • Labels:
      None

      Description

      While trying to import data from db on cloud, it shows this in logs:

      SEVERE: Full Import failed:org.apache.solr.handler.dataimport.DataImportHandlerException: Unable to PropertyWriter implementation:ZKPropertiesWriter
      at org.apache.solr.handler.dataimport.DataImporter.createPropertyWriter(DataImporter.java:336)
      at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:418)
      at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:487)
      at org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:468)
      Caused by: org.apache.solr.common.cloud.ZooKeeperException: ZkSolrResourceLoader does not support getConfigDir() - likely, what you are trying to do is not supported in ZooKeeper mode
      at org.apache.solr.cloud.ZkSolrResourceLoader.getConfigDir(ZkSolrResourceLoader.java:100)
      at org.apache.solr.handler.dataimport.SimplePropertiesWriter.init(SimplePropertiesWriter.java:91)
      at org.apache.solr.handler.dataimport.ZKPropertiesWriter.init(ZKPropertiesWriter.java:45)
      at org.apache.solr.handler.dataimport.DataImporter.createPropertyWriter(DataImporter.java:334)
      ... 3 more

      Exception in thread "Thread-306" java.lang.NullPointerException
      at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:427)
      at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:487)
      at org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:468)

      1. SOLR-4112.patch
        13 kB
        Shalin Shekhar Mangar
      2. SOLR-4112.patch
        3 kB
        James Dyer

        Activity

        Hide
        Commit Tag Bot added a comment -

        [branch_4x commit] Shalin Shekhar Mangar
        http://svn.apache.org/viewvc?view=revision&revision=1432048

        SOLR-4112: Fixed DataImportHandler ZKAwarePropertiesWriter implementation so import works fine with SolrCloud clusters

        Show
        Commit Tag Bot added a comment - [branch_4x commit] Shalin Shekhar Mangar http://svn.apache.org/viewvc?view=revision&revision=1432048 SOLR-4112 : Fixed DataImportHandler ZKAwarePropertiesWriter implementation so import works fine with SolrCloud clusters
        Hide
        Shalin Shekhar Mangar added a comment -

        Fixed in trunk and branch_4x.

        Thanks for taking care of the extra sysout line Erick.

        Show
        Shalin Shekhar Mangar added a comment - Fixed in trunk and branch_4x. Thanks for taking care of the extra sysout line Erick.
        Hide
        Shalin Shekhar Mangar added a comment -

        Oops, I committed it already with a change log entry in both trunk and branch_4x.

        Show
        Shalin Shekhar Mangar added a comment - Oops, I committed it already with a change log entry in both trunk and branch_4x.
        Hide
        Erick Erickson added a comment -

        Two minor nits then....

        TestZKPropertiesWriter.java has an uncommented "System.out.println" in it, probably want to take that out/comment?

        CHANGES.txt could use an entry.....

        FWIW,
        Erick

        Show
        Erick Erickson added a comment - Two minor nits then.... TestZKPropertiesWriter.java has an uncommented "System.out.println" in it, probably want to take that out/comment? CHANGES.txt could use an entry..... FWIW, Erick
        Hide
        Shalin Shekhar Mangar added a comment -

        I'm about to do just that. I was waiting for the whole suite run to finish

        Show
        Shalin Shekhar Mangar added a comment - I'm about to do just that. I was waiting for the whole suite run to finish
        Hide
        Erick Erickson added a comment -

        Shalin:

        Thanks! Since you did the hard part, want me to check it in?

        Show
        Erick Erickson added a comment - Shalin: Thanks! Since you did the hard part, want me to check it in?
        Hide
        Shalin Shekhar Mangar added a comment -

        Added a test which uses DIH on a ZK Aware solr

        Show
        Shalin Shekhar Mangar added a comment - Added a test which uses DIH on a ZK Aware solr
        Hide
        Sauvik Sarkar added a comment -

        We are stuck with this issue after we upgraded from 4.0 to 4.1 Please let me know when will this be committed.

        Show
        Sauvik Sarkar added a comment - We are stuck with this issue after we upgraded from 4.0 to 4.1 Please let me know when will this be committed.
        Hide
        Shalin Shekhar Mangar added a comment -

        I can take this up and write some tests. Will make sure that it goes into 4.1

        Show
        Shalin Shekhar Mangar added a comment - I can take this up and write some tests. Will make sure that it goes into 4.1
        Hide
        Erick Erickson added a comment -

        I really want to insure that this gets into 4.1, or explicitly decide not to as part of the release process... so moving back to blocker. If nothing else, we can commit w/o tests. Honest, I spent some time trying to make some tests, but between not knowing much about either ZK or DIH tests, I got pretty lost.....

        Show
        Erick Erickson added a comment - I really want to insure that this gets into 4.1, or explicitly decide not to as part of the release process... so moving back to blocker. If nothing else, we can commit w/o tests. Honest, I spent some time trying to make some tests, but between not knowing much about either ZK or DIH tests, I got pretty lost.....
        Hide
        Erick Erickson added a comment -

        I'm going to go ahead and commit this in a day or two unless someone objects. If I do before some angel creates tests, I'll open up a new JIRA for tests referring back to this one.

        Show
        Erick Erickson added a comment - I'm going to go ahead and commit this in a day or two unless someone objects. If I do before some angel creates tests, I'll open up a new JIRA for tests referring back to this one.
        Hide
        Erick Erickson added a comment -

        Making this a blocker so we are sure to deal with it before 4.1 is cut. I won't have time to deal until next week and don't want to forget it.

        Show
        Erick Erickson added a comment - Making this a blocker so we are sure to deal with it before 4.1 is cut. I won't have time to deal until next week and don't want to forget it.
        Hide
        Mark Miller added a comment -

        Yup, we already have some base classes that I would use to build on. I would just help get it setup to run from the DIH module and offer any help around getting DIH configured and running with it. If you can easily just figure that out, then by all means, go on ahead. If you have any questions, let me know.

        Show
        Mark Miller added a comment - Yup, we already have some base classes that I would use to build on. I would just help get it setup to run from the DIH module and offer any help around getting DIH configured and running with it. If you can easily just figure that out, then by all means, go on ahead. If you have any questions, let me know.
        Hide
        James Dyer added a comment -

        Erick, I certainly mean for this to be committed prior to a 4.1 (possibly it should be a Blocker), and if you want to commit, then go ahead. My only concern is that if it gets committed without test coverage that a new issue be opened to add a test later.

        I'm tired of breaking DIH every time I commit a change because the test coverage was inadequate and I was lulled into a false sense of security by "All Tests Pass". ZKPropertiesWriter is a case in point. This was committed back in February with no test coverage and then I come along and break it in November. I've been trying lately to write better tests for DIH but if people's attitude is "DIH is a low-quality contrib mod...I don't need to write a test", then what's the point? (we all have stories about working with developers who created "negative productivity". I'm trying to personally stay out of anyone's story like that here!) Then again my concern about coverage shouldn't hold this hostage so it misses the next release.

        I also wonder if SOLR-3911 means changing how ZKPropertiesWriter works entirely, or if it is even necessary at all? I don't know enough about SolrCloud (yet...its on my to-do list) to understand how SOLR-3911 might change what needs to be done here, but maybe there is a different, better approach now?

        Mark, you said something about setting up a base test class to build on, but I was just looking at AbstrackDistribZKTestCase and isn't such a base class already there? Isn't all we need to do is to set up a 1-node, 1-shard "cluster" and have it import 1 document then open the Properties file with ZKPropertiesWriter and see if it exists and can be read? Someday we can get fancy and expand TestSqlEntityProcessorDelta to sometimes run in full distributed mode but for now something quick and simple would suffice, right?

        Show
        James Dyer added a comment - Erick, I certainly mean for this to be committed prior to a 4.1 (possibly it should be a Blocker), and if you want to commit, then go ahead. My only concern is that if it gets committed without test coverage that a new issue be opened to add a test later. I'm tired of breaking DIH every time I commit a change because the test coverage was inadequate and I was lulled into a false sense of security by "All Tests Pass". ZKPropertiesWriter is a case in point. This was committed back in February with no test coverage and then I come along and break it in November. I've been trying lately to write better tests for DIH but if people's attitude is "DIH is a low-quality contrib mod...I don't need to write a test", then what's the point? (we all have stories about working with developers who created "negative productivity". I'm trying to personally stay out of anyone's story like that here!) Then again my concern about coverage shouldn't hold this hostage so it misses the next release. I also wonder if SOLR-3911 means changing how ZKPropertiesWriter works entirely, or if it is even necessary at all? I don't know enough about SolrCloud (yet...its on my to-do list) to understand how SOLR-3911 might change what needs to be done here, but maybe there is a different, better approach now? Mark, you said something about setting up a base test class to build on, but I was just looking at AbstrackDistribZKTestCase and isn't such a base class already there? Isn't all we need to do is to set up a 1-node, 1-shard "cluster" and have it import 1 document then open the Properties file with ZKPropertiesWriter and see if it exists and can be read? Someday we can get fancy and expand TestSqlEntityProcessorDelta to sometimes run in full distributed mode but for now something quick and simple would suffice, right?
        Hide
        Erick Erickson added a comment -

        I've been working with a client that has this exact problem, and this patch cures it on last night's 4x code. If I get the time this weekend, I'd like to
        1> try it on trunk
        2> make sure all tests work in both
        3> commit it to both trunk and 4x.

        Other then the perennial "make some tests too", any objections?

        James:

        Since I have the driving need, should I just assign this to myself?

        Show
        Erick Erickson added a comment - I've been working with a client that has this exact problem, and this patch cures it on last night's 4x code. If I get the time this weekend, I'd like to 1> try it on trunk 2> make sure all tests work in both 3> commit it to both trunk and 4x. Other then the perennial "make some tests too", any objections? James: Since I have the driving need, should I just assign this to myself?
        Hide
        Mark Miller added a comment -

        don't know how to write a good test for SolrCloud stuff

        And I don't know DIH at all - but perhaps I can setup some base test class for you to build on.

        Show
        Mark Miller added a comment - don't know how to write a good test for SolrCloud stuff And I don't know DIH at all - but perhaps I can setup some base test class for you to build on.
        Hide
        James Dyer added a comment -

        It looks like all we need for this now is a unit test for ZKPropertiesWriter, which I do not think has any test coverage? (After breaking a bunch of things in the DIH included with Solr 3.6, I'm really trying to improve the test situation with DIH. But I don't know how to write a good test for SolrCloud stuff...)

        Show
        James Dyer added a comment - It looks like all we need for this now is a unit test for ZKPropertiesWriter, which I do not think has any test coverage? (After breaking a bunch of things in the DIH included with Solr 3.6, I'm really trying to improve the test situation with DIH. But I don't know how to write a good test for SolrCloud stuff...)
        Hide
        Deniz Durmus added a comment -

        As a feedback, would like to say that the patch that James Dyer prepared seems to solve the exceptions

        Show
        Deniz Durmus added a comment - As a feedback, would like to say that the patch that James Dyer prepared seems to solve the exceptions
        Hide
        James Dyer added a comment -

        Please try the attached patch and let me know if this fixes your issue. This may have been broken with SOLR-4051 (Trunk r1408873). You might also want to try Trunk prior to this r1408873 and verify it works there.

        Unfortunately I am behind the times and do not know much about solrcloud. Perhaps Sami can help out?

        Eventually I would also like to figure out how to write a good test for ZKPropertiesWriter. I think it has no coverage and thus easily broken.

        Show
        James Dyer added a comment - Please try the attached patch and let me know if this fixes your issue. This may have been broken with SOLR-4051 (Trunk r1408873). You might also want to try Trunk prior to this r1408873 and verify it works there. Unfortunately I am behind the times and do not know much about solrcloud. Perhaps Sami can help out? Eventually I would also like to figure out how to write a good test for ZKPropertiesWriter. I think it has no coverage and thus easily broken.

          People

          • Assignee:
            Shalin Shekhar Mangar
            Reporter:
            Deniz Durmus
          • Votes:
            1 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development