Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.7.1
    • Fix Version/s: 0.7.2, 0.8.0
    • Component/s: new service
    • Labels:
    1. WHIRR-465-0.7.patch
      49 kB
      Andrei Savu
    2. WHIRR-465-0_7_2.patch
      53 kB
      Frank Scholten
    3. WHIRR-465.patch
      68 kB
      Frank Scholten
    4. WHIRR-465.patch
      66 kB
      Frank Scholten
    5. WHIRR-465.patch
      50 kB
      Frank Scholten
    6. WHIRR-465.diff
      27 kB
      David Arthur
    7. WHIRR-465-1.diff
      4 kB
      David Arthur
    8. WHIRR-465.diff
      19 kB
      David Arthur

      Activity

      Hide
      David Arthur added a comment - - edited

      Adds Solr to Whirr

      Example recipe:

      whirr.cluster-name=solr
      whirr.instance-templates=1 zookeeper,2 solr
      whirr.solr.shards.num=8
      whirr.solr.collection.name=myconf
      whirr.solr.jetty.port=8080
      whirr.provider=aws-ec2
      whirr.identity=${env:AWS_ACCESS_KEY_ID}
      whirr.credential=${env:AWS_SECRET_ACCESS_KEY}
      

      N.B., that a ZooKeeper node is required for things to work correctly. It does not need to be a standalone ZooKeeper though, it can live along side Solr.

      Show
      David Arthur added a comment - - edited Adds Solr to Whirr Example recipe: whirr.cluster-name=solr whirr.instance-templates=1 zookeeper,2 solr whirr.solr.shards.num=8 whirr.solr.collection.name=myconf whirr.solr.jetty.port=8080 whirr.provider=aws-ec2 whirr.identity=${env:AWS_ACCESS_KEY_ID} whirr.credential=${env:AWS_SECRET_ACCESS_KEY} N.B., that a ZooKeeper node is required for things to work correctly. It does not need to be a standalone ZooKeeper though, it can live along side Solr.
      Hide
      Andrei Savu added a comment -

      Nice work! Here are some quick comments / questions:

      • can we reference a specific Apache Solr 4.0 build? Or star a build? Deploying from snapshot can be brittle
      • it should be possible to specify the .tar.gz URL
      • we need an integration test to show that the service starts as expected (indexing a document? status page?)
      • please generate the patch using "git diff --no-prefix trunk" or "svn diff"
      • use getInstallFunction to get the name of the script for install_java
      • it would be nice to have an init script. See core/src/main/resources/functions/ {install,remove}

        _service.sh

      • as we have a start script now we should also have a config and a stop script
      • we usually install services in /usr/local/

      Let me know if I can help on the way. I'm available most of the time on IRC at #whirr (asavu)

      Show
      Andrei Savu added a comment - Nice work! Here are some quick comments / questions: can we reference a specific Apache Solr 4.0 build? Or star a build? Deploying from snapshot can be brittle it should be possible to specify the .tar.gz URL we need an integration test to show that the service starts as expected (indexing a document? status page?) please generate the patch using "git diff --no-prefix trunk" or "svn diff" use getInstallFunction to get the name of the script for install_java it would be nice to have an init script. See core/src/main/resources/functions/ {install,remove} _service.sh as we have a start script now we should also have a config and a stop script we usually install services in /usr/local/ Let me know if I can help on the way. I'm available most of the time on IRC at #whirr (asavu)
      Hide
      David Arthur added a comment - - edited

      Updated patch

      • can we reference a specific Apache Solr 4.0 build? Or star a build? Deploying from snapshot can be brittle
      • it should be possible to specify the .tar.gz URL
      • please generate the patch using "git diff --no-prefix trunk" or "svn diff"
      • use getInstallFunction to get the name of the script for install_java
      • as we have a start script now we should also have a config and a stop script
      • we usually install services in /usr/local/

      Still need a more robust start/stop mechanism and some integration tests

      Show
      David Arthur added a comment - - edited Updated patch can we reference a specific Apache Solr 4.0 build? Or star a build? Deploying from snapshot can be brittle it should be possible to specify the .tar.gz URL please generate the patch using "git diff --no-prefix trunk" or "svn diff" use getInstallFunction to get the name of the script for install_java as we have a start script now we should also have a config and a stop script we usually install services in /usr/local/ Still need a more robust start/stop mechanism and some integration tests
      Hide
      David Arthur added a comment - - edited

      This is an additive patch, apply it after first applying WHIRR-465.diff

      Solr uses hostnames instead of IPs to communicate. B/c of this we need to update the /etc/hosts file on each Solr machine

      Show
      David Arthur added a comment - - edited This is an additive patch, apply it after first applying WHIRR-465 .diff Solr uses hostnames instead of IPs to communicate. B/c of this we need to update the /etc/hosts file on each Solr machine
      Hide
      Andrei Savu added a comment -

      Looks great! Ready to commit from my point of view as soon as we add an integration test. I will take care of improving how we handle /etc/hosts & private hostnames inside the Whirr core.

      Show
      Andrei Savu added a comment - Looks great! Ready to commit from my point of view as soon as we add an integration test. I will take care of improving how we handle /etc/hosts & private hostnames inside the Whirr core.
      Hide
      Andrei Savu added a comment -

      Moving on the roadmap for 0.8.0.

      Show
      Andrei Savu added a comment - Moving on the roadmap for 0.8.0.
      Hide
      Andrei Savu added a comment -

      Hi David! Any updates?

      Show
      Andrei Savu added a comment - Hi David! Any updates?
      Hide
      Andrei Savu added a comment -

      Is the trunk working as expected for you now? I know we had some issue with ZooKeeper & JDK.

      Show
      Andrei Savu added a comment - Is the trunk working as expected for you now? I know we had some issue with ZooKeeper & JDK.
      Hide
      Andrei Savu added a comment -

      David are you still working on this? Do you mind if I takeover & finish this patch? (maybe we can ship it in 0.7.2)

      Show
      Andrei Savu added a comment - David are you still working on this? Do you mind if I takeover & finish this patch? (maybe we can ship it in 0.7.2)
      Hide
      David Arthur added a comment -

      Andrei, feel free to take over. The latest diff includes an integration test I started work on.

      Show
      David Arthur added a comment - Andrei, feel free to take over. The latest diff includes an integration test I started work on.
      Hide
      Frank Scholten added a comment -

      I think it would be great if you could specify a schema.xml and solrconfig.xml file (via file:// urls) that can be installed under Solr home. We would have to create a script (similar to install_tarball.sh) which creates the destination dirs if needed and moves the file. Correct?

      Show
      Frank Scholten added a comment - I think it would be great if you could specify a schema.xml and solrconfig.xml file (via file:// urls) that can be installed under Solr home. We would have to create a script (similar to install_tarball.sh) which creates the destination dirs if needed and moves the file. Correct?
      Hide
      Andrei Savu added a comment -

      Right & the upload of local files is something that Whirr already knows how to handle.

      Show
      Andrei Savu added a comment - Right & the upload of local files is something that Whirr already knows how to handle.
      Hide
      Andrei Savu added a comment -

      Frank do you want to finish this patch? I want to do it but I won't have enough time until later next week.

      Show
      Andrei Savu added a comment - Frank do you want to finish this patch? I want to do it but I won't have enough time until later next week.
      Hide
      Frank Scholten added a comment -

      Yes. I'll add an integration test and the configuring of schema.xml and solconfig.xml

      Show
      Frank Scholten added a comment - Yes. I'll add an integration test and the configuring of schema.xml and solconfig.xml
      Hide
      Frank Scholten added a comment -

      I wrote an integration test for the Solr service but I am having problems running it via BYON on Vagrant VMs.

      Check out the WHIRR-465 branch at my Github https://github.com/frankscholten/whirr/tree/WHIRR-465

      If you run 'vagrant up' in services/solr/src/test/resources and then run SolrServiceTest it throws the exception

      java.util.NoSuchElementException: no image matched predicate: and(locationEqualsOrChildOf(byon),And(osFamily(ubuntu),osVersion(10.04)))

      Is this a problem in the YAML file?

      Show
      Frank Scholten added a comment - I wrote an integration test for the Solr service but I am having problems running it via BYON on Vagrant VMs. Check out the WHIRR-465 branch at my Github https://github.com/frankscholten/whirr/tree/WHIRR-465 If you run 'vagrant up' in services/solr/src/test/resources and then run SolrServiceTest it throws the exception java.util.NoSuchElementException: no image matched predicate: and(locationEqualsOrChildOf(byon),And(osFamily(ubuntu),osVersion(10.04))) Is this a problem in the YAML file?
      Hide
      Frank Scholten added a comment -

      I did some debugging and I saw that images parameter in the jclouds TemplateBuilderImpl constructor is null and because of this the predicate filter in the build() throws the exception.

      Show
      Frank Scholten added a comment - I did some debugging and I saw that images parameter in the jclouds TemplateBuilderImpl constructor is null and because of this the predicate filter in the build() throws the exception.
      Hide
      Frank Scholten added a comment -

      How about splitting this one up in separate incremental features?

      We could start out with a simple solr role that starts a single solr service with a tarball containing core configuration (solr.xml, solrconfig.xml, schema.xml). We can create a separate ticket for 3x solr-master and solr-slave roles for a replicated setup. Then there is also the SolrCloud with Zookeeper setup.

      WDYT?

      Show
      Frank Scholten added a comment - How about splitting this one up in separate incremental features? We could start out with a simple solr role that starts a single solr service with a tarball containing core configuration (solr.xml, solrconfig.xml, schema.xml). We can create a separate ticket for 3x solr-master and solr-slave roles for a replicated setup. Then there is also the SolrCloud with Zookeeper setup. WDYT?
      Hide
      David Arthur added a comment -

      +1 Just getting single-node Solr in is a good start.

      How about the following breakdown:

      • Single-node Solr (WHIRR-465, this issue)
      • SolrCloud + ZooKeeper
      • Advanced configuration for both (BYO solrconfig.xml, schema.xml)
      Show
      David Arthur added a comment - +1 Just getting single-node Solr in is a good start. How about the following breakdown: Single-node Solr ( WHIRR-465 , this issue) SolrCloud + ZooKeeper Advanced configuration for both (BYO solrconfig.xml, schema.xml)
      Hide
      David Arthur added a comment -

      Reassigning to you, Frank. I'll continue watching this issue and providing feedback.

      Thanks again for picking this up!

      Show
      David Arthur added a comment - Reassigning to you, Frank. I'll continue watching this issue and providing feedback. Thanks again for picking this up!
      Hide
      Frank Scholten added a comment -

      Ok. I do think that even for single-node solr you need be able to configure solr.xml, solrconfig.xml and schema.xml before startup. I already have code for this. It installs a tarball with core configuration into $SOLR_HOME.

      Show
      Frank Scholten added a comment - Ok. I do think that even for single-node solr you need be able to configure solr.xml, solrconfig.xml and schema.xml before startup. I already have code for this. It installs a tarball with core configuration into $SOLR_HOME.
      Hide
      Frank Scholten added a comment -

      Single node Solr service.

      Test passes but I do get a concurrent excecution exception when the blobstore is destroyed when the test finishes. How can this be fixed?

      Show
      Frank Scholten added a comment - Single node Solr service. Test passes but I do get a concurrent excecution exception when the blobstore is destroyed when the test finishes. How can this be fixed?
      Hide
      Frank Scholten added a comment -

      Currently the bucket with the Solr configuration tarball is not destroyed when the cluster is taken down.

      Show
      Frank Scholten added a comment - Currently the bucket with the Solr configuration tarball is not destroyed when the cluster is taken down.
      Hide
      Andrei Savu added a comment -

      I think that's a race condition on jvm shutdown - all shutdown hooks are executed in parallel. Maybe we should do an explicit call to BlobCache.dropAndCloseAll():

      https://github.com/apache/whirr/blob/trunk/core/src/main/java/org/apache/whirr/util/BlobCache.java

      Show
      Andrei Savu added a comment - I think that's a race condition on jvm shutdown - all shutdown hooks are executed in parallel. Maybe we should do an explicit call to BlobCache.dropAndCloseAll(): https://github.com/apache/whirr/blob/trunk/core/src/main/java/org/apache/whirr/util/BlobCache.java
      Hide
      Andrei Savu added a comment -

      I've created a review request at https://reviews.apache.org/r/5076/ and added a few comments. I am also seeing the java.util.concurrent.RejectedExecutionException and I'm looking for an workaround.

      Show
      Andrei Savu added a comment - I've created a review request at https://reviews.apache.org/r/5076/ and added a few comments. I am also seeing the java.util.concurrent.RejectedExecutionException and I'm looking for an workaround.
      Hide
      Frank Scholten added a comment - - edited

      Updated patch which removes the BlobCache. Tested on aws-ec2

      Show
      Frank Scholten added a comment - - edited Updated patch which removes the BlobCache. Tested on aws-ec2
      Hide
      Andrei Savu added a comment -

      Also working fine with cloudservers-uk & cloudfiles. The only change I made to the initial patch was to add BlobCache.dropAndCloseAll() to SolrServiceTest.after

      Show
      Andrei Savu added a comment - Also working fine with cloudservers-uk & cloudfiles. The only change I made to the initial patch was to add BlobCache.dropAndCloseAll() to SolrServiceTest.after
      Hide
      Andrei Savu added a comment -

      And one last thing: don't forget to add whirr-solr as a dependency to whirr-cli.

      Show
      Andrei Savu added a comment - And one last thing: don't forget to add whirr-solr as a dependency to whirr-cli.
      Hide
      Frank Scholten added a comment -

      Updated patch with changes for review request https://reviews.apache.org/r/5076/

      Show
      Frank Scholten added a comment - Updated patch with changes for review request https://reviews.apache.org/r/5076/
      Hide
      Andrei Savu added a comment -

      +1

      Show
      Andrei Savu added a comment - +1
      Hide
      Andrei Savu added a comment -

      How about having a version of this that we can ship with 0.7.2? The code is almost the same (except the OSGI bits & start/stop functions).

      Show
      Andrei Savu added a comment - How about having a version of this that we can ship with 0.7.2? The code is almost the same (except the OSGI bits & start/stop functions).
      Hide
      Frank Scholten added a comment -

      Committed to trunk.

      Ok, I'll work on a 0.7.2 patch.

      Show
      Frank Scholten added a comment - Committed to trunk. Ok, I'll work on a 0.7.2 patch.
      Hide
      Frank Scholten added a comment -

      Here is a work in progress patch for 0.7.2. Somehow the Solr config tarball doesn't get installed under /usr/local/solr

      WDYT?

      Show
      Frank Scholten added a comment - Here is a work in progress patch for 0.7.2. Somehow the Solr config tarball doesn't get installed under /usr/local/solr WDYT?
      Hide
      Andrei Savu added a comment -

      Updated patch for latest 0.7 branch. Building fine but integration tests still fail.

      Show
      Andrei Savu added a comment - Updated patch for latest 0.7 branch. Building fine but integration tests still fail.

        People

        • Assignee:
          Frank Scholten
          Reporter:
          Andrei Savu
        • Votes:
          5 Vote for this issue
          Watchers:
          8 Start watching this issue

          Dates

          • Created:
            Updated:
            Resolved:

            Development