Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.5.0
    • Component/s: None
    • Labels:
      None

      Description

      ElasticSearch is a distributed RESTful search engine built for the cloud.

      [1] http://www.elasticsearch.org/

      1. WHIRR-261.patch
        37 kB
        Andrei Savu
      2. WHIRR-261.patch
        37 kB
        Andrei Savu
      3. WHIRR-261.patch
        30 kB
        Andrei Savu
      4. WHIRR-261.patch
        18 kB
        Andrei Savu
      5. WHIRR-261.patch
        17 kB
        Andrei Savu

        Activity

        Hide
        Russell Jurney added a comment -

        Yes, on EC2 I need to either add filter rules to the recipe - such as a set of IPs that can access it, or give it auth/access control. Maybe both.

        Show
        Russell Jurney added a comment - Yes, on EC2 I need to either add filter rules to the recipe - such as a set of IPs that can access it, or give it auth/access control. Maybe both.
        Hide
        Andrei Savu added a comment -

        Thanks for adding the new issues to JIRA! Do you need elasticsearch-jetty because you are exposing the cluster on the internet?

        Show
        Andrei Savu added a comment - Thanks for adding the new issues to JIRA! Do you need elasticsearch-jetty because you are exposing the cluster on the internet?
        Hide
        Russell Jurney added a comment -

        See WHIRR-547 for jetty access c

        Show
        Russell Jurney added a comment - See WHIRR-547 for jetty access c
        Hide
        Andrei Savu added a comment -

        Good idea! Can you add two more jira issues? (one for destroy instructions and one for jetty as a service) Any contribution is highly appreciated!

        Show
        Andrei Savu added a comment - Good idea! Can you add two more jira issues? (one for destroy instructions and one for jetty as a service) Any contribution is highly appreciated!
        Hide
        Russell Jurney added a comment -

        btw - please print destroy-cluster instructions at the end of the ec2 recipe!

        Show
        Russell Jurney added a comment - btw - please print destroy-cluster instructions at the end of the ec2 recipe!
        Hide
        Russell Jurney added a comment -

        This is awesome - works in a one-liner. What would be a really cool addition would be supporting elasticsearch-jetty to provide authentication and access control.

        Show
        Russell Jurney added a comment - This is awesome - works in a one-liner. What would be a really cool addition would be supporting elasticsearch-jetty to provide authentication and access control.
        Hide
        Andrei Savu added a comment -

        I've just committed this. Thanks Adrian for reviewing.

        Show
        Andrei Savu added a comment - I've just committed this. Thanks Adrian for reviewing.
        Hide
        Andrei Savu added a comment -

        Minor updates to patch:

        • if not on EC2 configure unicast discovery (most of the cloud providers deny multicast)
        • retry tarball download on failure

        Integration tests are passing. I'm planning to commit this tomorrow. We can fix any remaining undiscovered issues in new JIRAs.

        Show
        Andrei Savu added a comment - Minor updates to patch: if not on EC2 configure unicast discovery (most of the cloud providers deny multicast) retry tarball download on failure Integration tests are passing. I'm planning to commit this tomorrow. We can fix any remaining undiscovered issues in new JIRAs.
        Hide
        Andrei Savu added a comment -

        I will submit a new version of this patch Monday with more improvements. Thanks for reviewing.

        Show
        Andrei Savu added a comment - I will submit a new version of this patch Monday with more improvements. Thanks for reviewing.
        Hide
        Adrian Cole added a comment -

        +1

        looks pretty straightforward. We can probably make this more robust over time, esp as we add features to whirr such as topology changing and resilience. For now, seems a good start.

        tested on aws-ec2 and cloudservers-us

        Show
        Adrian Cole added a comment - +1 looks pretty straightforward. We can probably make this more robust over time, esp as we add features to whirr such as topology changing and resilience. For now, seems a good start. tested on aws-ec2 and cloudservers-us
        Hide
        Andrei Savu added a comment -

        Tested with AWS and Rackspace. I'm happy with this patch version. Please review.

        Show
        Andrei Savu added a comment - Tested with AWS and Rackspace. I'm happy with this patch version. Please review.
        Hide
        Andrei Savu added a comment -

        By default Whirr will configure elasticsearch to use the EC2 discovery module on AWS and store the index in memory with no disk data persistence. On cloudservers I'm thinking about using unicast and build a list of hosts. All configuration parameters can be set in the cluster spec .properties file. A typical recipe for AWS looks like this:

        # Change the cluster name here
        whirr.cluster-name=elasticsearch
        # This is also the elasticsearch cluster name
        
        # Change the number of machines in the cluster here
        whirr.instance-templates=2 elasticsearch
        whirr.hardware-min-ram=2048
        
        # For EC2 set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables.
        whirr.provider=aws-ec2
        whirr.identity=${env:AWS_ACCESS_KEY_ID}
        whirr.credential=${env:AWS_SECRET_ACCESS_KEY}
        
        # S3 persistence
        es.gateway.type=s3
        es.gateway.s3.bucket=elasticsearch
        
        Show
        Andrei Savu added a comment - By default Whirr will configure elasticsearch to use the EC2 discovery module on AWS and store the index in memory with no disk data persistence. On cloudservers I'm thinking about using unicast and build a list of hosts. All configuration parameters can be set in the cluster spec .properties file. A typical recipe for AWS looks like this: # Change the cluster name here whirr.cluster-name=elasticsearch # This is also the elasticsearch cluster name # Change the number of machines in the cluster here whirr.instance-templates=2 elasticsearch whirr.hardware-min-ram=2048 # For EC2 set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables. whirr.provider=aws-ec2 whirr.identity=${env:AWS_ACCESS_KEY_ID} whirr.credential=${env:AWS_SECRET_ACCESS_KEY} # S3 persistence es.gateway.type=s3 es.gateway.s3.bucket=elasticsearch
        Hide
        Shay Banon added a comment -

        Heya, not too familiar with whirr, so I did not exactly saw where the discovery part is configured? Does it use elasticsearch built in ec2 based discovery module, or does it build a unicast list of hosts for it to use?

        Also, the default gateway is local, meaning that data is stored on the local/ebs drive. Of course, when on ec2, local drives are wiped, but, if on EBS, then it can restore its state from there.

        Show
        Shay Banon added a comment - Heya, not too familiar with whirr, so I did not exactly saw where the discovery part is configured? Does it use elasticsearch built in ec2 based discovery module, or does it build a unicast list of hosts for it to use? Also, the default gateway is local, meaning that data is stored on the local/ebs drive. Of course, when on ec2, local drives are wiped, but, if on EBS, then it can restore its state from there.
        Hide
        Andrei Savu added a comment -

        This patch generates the configuration file using options from the cluster spec. I've done testing on AWS. The only remaining issue: support for cloudservers. I appreciate any feedback on this patch.

        Show
        Andrei Savu added a comment - This patch generates the configuration file using options from the cluster spec. I've done testing on AWS. The only remaining issue: support for cloudservers. I appreciate any feedback on this patch.
        Hide
        Andrei Savu added a comment -

        Added integration test: check that all nodes join the cluster.

        Show
        Andrei Savu added a comment - Added integration test: check that all nodes join the cluster.
        Hide
        Andrei Savu added a comment -

        Preliminary version of the patch: no tests, no configuration parameters.

        Show
        Andrei Savu added a comment - Preliminary version of the patch: no tests, no configuration parameters.

          People

          • Assignee:
            Andrei Savu
            Reporter:
            Andrei Savu
          • Votes:
            1 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development