Preliminary version of the patch: no tests, no configuration parameters.

Added integration test: check that all nodes join the cluster.

This patch generates the configuration file using options from the cluster spec. I've done testing on AWS. The only remaining issue: support for cloudservers. I appreciate any feedback on this patch.

Heya, not too familiar with whirr, so I did not exactly saw where the discovery part is configured? Does it use elasticsearch built in ec2 based discovery module, or does it build a unicast list of hosts for it to use?

Also, the default gateway is local, meaning that data is stored on the local/ebs drive. Of course, when on ec2, local drives are wiped, but, if on EBS, then it can restore its state from there.

By default Whirr will configure elasticsearch to use the EC2 discovery module on AWS and store the index in memory with no disk data persistence. On cloudservers I'm thinking about using unicast and build a list of hosts. All configuration parameters can be set in the cluster spec .properties file. A typical recipe for AWS looks like this:

# Change the cluster name here
whirr.cluster-name=elasticsearch
# This is also the elasticsearch cluster name

# Change the number of machines in the cluster here
whirr.instance-templates=2 elasticsearch
whirr.hardware-min-ram=2048

# For EC2 set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables.
whirr.provider=aws-ec2
whirr.identity=${env:AWS_ACCESS_KEY_ID}
whirr.credential=${env:AWS_SECRET_ACCESS_KEY}

# S3 persistence
es.gateway.type=s3
es.gateway.s3.bucket=elasticsearch

Tested with AWS and Rackspace. I'm happy with this patch version. Please review.

+1

looks pretty straightforward. We can probably make this more robust over time, esp as we add features to whirr such as topology changing and resilience. For now, seems a good start.

tested on aws-ec2 and cloudservers-us

I will submit a new version of this patch Monday with more improvements. Thanks for reviewing.

Minor updates to patch:

• if not on EC2 configure unicast discovery (most of the cloud providers deny multicast)
• retry tarball download on failure

Integration tests are passing. I'm planning to commit this tomorrow. We can fix any remaining undiscovered issues in new JIRAs.

I've just committed this. Thanks Adrian for reviewing.

This is awesome - works in a one-liner. What would be a really cool addition would be supporting elasticsearch-jetty to provide authentication and access control.

btw - please print destroy-cluster instructions at the end of the ec2 recipe!

Good idea! Can you add two more jira issues? (one for destroy instructions and one for jetty as a service) Any contribution is highly appreciated!

See WHIRR-547 for jetty access c

Thanks for adding the new issues to JIRA! Do you need elasticsearch-jetty because you are exposing the cluster on the internet?

Yes, on EC2 I need to either add filter rules to the recipe - such as a set of IPs that can access it, or give it auth/access control. Maybe both.