[METRON-609] Enhance Mpack to handle single-node and small-cluster installs of Elasticsearch

• Type: Improvement
• Status: Done
• Priority: Major
• Resolution: Done
• Fix Version/s: 0.3.1
• Assignee: Matthew Foley
• Reporter: Matthew Foley
• Created: 04/Dec/16 20:16
• Updated: 15/May/17 16:29
• Resolved: 15/May/17 16:29

Description

The current Mpack for Ambari install of Metron requires that Elasticsearch be installed with 1+ dedicated Masters and 3+ dedicated Slaves. It does not support combined master/datanode configuration (non-"dedicated" Master), which is the recommended way to run a single-node configuration of Elasticsearch; nor does it allow less than 3 dedicated Slaves, thereby requiring the cluster have at least 4 nodes.

This jira proposes to enable 1-, 2-, and 3-node clusters to have a working Elasticsearch install via the Mpack. It will also allow non-dedicated master/datanodes, which are required for single-node and useful for few-node clusters.

I intend to also include the following enhancements and fixes:

• Determine whether elastic-sysconfig:data_dir and elastic-site:path_data should have same default value and if so fix it. (I think they should, and they don't. Probably there should only be one variable instead of two.)
• Provide Quick Links in Ambari service page for Elasticsearch to the self-report pages for Elasticsearch health, node list, and other "_cat" status. May include some "_cluster" status.
• Improve various mouse-over Description fields in the GUI
• Improve the order of fields in the GUI, to group at the beginning the fields that must be modified or are commonly modified. Possibly separate "basic" and "advanced" configs.
• Provide a README.md that describes what is going on with the various resource files in the Mpack src
• Enhance the wiki page for cluster installation, with regard to single-node install with Mpack.
• Database-preserving upgrade from version 1.0.0.0.

Also see https://github.com/apache/incubator-metron/tree/master/metron-deployment#current-limitations for some other known issues. And the documentation in that README.md page is significantly out of date wrt the Mpack.

Issue Links

• blocks: METRON-671 Refactor existing Ansible deployment to use Ambari MPack
• relates to: METRON-608 Mpack to install a single-node test cluster
• links to: GitHub Pull Request #425
• links to: GitHub Pull Request #425

Sub-Tasks

• Mpack bug fixes and improvements (not related to singlenode install): Done
• Fix square-bracket behavior and default network interface bindings for ES: Done