[BIGTOP-1235] Speed-up init-hdfs.sh for complete HCFS compliant provisioning

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.7.0
• Fix Version/s: 1.0.0
• Component/s: deployment, general
• Assignee: Konstantin I Boudnik
• Reporter: jay vyas
• Created: 01/Mar/14 16:34
• Updated: 11/May/15 21:39
• Resolved: 15/Mar/15 21:29

Description

BIGTOP-1200 provides file schema in JSON format, for any HCFS related tool to read.

BIGTOP-952 meanwhile give us fast provisioning by running in a single JVM (without multiple haoop fs -... commands_).

Thus, when those two are finished, we can replace init-hdfs.sh (with the exception of a few extra shell commands in bottom of that script which still will need to be called).

So this JIRA will:

• create a new init-hcfs.groovy script which calls provision.groovy in BIGTOP-952, with init-hcfs.json as input.
• add in the remaining straggler commands from init-hdfs.sh which dont fit into the generic json format as shell commands.
• delete init-hdfs.sh entirely
• update puppet scripts to use init-hcfs.groovy, calling them with the "hdfs" argument.

Issue Links

• breaks: BIGTOP-1817 bigtop-utils deb is missing init-hcfs.groovy file
• breaks: BIGTOP-1865 circular dependency between bigtop-groovy and bigtop-utils
• is related to: BIGTOP-1763 Handle broken symlinks when copying jars
• is related to: BIGTOP-952 init-hdfs.sh is dog slow. Let's replace it with a direct HDFS API calls and better layout management
• is related to: BIGTOP-1200 Implement Generic Text File to define HCFS filesystem semantics