[FLINK-959] Automated bare-metal deployment of FLINK on Amazon EC2 and OpenStack instances

• Type: New Feature
• Status: Resolved
• Priority: Minor
• Resolution: Invalid
• Affects Version/s: pre-apache-0.5
• Fix Version/s: pre-apache-0.5
• Component/s: Build System
• Assignee: Tobias
• Reporter: Tobias
• Created: 20/Jun/14 10:36
• Updated: 11/Oct/16 14:43
• Resolved: 11/Oct/16 14:43

Description

This python script does start Amazon ec2|OpenStack instances to install java+hadoop and configure hdfs/yarn via puppet. In order to run FLINK on top of hadoop YARN.

In order to install java and hadoop binaries are downloaded by the script and handed over to puppet for automated provisioning.

User-data scripts are used to install puppet (only debian) on the master and slave instances. Accordingly security groups are created and configured.
The master instance then starts a self configuration process, so that the puppet modules are set up according to the cluster structure.

The master detects if hadoop YARN web interface is accessible and waits for all expected nodes to be up and running. Then a stratosphere yarn session is started. Taskmanager and Jobmanager memory allocations are set up in the instances.cfg.

Notes:

• Configuration reserves 600mb for the operating system and allocates the rest for the YARN node.
• The Flink web interface is not accessible because the yarn.web.proxy throws a NullpointerException
• Only runs on Debian derivatives because it uses apt-get
• Tested with ubuntu-13.08
• FLINK is still named Stratosphere

Code at: https://github.com/tobwiens/StratopshereBareMetalProvPuppet

Issue Links

• supercedes: FLINK-40 [GitHub] Cloud connector