[MAPREDUCE-2911] Hamster: Hadoop And Mpi on the same cluSTER

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Later
• Affects Version/s: 0.23.0
• Component/s: mrv2

Environment

All Unix-Environments

• Assignee: Unassigned
• Reporter: Milind Barve
• Created: 30/Aug/11 07:04
• Updated: 04/Jun/18 19:54
• Resolved: 22/May/12 17:36

Description

MPI is commonly used for many machine-learning applications. OpenMPI (http://www.open-mpi.org/) is a popular BSD-licensed version of MPI. In the past, running MPI application on a Hadoop cluster was achieved using Hadoop Streaming (http://videolectures.net/nipsworkshops2010_ye_gbd/), but it was kludgy. After the resource-manager separation from JobTracker in Hadoop, we have all the tools needed to make MPI a first-class citizen on a Hadoop cluster. I am currently working on the patch to make MPI an application-master. Initial version of this patch will be available soon (hopefully before September 10.) This jira will track the development of Hamster: The application master for MPI.

Issue Links

• is related to: SPARK-24374 SPIP: Support Barrier Execution Mode in Apache Spark