Details
-
Type:
New Feature
-
Status: Resolved
-
Priority:
Major
-
Resolution: Later
-
Affects Version/s: 0.23.0
-
Fix Version/s: None
-
Component/s: mrv2
-
Labels:None
-
Environment:
All Unix-Environments
Description
MPI is commonly used for many machine-learning applications. OpenMPI (http://www.open-mpi.org/) is a popular BSD-licensed version of MPI. In the past, running MPI application on a Hadoop cluster was achieved using Hadoop Streaming (http://videolectures.net/nipsworkshops2010_ye_gbd/), but it was kludgy. After the resource-manager separation from JobTracker in Hadoop, we have all the tools needed to make MPI a first-class citizen on a Hadoop cluster. I am currently working on the patch to make MPI an application-master. Initial version of this patch will be available soon (hopefully before September 10.) This jira will track the development of Hamster: The application master for MPI.
Attachments
Issue Links
- is related to
-
SPARK-24374 SPIP: Support Barrier Scheduling in Apache Spark
-
- In Progress
-
