Details
-
New Feature
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
Description
Today Apex has two modes of execution: Embedded mode (everything running in a single JVM) and YARN. There has been a few questions around native support for Mesos. A cursory look suggests that Mesos support can be added by reimplementing the YARN specific portions in the master (AppMasterService, ContainerLauncher) and limited changes to the streaming container driver.
Mesos has a different model of resource allocation: The master offers resources to the framework while in YARN resources are requested. Apex master needs to implement the "framework scheduler" that is responsible to accept the resources and control the tasks.
http://mesos.apache.org/documentation/latest/app-framework-development-guide/
Tasks are launched through executors, command line and docker executors are provided.
Apex also requires support to deploy the dependencies to the nodes on which the streaming containers are launched. YARN supports that through resource localization. Mesos supports this through the fetcher, which can copy the resources to the slave node.