[SPARK-5197] Support external shuffle service in fine-grained mode on mesos cluster

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Won't Fix
• Component/s: Deploy, Mesos, Shuffle, Spark Core
• Assignee: Unassigned
• Reporter: Jongyoul Lee
• Created: 12/Jan/15 01:24
• Updated: 17/May/20 18:30
• Resolved: 01/Sep/16 17:49

Description

I think dynamic allocation is almost satisfied on mesos' fine-grained mode, which already offers resources dynamically, and returns automatically when a task is finished. It, however, doesn't have a mechanism on support external shuffle service like yarn's way, which is AuxiliaryService. Because mesos doesn't support AusiliaryService, we think a different way to do this.

• Launching a shuffle service like a spark job on same cluster
  • Pros
    • Support multi-tenant environment
    • Almost same way like yarn
  • Cons
    • Control long running 'background' job - service - when mesos runs
    • Satisfy all slave - or host - to have one shuffle service all the time
• Launching jobs within shuffle service
  • Pros
    • Easy to implement
    • Don't consider whether shuffle service exists or not.
  • Cons
    • exists multiple shuffle services under multi-tenant environment
    • Control shuffle service port dynamically on multi-user environment

In my opinion, the first one is better idea to support external shuffle service. Please leave comments.

Issue Links

• duplicates: SPARK-4286 Support External Shuffle Service with Mesos integration