[KAFKA-1754] KOYA - Kafka on YARN

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: packaging
• Assignee: Unassigned
• Reporter: Thomas Weise
• Created: 05/Nov/14 00:29
• Updated: 04/Sep/18 15:15

Description

YARN (Hadoop 2.x) has enabled clusters to be used for a variety of workloads, emerging as distributed operating system for big data applications. Initiatives are on the way to bring long running services under the YARN umbrella, leveraging it for centralized resource management and operations (YARN-896 and examples such as HBase, Accumulo or Memcached through Slider). This JIRA is to propose KOYA (Kafka On Yarn), a YARN application master to launch and manage Kafka clusters running on YARN. Brokers will use resources allocated through YARN with support for recovery, monitoring etc. Please see attached for more details.

Issue Links

• duplicates: KAFKA-949 Integrate kafka into YARN
• is related to: YARN-2817 Disk drive as a resource in YARN