[SPARK-4605] Proposed Contribution: Spark Kernel to enable interactive Spark applications

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Assignee: Unassigned
• Reporter: Chip Senkbeil
• Created: 25/Nov/14 18:56
• Updated: 01/Dec/15 08:08
• Resolved: 19/Jun/15 10:44

Description

Project available on Github: https://github.com/ibm-et/spark-kernel

---

This architecture is describing running kernel code that was demonstrated at the StrataConf in Barcelona, Spain.

---

Enables applications to interact with a Spark cluster using Scala in several ways:

• Defining and running core Spark Tasks
• Collecting results from a cluster without needing to write to external data store
  • Ability to stream results using well-defined protocol
• Arbitrary Scala code definition and execution (without submitting heavy-weight jars)

Applications can be hosted and managed separate from the Spark cluster using the kernel as a proxy to communicate requests.

The Spark Kernel implements the server side of the IPython Kernel protocol, the rising “de-facto” protocol for language (Python, Haskell, etc.) execution.
Inherits a suite of industry adopted clients such as the IPython Notebook.