Details
-
New Feature
-
Status: Resolved
-
Major
-
Resolution: Won't Fix
-
None
-
None
-
None
-
None
Description
Project available on Github: https://github.com/ibm-et/spark-kernel
This architecture is describing running kernel code that was demonstrated at the StrataConf in Barcelona, Spain.
Enables applications to interact with a Spark cluster using Scala in several ways:
- Defining and running core Spark Tasks
- Collecting results from a cluster without needing to write to external data store
- Ability to stream results using well-defined protocol
- Arbitrary Scala code definition and execution (without submitting heavy-weight jars)
Applications can be hosted and managed separate from the Spark cluster using the kernel as a proxy to communicate requests.
The Spark Kernel implements the server side of the IPython Kernel protocol, the rising “de-facto” protocol for language (Python, Haskell, etc.) execution.
Inherits a suite of industry adopted clients such as the IPython Notebook.