[GEODE-9] Spark Integration

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.0.0-incubating.M1
• Component/s: core, extensions
• Labels: asf-migration experimental
• Assignee: Jason Huynh
• Reporter: Dan Smith
• Created: 07/May/15 19:54
• Updated: 12/Aug/16 16:52
• Resolved: 27/Jul/15 22:25

Description

This is a feature that has been under development for GemFire but was not part of the initial drop of code for geode.

Geode as a data store for Spark applications is what is being enabled here. By providing a bridge style connector for Spark applications, Geode can become the data store for storing intermediate and final state for Spark applications and allow reference data stored in the in memory tier to be accessed very efficiently for applications

Expose Geode regions as Spark RDDs
Write Spark RDDs to Geode Regions
Execute arbitrary OQL queries in your spark applications

Sub-Tasks

• Resolve Build Issues: Closed
• There's race condition in DefaultGemFireConnection.getRegionProxy: Closed
• Change gfsh Path for Integration Tests: Closed
• RDD.saveToGemfire() can not handle big dataset (1M entries per partition): Closed
• Document Change: Rename GemFire to Geode: Closed