[IGNITE-3084] Spark Data Frames Support in Apache Ignite

• Type: New Feature
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.5.0.final
• Fix Version/s: 2.4
• Component/s: spark
• Labels: bigdata important
• Assignee: Nikolay Izhikov
• Reporter: Vladimir Ozerov
• Created: 04/May/16 11:02
• Updated: 29/Dec/17 21:16
• Resolved: 29/Dec/17 21:15

Description

Apache Spark already benefits from integration with Apache Ignite. The latter provides shared RDDs, an implementation of Spark RDD, that help Spark to share a state between Spark workers and execute SQL queries much faster. The next logical step is to enable support for modern Spark Data Frames API in a similar way.

As a contributor, you will be fully in charge of the integration of Spark Data Frame API and Apache Ignite.

Issue Links

• is required by: IGNITE-7077 Spark Data Frame Support. Strategy to convert complete query to Ignite SQL
• is required by: IGNITE-7337 Spark Data Frames: support saving a data frame in Ignite
• relates to: IGNITE-4517 Add ability to execute SQL queries on certain partition(s)
• requires: SPARK-17767 Spark SQL ExternalCatalog API custom implementation support
• links to: GitHub Pull Request #2742
• links to: Ignite devlist
• links to: Spark devlist
• links to: upsource review