[HBASE-13992] Integrate SparkOnHBase into HBase

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0-alpha-1, connector-1.0.0
• Component/s: hbase-connectors, spark

Release Note

This release includes initial support for running Spark against HBase with a richer feature set than was previously possible with MapReduce bindings:

* Support for Spark and Spark Streaming against Spark 2.1.1
* RDD/DStream formation from scan operations
* convenience methods for interacting with HBase from an HBase backed RDD / DStream instance
* examples in both the Spark Java API and Spark Scala API
* support for running against a secure HBase cluster

• Assignee: Theodore michael Malaska
• Reporter: Theodore michael Malaska
• Created: 29/Jun/15 19:02
• Updated: 28/Sep/24 02:50
• Resolved: 28/Jul/15 16:58

Description

This Jira is to ask if SparkOnHBase can find a home in side HBase core.

Here is the github:
https://github.com/cloudera-labs/SparkOnHBase

I am the core author of this project and the license is Apache 2.0

A blog explaining this project is here
http://blog.cloudera.com/blog/2014/12/new-in-cloudera-labs-sparkonhbase/

A spark Streaming example is here
http://blog.cloudera.com/blog/2014/11/how-to-do-near-real-time-sessionization-with-spark-streaming-and-apache-hadoop/

A real customer using this in produce is blogged here
http://blog.cloudera.com/blog/2015/03/how-edmunds-com-used-spark-streaming-to-build-a-near-real-time-dashboard/

Please debate and let me know what I can do to make this happen.

Issue Links

• duplicates: HBASE-11482 Optimize HBase TableInput/OutputFormats for exposing tables and snapshots as Spark RDDs
• is cloned by: HBASE-20570 CLONE - Integrate SparkOnHBase into HBase
• is depended upon by: HBASE-14149 Add Data Frame support for HBase-Spark Module
• is depended upon by: HBASE-14150 Add BulkLoad functionality to HBase-Spark Module
• is depended upon by: HBASE-14160 backport hbase-spark module to branch-1 and branch-2
• is depended upon by: HBASE-14158 Add documentation for Initial Release for HBase-Spark Module integration
• is depended upon by: HBASE-14159 Resolve warning introduced by HBase-Spark module
• is duplicated by: HBASE-20570 CLONE - Integrate SparkOnHBase into HBase
• is related to: SPARK-2447 Add common solution for sending upsert actions to HBase (put, deletes, and increment)
• is related to: HBASE-11843 MapReduce classes shouldn't be in hbase-server
• links to: review board