[HBASE-14150] Add BulkLoad functionality to HBase-Spark Module

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0-alpha-1, connector-1.0.0
• Component/s: hbase-connectors, spark
• Hadoop Flags: Reviewed
• Assignee: Theodore michael Malaska
• Reporter: Theodore michael Malaska
• Created: 23/Jul/15 00:38
• Updated: 24/Jun/22 19:30
• Resolved: 12/Aug/15 15:29

Description

Add on to the work done in HBASE-13992 to add functionality to do a bulk load from a given RDD.

This will do the following:
1. figure out the number of regions and sort and partition the data correctly to be written out to HFiles
2. Also unlike the MR bulkload I would like that the columns to be sorted in the shuffle stage and not in the memory of the reducer. This will allow this design to support super wide records with out going out of memory.

Issue Links

• depends upon: HBASE-13992 Integrate SparkOnHBase into HBase
• is depended upon by: HBASE-14340 Add second bulk load option to Spark Bulk Load to send puts as the value
• is depended upon by: HBASE-14158 Add documentation for Initial Release for HBase-Spark Module integration
• is depended upon by: HBASE-14216 Consolidate MR and Spark BulkLoad shared functions and string consts
• is depended upon by: HBASE-14217 Add Java access to Spark bulk load functionality
• links to: Review Board