[HBASE-14849] Add option to set block cache to false on SparkSQL executions

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0-alpha-1, connector-1.0.0
• Component/s: hbase-connectors, spark
• Hadoop Flags: Reviewed

Release Note

For user configurable parameters for HBase datasources. Please refer to org.apache.hadoop.hbase.spark.datasources.HBaseSparkConf for details.

User can either set them in SparkConf, which will take effect globally, or configure it per table, which will overwrite the value set in SparkConf. If not set, the default value will take effect.

Currently three parameters are supported.
1. spark.hbase.blockcache.enable for blockcache enable/disable. Default is enable, but note that this potentially may slow down the system.
2. spark.hbase.cacheSize for cache size when performing HBase table scan. Default value is 1000
3. spark.hbase.batchNum for the batch number when performing HBase table scan. Default value is 1000.

• Assignee: Zhan Zhang
• Reporter: Theodore michael Malaska
• Created: 19/Nov/15 20:55
• Updated: 26/Apr/19 10:14
• Resolved: 19/Dec/15 23:15

Description

I was working at a client with a ported down version of the Spark module for HBase and realized we didn't add an option to turn of block cache for the scans.

At the client I just disabled all caching with Spark SQL, this is an easy but very impactful fix.

The fix for this patch will make this configurable

Issue Links

• links to: reviewboard