Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-14849

Add option to set block cache to false on SparkSQL executions

    XMLWordPrintableJSON

    Details

    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      For user configurable parameters for HBase datasources. Please refer to org.apache.hadoop.hbase.spark.datasources.HBaseSparkConf for details.

      User can either set them in SparkConf, which will take effect globally, or configure it per table, which will overwrite the value set in SparkConf. If not set, the default value will take effect.

      Currently three parameters are supported.
      1. spark.hbase.blockcache.enable for blockcache enable/disable. Default is enable, but note that this potentially may slow down the system.
      2. spark.hbase.cacheSize for cache size when performing HBase table scan. Default value is 1000
      3. spark.hbase.batchNum for the batch number when performing HBase table scan. Default value is 1000.
      Show
      For user configurable parameters for HBase datasources. Please refer to org.apache.hadoop.hbase.spark.datasources.HBaseSparkConf for details. User can either set them in SparkConf, which will take effect globally, or configure it per table, which will overwrite the value set in SparkConf. If not set, the default value will take effect. Currently three parameters are supported. 1. spark.hbase.blockcache.enable for blockcache enable/disable. Default is enable, but note that this potentially may slow down the system. 2. spark.hbase.cacheSize for cache size when performing HBase table scan. Default value is 1000 3. spark.hbase.batchNum for the batch number when performing HBase table scan. Default value is 1000.

      Description

      I was working at a client with a ported down version of the Spark module for HBase and realized we didn't add an option to turn of block cache for the scans.

      At the client I just disabled all caching with Spark SQL, this is an easy but very impactful fix.

      The fix for this patch will make this configurable

        Attachments

        1. HBASE-14849-2.patch
          19 kB
          Zhan Zhang
        2. HBASE-14849-1.patch
          17 kB
          Zhan Zhang
        3. HBASE-14849.patch
          17 kB
          Zhan Zhang

          Issue Links

            Activity

              People

              • Assignee:
                zhanzhang Zhan Zhang
                Reporter:
                ted.m Theodore michael Malaska
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: