[DRILL-6575] Add store.hive.conf.properties option to allow set Hive properties at session level

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.13.0
• Fix Version/s: 1.14.0
• Labels: doc-complete ready-to-commit
• Assignee: Arina Ielchiieva
• Reporter: Arina Ielchiieva
• Reviewer: Vitalii Diravka
• Created: 02/Jul/18 16:38
• Updated: 11/Jul/18 23:32
• Resolved: 09/Jul/18 12:18

Description

Use case

Hive external table ddl:

create external table my(key int, val string)
row format delimited
fields terminated by ','
stored as textfile
location '/data/my_tbl';

Path /data/my_tb contains sub directory and file in it: /data/my_tbl/sub_dir/data.txt with the following data:

1, value_1
2, value_2

When querying such table from Hive, user gets the following exception:

Failed with exception java.io.IOException:java.io.IOException: Not a file: file:///data/my_tbl/sub_dir

To be able to query this table user needs to set two properties to true: hive.mapred.supports.subdirectories and mapred.input.dir.recursive.
They can be set at system level in hive-site.xml or at session in Hive console:

set hive.mapred.supports.subdirectories=true;
set mapred.input.dir.recursive=true;

Currently to be able to query such table from Drill, user can specify this properties in Hive plugin only:

{
  "type": "hive",
  "configProps": {
    "hive.metastore.uris": "thrift://localhost:9083",
    "hive.metastore.sasl.enabled": "false",
    "hbase.zookeeper.quorum": "localhost",
    "hbase.zookeeper.property.clientPort": "5181",
    "hive.mapred.supports.subdirectories": "true",
    "mapred.input.dir.recursive": "true"
  }
  "enabled": true
}

Jira scope
This Jira aims to add new session option to Drill store.hive.conf.properties which will allow user to specify hive properties at session level.
User should write properties in string delimited by new line symbol. Properties values should NOT be set in double-quotes or any other quotes otherwise they would be parsed incorrectly. Property name and value should be separated by =. Each `alter session set` will override previously set properties at session level. If during query Drill couldn't unparse property string, warning will be logged. Properties will be parsed by loading into java.util.Properties. Default value is empty string ("").

Example:

alter session set `store.hive.conf.properties` = 'hive.mapred.supports.subdirectories=true\nmapred.input.dir.recursive=true';

Issue Links

• links to: GitHub Pull Request #1365