Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-6575

Add store.hive.conf.properties option to allow set Hive properties at session level



    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.13.0
    • 1.14.0
    • None


      Use case

      Hive external table ddl:

      create external table my(key int, val string)
      row format delimited
      fields terminated by ','
      stored as textfile
      location '/data/my_tbl';

      Path /data/my_tb contains sub directory and file in it: /data/my_tbl/sub_dir/data.txt with the following data:

      1, value_1
      2, value_2

      When querying such table from Hive, user gets the following exception:

      Failed with exception java.io.IOException:java.io.IOException: Not a file: file:///data/my_tbl/sub_dir

      To be able to query this table user needs to set two properties to true: hive.mapred.supports.subdirectories and mapred.input.dir.recursive.
      They can be set at system level in hive-site.xml or at session in Hive console:

      set hive.mapred.supports.subdirectories=true;
      set mapred.input.dir.recursive=true;

      Currently to be able to query such table from Drill, user can specify this properties in Hive plugin only:

        "type": "hive",
        "configProps": {
          "hive.metastore.uris": "thrift://localhost:9083",
          "hive.metastore.sasl.enabled": "false",
          "hbase.zookeeper.quorum": "localhost",
          "hbase.zookeeper.property.clientPort": "5181",
          "hive.mapred.supports.subdirectories": "true",
          "mapred.input.dir.recursive": "true"
        "enabled": true

      Jira scope
      This Jira aims to add new session option to Drill store.hive.conf.properties which will allow user to specify hive properties at session level.
      User should write properties in string delimited by new line symbol. Properties values should NOT be set in double-quotes or any other quotes otherwise they would be parsed incorrectly. Property name and value should be separated by =. Each `alter session set` will override previously set properties at session level. If during query Drill couldn't unparse property string, warning will be logged. Properties will be parsed by loading into java.util.Properties. Default value is empty string ("").


      alter session set `store.hive.conf.properties` = 'hive.mapred.supports.subdirectories=true\nmapred.input.dir.recursive=true';


        Issue Links



              arina Arina Ielchiieva
              arina Arina Ielchiieva
              Vitalii Diravka Vitalii Diravka
              0 Vote for this issue
              3 Start watching this issue