Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-6575

Add store.hive.conf.properties option to allow set Hive properties at session level

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.13.0
    • Fix Version/s: 1.14.0
    • Component/s: None

      Description

      Use case

      Hive external table ddl:

      create external table my(key int, val string)
      row format delimited
      fields terminated by ','
      stored as textfile
      location '/data/my_tbl';
      

      Path /data/my_tb contains sub directory and file in it: /data/my_tbl/sub_dir/data.txt with the following data:

      1, value_1
      2, value_2
      

      When querying such table from Hive, user gets the following exception:

      Failed with exception java.io.IOException:java.io.IOException: Not a file: file:///data/my_tbl/sub_dir
      

      To be able to query this table user needs to set two properties to true: hive.mapred.supports.subdirectories and mapred.input.dir.recursive.
      They can be set at system level in hive-site.xml or at session in Hive console:

      set hive.mapred.supports.subdirectories=true;
      set mapred.input.dir.recursive=true;
      

      Currently to be able to query such table from Drill, user can specify this properties in Hive plugin only:

      {
        "type": "hive",
        "configProps": {
          "hive.metastore.uris": "thrift://localhost:9083",
          "hive.metastore.sasl.enabled": "false",
          "hbase.zookeeper.quorum": "localhost",
          "hbase.zookeeper.property.clientPort": "5181",
          "hive.mapred.supports.subdirectories": "true",
          "mapred.input.dir.recursive": "true"
        }
        "enabled": true
      }
      

      Jira scope
      This Jira aims to add new session option to Drill store.hive.conf.properties which will allow user to specify hive properties at session level.
      User should write properties in string delimited by new line symbol. Properties values should NOT be set in double-quotes or any other quotes otherwise they would be parsed incorrectly. Property name and value should be separated by =. Each `alter session set` will override previously set properties at session level. If during query Drill couldn't unparse property string, warning will be logged. Properties will be parsed by loading into java.util.Properties. Default value is empty string ("").

      Example:

      alter session set `store.hive.conf.properties` = 'hive.mapred.supports.subdirectories=true\nmapred.input.dir.recursive=true';
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                arina Arina Ielchiieva
                Reporter:
                arina Arina Ielchiieva
                Reviewer:
                Vitalii Diravka
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: