Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-24058

Llap external client - Enhancements for running in cloud environment

    XMLWordPrintableJSON

    Details

    • Type: Task
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: llap
    • Labels:
      None

      Description

      When we query using llap external client library, following happens currently -

      1. We first need to get splits using LlapBaseInputFormat#getSplits(), this just needs Hive server JDBC url.

      2. We then submit those splits to llap and obtain record reader to read data using LlapBaseInputFormat#getRecordReader(). In this step we need following at client side -

      • hive.zookeeper.quorum
        -hive.llap.daemon.service.hosts

      We need to connect to zk to discover llap daemons.

      3. Record reader so obtained needs to initiate a TCP connection from client to LLAP Daemon to submit the split.

      4. It also needs to initiate another TCP connection from client to output format port in LLAP Daemon to read the data.

      In cloud based deployments, we may not be able to make direct connections to Zk registry and LLAP daemons from client as it might run outside vpc.

      For 2, we can move daemon discovery logic to get_splits UDF itself which will run in HS2.
      For scenarios like 3 and 4, we can expose additional ports on LLAP with additional auth mechanism.

        Attachments

          Activity

            People

            • Assignee:
              ShubhamChaurasia Shubham Chaurasia
              Reporter:
              ShubhamChaurasia Shubham Chaurasia
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 3.5h
                3.5h