[HIVE-24058] Llap external client - Enhancements for running in cloud environment

• Type: Task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: llap
• Assignee: Shubham Chaurasia Assign to me
• Reporter: Shubham Chaurasia
• Created: 23/Aug/20 09:11
• Updated: 23/Aug/20 09:11

Description

When we query using llap external client library, following happens currently -

1. We first need to get splits using LlapBaseInputFormat#getSplits(), this just needs Hive server JDBC url.

2. We then submit those splits to llap and obtain record reader to read data using LlapBaseInputFormat#getRecordReader(). In this step we need following at client side -

• hive.zookeeper.quorum
  -hive.llap.daemon.service.hosts

We need to connect to zk to discover llap daemons.

3. Record reader so obtained needs to initiate a TCP connection from client to LLAP Daemon to submit the split.

4. It also needs to initiate another TCP connection from client to output format port in LLAP Daemon to read the data.

In cloud based deployments, we may not be able to make direct connections to Zk registry and LLAP daemons from client as it might run outside vpc.

For 2, we can move daemon discovery logic to get_splits UDF itself which will run in HS2.
For scenarios like 3 and 4, we can expose additional ports on LLAP with additional auth mechanism.

Sub-Tasks

• Llap external client - Initial changes for running in cloud environment: Resolved
• Spark job fails with fixed LlapTaskUmbilicalServer port: Closed
• Node heartbeats from LLAP Daemon to the client are not matching leading to timeout in cloud environment: Closed
• Check if we can interchange client and server sides for umbilical for external client flow: Open