[HIVE-12991] LLAP: Add interface to allow other applications to retrieve data

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Unassigned
• Reporter: Jason Dere
• Created: 03/Feb/16 19:06
• Updated: 03/Feb/16 19:06

Description

Expose an interface for retrieving data from LLAP. This could be used by applications such as Pig or Spark if they wish to get their data using LLAP.

Will use the LLAP branch to commit progress.

Sub-Tasks

• Rebase LLAP branch with master: Resolved
• Create initial InputFormat + record readers/writers: Resolved
• Add client to communicate with interface, initial split setup: Resolved
• Wire the client to submit execution fragments: Resolved
• Make HiveSplitGenerator usable independent of Tez: Open
• JDBC split refactoring and handle some edge cases: Resolved
• Fixes for LlapDump and FileSinkoperator: Resolved
• Merge master into llap branch: Resolved
• LLAP: Add end-to-end test for LlapInputFormat: Resolved
• LLAP: Replace use of ServerSocket with netty in LlapOutputFormatService: Resolved
• Merge master into llap branch: Resolved
• LlapInputFormat should get LLAP ports from the LLAP service registry: Resolved
• LlapTaskUmbilicalExternalClient should handle submission rejection/failures/timeouts from LLAP daemon: Resolved
• LLAP external submission client ends up attempting to find an LLAP instance based on the submitting user instead of the hive user: Resolved
• Merge master into llap branch: Resolved
• LLAP output format service not actually registered in LLAP registry: Resolved
• Allow LlapRecordReader to parse/output rows: Resolved
• Move around some of the classes created during llap branch work: Resolved
• Get rid of wrapped LlapInputSplit/InputFormat classes: Resolved
• Update LlapDump: Resolved
• Misc cleanup on llap branch: Resolved
• Merge llap branch work to master: Closed
• LLAP: Work out the details of LlapTaskUmbilicalExternalClient tokenIdentifier: Open
• LLAP: Other todo items in LlapTaskUmbilicalExternalClient: Open
• LlapBaseInputFormat: figure out where credentials come from: Open
• Use loginUser from UGI to get llap user when generating LLAP splits.: Resolved
• LLAP: handle case where no service instance is found on the host specified in the input split: Closed
• usingTezAm field not required in LLAP SubmitWorkRequestProto: Resolved
• LlapOutputFormatService port should be able to be set via conf: Resolved
• Fix TestLlapOutputFormat: Resolved
• Security for LlapOutputFormatService: Resolved
• LlapTaskUmbilicalExternalClient should be closed by the record reader: Closed
• HIVE_RPC_QUERY_PLAN should always be set when generating LLAP splits: Closed
• NPE in TaskLocationHints during LLAP GetSplits request: Closed
• LLAP: external client output is writing to channel before it is writable again: Closed
• LLAP input split should get task attempt number from conf if available: Closed
• LLAP output format connection should wait for all writes to finish before closing channel: Closed
• LLAP external recordreader not returning non-ascii string properly: Closed
• Disable LlapZookeeperRegistry ZK auth setup for external clients: Closed
• LLAP external client on secure cluster: Protocol interface org.apache.hadoop.hive.llap.protocol.LlapTaskUmbilicalProtocol is not known: Closed
• LLAP input format should escape the query string being passed to getSplits(): Closed
• Clean up driver instance in get_splits: Closed
• Support complex types in external LLAP InputFormat: Closed
• Improve end-of-data checking for LLAP input format: Closed
• LlapProtocolClientProxy stack trace when using llap input format: Closed
• LlapInputFormat: Seeing "output error" WARN message: Closed
• Add test for more datatypes for LlapInputFormat: Closed
• Use LazyBinarySerDe for LLAP InputFormat: Closed
• LlapTaskUmbilicalExternalClient should not start new umbilical server for every fragment request: Closed
• "Timed out getting readerEvents" error from external LLAP client: Closed
• External LLAP client: connection to HS2 should be kept open until explicitly closed: Closed
• Set current database for external LLAP interface: Closed
• External LLAP client: acquire locks for tables queried directly by LLAP: Closed