[HIVE-7926] long-lived daemons for query fragment execution, I/O and caching

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Labels: TODOC2.0
• Target Version/s: 2.0.0

Release Note

LLAP is the new hybrid execution model that enables efficiencies across queries, such as caching of columnar data, JIT-friendly operator pipelines, and reduced overhead for multiple queries (including concurrent queries), as well as new performance features like asynchronous I/O, pre-fetching and multi-threaded processing. The hybrid model consists of a long-lived service interacting with on-demand elastic containers serving as a tightly integrated DAG-based framework for query execution.

The first version of LLAP is being shipped in Hive 2.0 release. The component has been extensively exercised on test and live clusters, and tested, but is expected to have rough edges in this initial release.
The current limitations are: supported with Tez only; does not support ACID tables; the I/O elevator and cache only support ORC format and vectorized execution.

• Assignee: Sergey Shelukhin
• Reporter: Sergey Shelukhin
• Created: 01/Sep/14 05:23
• Updated: 07/Oct/16 16:53
• Resolved: 18/Nov/15 20:25

Description

We are proposing a new execution model for Hive that is a combination of existing process-based tasks and long-lived daemons running on worker nodes. These nodes can take care of efficient I/O, caching and query fragment execution, while heavy lifting like most joins, ordering, etc. can be handled by tasks.
The proposed model is not a 2-system solution for small and large queries; neither it is a separate execution engine like MR or Tez. It can be used by any Hive execution engine, if support is added; in future even external products (e.g. Pig) can use it.

The document with high-level design we are proposing will be attached shortly.

Sub-Tasks

• LLAP: separate decoding thread from read/uncompress thread: Open
• LLAP: implement pause and stop for async data production: Open
• LLAP: row-level vectorized SARGs: Patch Available
• LLAP: (IO) consider using reactive framework to string together parts of code: Open
• LLAP: ORC decoding of row groups for complex types: Open
• LLAP: consider specialized "transient" metadata cache: Open
• documentation for llap: Open
• LLAP: Add rack based scheduling of work: Open
• fix yarn service registry not found in ut problem: Open
• LLAP : add LLAP IO read debug tool: Open
• LLAP: adjust allocation after decompression: Open
• LLAP: enable yourkit profiling of tasks: Patch Available
• LLAP: Use task number, attempt number to cache plans: Open
• LLAP: Tez heartbeats are delayed by ~500+ ms due to Hadoop IPC client: Open
• LLAP: general cache deadlock avoidance: Open
• Move fragment execution onto a thread pool: Open
• LLAP: Make use of additional information to determine run/preemption order: Open
• LLAP: fix container sizing configuration for memory: Open
• LLAP: allocator occasionally has a spurious failure to allocate due to "partitioned" locking and has to retry: Open
• LLAP: Add counters for time lost per query due to preemption: Open
• LLAP: task scheduler thread-count keeps growing: Open
• LLAP: Exception reported while trying to kill a task: Open
• LLAP: Better handling of hostnames when sending heartbeats to the AM: Open
• LLAP: Serialize handling of requests / events for a query within daemons: Open
• LLAP: make ObjectCache for plans work properly in the daemon: Patch Available
• Create LLAP Monitor Daemon class and launch scripts: Open