[HIVE-7926] long-lived daemons for query fragment execution, I/O and caching

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Labels: TODOC2.0
• Target Version/s: 2.0.0

Release Note

LLAP is the new hybrid execution model that enables efficiencies across queries, such as caching of columnar data, JIT-friendly operator pipelines, and reduced overhead for multiple queries (including concurrent queries), as well as new performance features like asynchronous I/O, pre-fetching and multi-threaded processing. The hybrid model consists of a long-lived service interacting with on-demand elastic containers serving as a tightly integrated DAG-based framework for query execution.

The first version of LLAP is being shipped in Hive 2.0 release. The component has been extensively exercised on test and live clusters, and tested, but is expected to have rough edges in this initial release.
The current limitations are: supported with Tez only; does not support ACID tables; the I/O elevator and cache only support ORC format and vectorized execution.

• Assignee: Sergey Shelukhin
• Reporter: Sergey Shelukhin
• Created: 01/Sep/14 05:23
• Updated: 07/Oct/16 16:53
• Resolved: 18/Nov/15 20:25

Description

We are proposing a new execution model for Hive that is a combination of existing process-based tasks and long-lived daemons running on worker nodes. These nodes can take care of efficient I/O, caching and query fragment execution, while heavy lifting like most joins, ordering, etc. can be handled by tasks.
The proposed model is not a 2-system solution for small and large queries; neither it is a separate execution engine like MR or Tez. It can be used by any Hive execution engine, if support is added; in future even external products (e.g. Pig) can use it.

The document with high-level design we are proposing will be attached shortly.

Sub-Tasks

• LLAP: refactor prototype code to salvage some parts: Closed
• LLAP: introduce low-level cache for ORC: Closed
• LLAP: ORC production of encoded data, cache usage: Closed
• LLAP: ORC decoding of row-groups: Resolved
• LLAP: separate decoding thread from read/uncompress thread: Open
• LLAP: implement pause and stop for async data production: Open
• LLAP: row-level vectorized SARGs: Patch Available
• LLAP: improve high-level cache from prototype: Resolved
• LLAP: Fix some static vars in the operator pipeline: Resolved
• LLAP: Enable local mode tests on tez to facilitate llap testing: Resolved
• LLAP: Add an execution daemon: Closed
• LLAP: (IO) consider using reactive framework to string together parts of code: Open
• LLAP: I'm the decider: Resolved
• LLAP: Widen the scope of ORC readers and related classes: Resolved
• LLAP: create a reasonable q file test for ORC IO: Closed
• LLAP: initialize IO during service startup, with service configuration: Closed
• LLAP: add check for udfs/udafs to llapdecider: Resolved
• LLAP: add heap mode to allocator (for q files, YARN w/o direct buffer accounting support): Resolved
• LLAP: design and implement proper metadata cache: Closed
• LLAP: ORC decoding of row groups for complex types: Open
• LLAP: Merge from trunk to llap branch 2/20/2015: Closed
• avoid log locks in operators: Closed
• LLAP: Fix issue with reading last row group of string column: Closed
• LLAP: use log4j 2 for llap (log to separate files, etc.): Closed
• LLAP: Reuse string dictionaries per column per stripe when processing row groups: Closed
• LLAP: Update launcher, scheduler to work with Tez changes: Closed
• LLAP: Misc fixes to launch scripts, startup error handling: Closed
• LLAP: hook up decider + dag utils: Resolved
• LLAP: Update tez dependency in branch: Closed
• LLAP: uber mode where applicable: Resolved
• LLAP: fix wrapinputformat for client/server; fix server init to use config: Closed
• LLAP: Make sure jar containing the Tez plugins is added as a local resource for the AM: Closed
• LLAP: Add a MiniLLAPCluster for tests: Closed
• LLAP: add simple way to determine wether you're running in daemon: Resolved
• LLAP: Add an option to disable uberization: Closed
• LLAP: add config var for uber mode: Resolved
• LLAP: InStream creation for every row group is expensive: Closed
• LLAP: config not passed to daemon init: Closed
• LLAP: need metrics for llap daemon: Closed
• LLAP: consider specialized "transient" metadata cache: Open
• LLAP: Add event logging for execution elements: Closed
• LLAP: Push work into daemons instead of the current pull: Closed
• prep object registry for multi threading: Closed
• LLAP: JMX web-service end points for monitoring & metrics: Closed
• LLAP: Distinguish between memory used by executors and the cache: Closed
• LLAP: Use a share-nothing scoreboard /status implementation: Resolved
• LLAP: Native Vectorization of Map Join: Closed
• LLAP: Make stripe level column readers thread safe: Closed
• LLAP: fix unit tests: Closed
• LLAP: Decision to use llap or uber is being lost in some reducers: Closed
• LLAP: instrument logging for fragments: Closed
• LLAP: change caches to use unique file ID, not name: Closed
• LLAP: add multi threaded object registry: Resolved
• documentation for llap: Open
• LLAP: Add executor threadpool metrics: Resolved
• LLAP: Fix exception with HDFS getFileInfo: Closed
• LLAP: Print ASCII art of LLAP during daemon startup: Closed
• LLAP: Slider deploy scripts: Closed
• LLAP: Add llap-daemon to the Tez session if the execution mode is LLAP: Closed
• LLAP: Add LLAP_DAEMON_LD_PATH to add native libs in: Closed
• LLAP: Provide per query counters: Resolved
• LLAP: Provide metrics for producer/consumer queue: Resolved
• LLAP: disable plan caching: Resolved
• LLAP: Sort out issues with UGI and cached FileSystems: Closed
• LLAP: Integrate MiniLLAPCluster into tests: Closed
• LLAP: Use service discovery in the scheduler to determine running daemons: Closed
• LLAP: Update usage of APIs changed by TEZ-2175 and TEZ-2187: Closed
• LLAP: Clean up structures and intermediate data when a query completes: Closed
• LLAP: Improvements to the Shuffle handler to avoid unnecessary disk scans: Closed
• LLAP: Avoid fetching data multiple times in case of broadcast: Closed
• LLAP: Vectorized Field-By-Field Serialize / Deserialize to support new Vectorized Map Join: Closed
• LLAP: YARN registry for Auto-organizing Slider instances: Closed
• LLAP: OrcColumnVectorProducer should handle reading isPresent columns only: Closed
• LLAP: LLAP Cached readers for StringDirectTreeReaders over-read data: Closed
• LLAP: EOFException in reader: Closed
• LLAP: Add rack based scheduling of work: Open
• LLAP: Handle task rejection from daemons in the AM: Closed
• fix yarn service registry not found in ut problem: Open
• MiniTez ut fail with missing configs: Closed
• yarn service registry should be an optional dependency: Resolved
• LLAP: Pass additional JVM args via Slider appConfig: Closed
• LLAP: NegativeArraySize exception on vector string reader: Closed
• LLAP: Hive sessions run before Slider registers to YARN registry fail to launch: Closed
• NPE in LLAP logs in heartbeat: Resolved
• LLAP : investigate showing LLAP IO usage in explain: Closed
• LLAP : add LLAP IO read debug tool: Open
• LLAP: q file test is broken again: Closed
• LLAP: Queued work times out: Closed
• LLAP: AM should get notifications on daemons going down or restarting: Closed
• LLAP: Create a fixed size execution queue for daemons: Closed
• LLAP: Scheduling of work from different queries within the daemon: Closed
• LLAP : llap_partitioned test is broken: Closed
• LLAP: cache counters are incorrect: Resolved
• LLAP : NPE in DiskRangeList helper init: Closed
• LLAP : disable ORC trace logging for now: Resolved
• LLAP: VectorMapJoinOperator gets an over-flow on batchSize of 1024: Closed
• LLAP: LowLevelCacheImpl does not match cached ranges properly: Closed
• LLAP: Cache hit rate could be wrong in some cases: Resolved
• LLAP: read file ID when generating splits to avoid extra NN call in the tasks: Closed
• LLAP: adjust allocation after decompression: Open
• LLAP: Object Estimator digs in via a deep hashCode(): Closed
• LLAP: Make the low-level IO threadpool configurable: Resolved
• LLAP: improve how buffers are locked for split: Closed
• LLAP: enable yourkit profiling of tasks: Patch Available
• LLAP: Cancelling tasks fails to stop cache filling threads: Closed
• LLAP: Generate consistent splits and locations for the same split across jobs: Closed
• LLAP: implement stopping the reader thread: Resolved
• LLAP: port updates from HIVE-9555 to llap branch in preparation for trunk merge: Closed
• LLAP: query 7 produces corrupted result with IO enabled: Resolved
• LLAP: query 17 tasks fail due to mapjoin issue: Resolved
• LLAP: reducers running in LLAP starve out map retries: Closed
• LLAP: Use task number, attempt number to cache plans: Open
• LLAP: Print execution modes in tez in-place UI: Resolved
• LLAP: Port changes to timestamp stream reader after timezone fix in trunk: Resolved
• LLAP: Fix ordering of execution modes: Resolved
• LLAP: BytesBytesMultiHashMap and mapjoin container should reuse refs: Closed
• LLAP: Tez heartbeats are delayed by ~500+ ms due to Hadoop IPC client: Open
• LLAP: GC issues 1: Closed
• LLAP: switch to sensible cache policy: Resolved
• LLAP: Varchar columns are not handled in encoded readers: Closed
• LLAP: ORC file contains compression buffers larger than bufferSize (OR reader has a bug): Closed
• LLAP: Avoid deserializing the plan > 1 times in a single thread: Closed
• LLAP: ORC BIGINT SARGs regressed after Parquet PPD fixes (HIVE-8122): Resolved
• LLAP: general cache deadlock avoidance: Open
• LLAP: ORC MemoryManager is singleton synchronized: Resolved
• LLAP: LLAP IO doesn't get invoked inside MiniTezCluster q tests: Resolved
• LLAP: Failures from TaskRunnerCallable should be reported: Closed
• LLAP: start fetching earlier: Closed
• Move fragment execution onto a thread pool: Open
• LLAP: Support caching of uncompressed ORC data: Closed
• LLAP: Loglevel for daemons as a startup option: Closed
• LLAP: Clean up encoded ORC tree readers after trunk merge: Resolved
• LLAP: Certain errors are not reported to the AM when a fragment fails: Closed
• LLAP: Make use of the timed version of getDagStatus in TezJobMonitor: Closed
• LLAP: Fix merge conflicts related to HIVE-10067: Resolved
• LLAP: orc_llap test fails again: Resolved
• LLAP: Explain plan diff in llap_partitioned.q test case: Resolved
• LLAP: Allow the runtime to check whether a task can run to completion: Closed
• LLAP: Handle errors while sending source state updates to the daemons: Closed
• LLAP: Fix timeouts for tasks which stay in the pending queue: Closed
• LLAP: Cache cleaner throws Negative sleep exception: Closed
• Hadoop reflectionutils has issues: Closed
• LLAP: NPE due to failure to find position: Closed
• LLAP: Self-joins with the cache enabled: Closed
• LLAP: IndexOutOfBound in MapJoinOperator: Closed
• LLAP: small improvement to unlocking cache data: Closed
• LLAP: merge trunk to branch 2015-04-15: Closed
• LLAP: randomness in random machine scheduling is not random enough: Resolved
• LLAP: query80 fails with vectorization cast issue: Resolved
• LLAP: Name IO elevator threads: Resolved
• LLAP: Tez in-place UI gets misaligned when state is TERMINATING: Resolved
• LLAP: query 54 doesn't finish in LLAP: Resolved
• LLAP: Notify AM of pre-emption: Resolved
• LLAP: Implement Tez SplitSizeEstimator for Orc: Resolved
• LLAP: scheduling occasionally schedules all the work on one machine: Resolved
• LLAP: Provide runtime information to daemons to decide on preemption order: Closed
• LLAP: Make use of additional information to determine run/preemption order: Open
• LLAP: NPE in scheduler in case of rejected tasks: Closed
• LLAP: NPE caused by HIVE-10397: Resolved
• LLAP: Factor known capacity into scheduling decisions: Closed
• LLAP: Limit number of threads used to communicate with a single LLAP instance to 1: Closed
• LLAP: Abort hive tez processor on interrupts: Closed
• LLAP: Consider replacing BytesBytesMultiHashMap with new fast hash table code of Native Vector Map Join: Resolved
• LLAP: fix container sizing configuration for memory: Open
• LLAP: NPE in IO when returning 0 rows with no projection: Closed
• LLAP: investigate why TPCH Q1 1k is slow: Resolved
• LLAP: Minor fixes after tez api enhancements for dag completion: Closed
• LLAP: Tez task is interrupted for unknown reason after an IPC exception and then fails to report completion: Closed
• LLAP: AsertionError cannot allocate when reading from orc: Closed
• LLAP: Resolve everything in llap-daemon-site.xml: Closed
• LLAP: Empty PPD splits from Cache throw error: Resolved
• LLAP: Must reset small table result columns for Native Vectorization of Map Join: Closed
• LLAP: Cleanup map join cache when a query completes: Closed
• LLAP: Add test case for all row groups selection with no column projection: Resolved
• LLAP: a different NPE in shuffle: Closed
• LLAP: Native Vector Map Join doesn't handle filtering and matching on LEFT OUTER JOIN repeated key correctly: Closed
• LLAP: Vector row extraction allocates new extractors per process method call instead of just once: Resolved
• LLAP: Invalid containerId prefix: Resolved
• LLAP: allocator occasionally has a spurious failure to allocate due to "partitioned" locking and has to retry: Open
• LLAP: registry; Tez attempted to schedule to daemon that didn't exist: Resolved
• LLAP: AM gets stuck completely if one node is dead: Resolved
• LLAP: AM task communication retry is too long: Closed
• LLAP: registry logs strange lines on daemons: Closed
• LLAP: investigate why GC with IO elevator disabled is so bad: Resolved
• LLAP: Wait queue pre-emption: Closed
• LLAP: Update tez dependency: Closed
• LLAP: Make use of the task runner which allows killing tasks: Closed
• LLAP: Add a mechanism for daemons to inform the AM about killed tasks: Closed
• LLAP: DAGs get stuck at start with no tasks executing: Resolved
• LLAP: DAG got stuck after reducer fetch failed: Resolved
• LLAP: Add counters for time lost per query due to preemption: Open
• LLAP: Log additional debug information in the scheduler: Closed
• LLAP: fix guava stopwatch conflict: Closed
• LLAP: task scheduler thread-count keeps growing: Open
• LLAP: rare NPE in IO: Resolved
• LLAP: dags get stuck in yet another way: Closed
• LLAP: Misc changes to daemon scheduling: Closed
• LLAP: Modify running / wait queues on on fragment finishable state changes: Closed
• LLAP: Add aging to wait queue tasks: Closed
• LLAP: Kill any fragments running in a daemon when a query completes: Closed
• LLAP: Provide current attempt start time for wait queue ordering: Resolved
• LLAP: Wait queue scheduler goes into tight loop: Closed
• LLAP: NPE when calling abort on the TezProcessor: Closed
• LLAP: Exception reported while trying to kill a task: Open
• LLAP: Improve the way task finishable information is processed: Closed
• LLAP: perf dips with IO elevator disabled: Resolved
• LLAP: add pre-fragment and per-table cache details: Closed
• LLAP: Utilities::gWorkMap needs to be cleaned in HiveServer2: Closed
• LLAP: Daemons should shutdown in case of fatal errors: Closed
• LLAP: DAGs get stuck in yet another way: Closed
• LLAP: preemption in AM due to failures / out of order scheduling: Closed
• LLAP: Better handling of hostnames when sending heartbeats to the AM: Open
• LLAP: Serialize handling of requests / events for a query within daemons: Open
• LLAP: Merge from trunk May 28th 2015: Closed
• LLAP: bug in split processing in IO elevator causes duplicate rows: Closed
• LLAP: make sure tests pass #1: Closed
• LLAP: Fixes to TaskReporter after recent Tez changes: Closed
• LLAP: HIVE-10778 has NPE: Closed
• LLAP: make sure the branch builds on hadoop-1: part 1 (non-llap): Closed
• LLAP: the return of the stuck DAG: Closed
• LLAP: ArrayIndexOOB exception in new mapjoin: Resolved
• LLAP: merge master into branch: Closed
• LLAP: Exception in InputInitializer when creating HiveSplitGenerator: Resolved
• LLAP: cache QF counters have a wrong value for consumer time: Closed
• LLAP: fix hadoop-1 build for good by removing llap-server from hadoop-1 build: Closed
• LLAP: elevator reads some useless data even if all RGs are eliminated by SARG: Closed
• LLAP: Set java.io.tmpdir correctly for LLAP Slider instance: Closed
• LLAP: merge master to branch: Closed
• LLAP: make ObjectCache for plans work properly in the daemon: Patch Available
• LLAP: expose what's running on the daemon thru JMX: Closed
• LLAP: support parallel query compilation in HS2: Closed
• LLAP: recent optimization introduced wrong assert to elevator causing test to fail: Closed
• LLAP: preemption appears to count against failure count for the task: Resolved
• LLAP: Allow finer control of startup options for LLAP: Closed
• LLAP: ShuffleHandler + Submit work init race condition: Closed
• LLAP: Accept --hiveconf parameters for the LlapServiceDriver: Closed
• LLAP: make sure tests pass #2: Closed
• LLAP: merge master into branch: Closed
• LLAP: Fix LLAP startup issues due to heap rounding errors: Closed
• LLAP: fix TestMiniTezCliDriverLocal on the branch: Resolved
• LLAP: test auto_sortmerge_join_5 on MiniTez fails with NPE: Closed
• LLAP: fix some tests in the branch and revert incorrectly committed changed out files (from HIVE-11014): Closed
• LLAP: some MiniTez tests have result changes compared to master: Closed
• LLAP: MiniTez tez_smb_main, tez_bmj_schema_evolution fail with NPE: Closed
• MiniTez mergejoin test fails with Tez input error (issue in merge join under certain conditions): Closed
• LLAP: disable the flaky TestLlapTaskSchedulerService test: Closed
• LLAP: merge master into branch: Closed
• LLAP: fix TestLlapTaskSchedulerService flakiness: Closed
• LLAP: SIGSEGV in Off-heap decompression routines: Closed
• LLAP: Cache arena size rounding: Closed
• LLAP: merge master into branch: Closed
• LLAP: more out file changes compared to master: Closed
• Fair ordering of fragments in wait queue: Closed
• LLAP: spark out file changes compared to master: Closed
• Remove unused LlapUtils class from ql.io.orc: Closed
• LLAP: clean up shuffle directories when DAG completes: Closed
• LLAP: merge master into branch: Closed
• LLAP: Cache BuddyAllocator throws NPE: Closed
• LLAP: more out file changes compared to master: Closed
• LLAP: occasional NPE in parallel queries in ORC reader: Closed
• LLAP: Fix the LLAP to ORC APIs: Closed
• LLAP: clean up ORC dependencies part 1: Closed
• LLAP: NPE in AMReporter: Closed
• LLAP: TaskExecutorService state is not cleaned up: Closed
• LLAP: PipelineSorter got stuck: Resolved
• LLAP: investigate locality issues: Resolved
• LLAP: Execution order within LLAP daemons should consider query-specific priority assigned to fragments: Closed
• LLAP: Register for finishable state change notifications when adding a task instead of when scheduling it: Closed
• LLAP: clean up ORC dependencies on object pools: Closed
• LLAP: Hybrid Map-join cache returns invalid data: Closed
• LLAP: Fix API usage to work with evolving Tez APIs: Closed
• LLAP: Fix API usage to work with evolving Tez APIs - TEZ-2005: Closed
• LLAP: move LlapConfiguration into HiveConf and document the settings: Closed
• LLAP: clean up ORC dependencies - move encoded reader path into a separate package and reader: Closed
• LLAP: Fix API usage to work with evolving Tez APIs - TEZ-{2651,2652,2653}: Closed
• LLAP: make sure JMX view is correct: Resolved
• LLAP: merge master into branch: Closed
• LLAP: ORC-related refactoring changes broke something: Closed
• Disabling llap cache allocate direct is not honored anymore: Resolved
• LLAP: Fix API usage to work with evolving Tez APIs - TEZ-2678: Closed
• LLAP: tez.runtime.compress doesn't appear to be honored for LLAP: Closed
• LLAP: implement LLAP UI as a separate service - part 1: Closed
• LLAP: move EncodedTreeReaderFactory, TreeReaderFactory bits that rely on orc.encoded, and StreamUtils if needed, to orc.encoded package: Closed
• LLAP: Some counters are incorrect: Closed
• Allow wait queue comparator to be specified as a classname: Closed
• LLAP: switch the branch to Tez master from branch: Closed
• LLAP: make sure tests pass #3: Closed
• LLAP: merge master into branch: Closed
• Create LLAP Monitor Daemon class and launch scripts: Open
• LLAP: merge master into branch: Closed
• LLAP: Update tez version to fix build: Resolved
• LLAP: Merge master into branch: Closed
• TaskExecutorService can reject work even if capacity is available: Closed
• LLAP: merge master into branch: Closed
• LLAP: MiniLlapCluster integration broke hadoop-1 build: Closed
• LLAP: Remove MiniLlapCluster from shim layer after hadoop-1 removal: Closed
• LLAP: merge master into branch: Closed
• LLAP: Create webapp/llap directory for MiniLlapCluster: Resolved
• LLAP: Generate golden files for all MiniLlapCluster tests: Closed
• LLAP: MiniLlapCluster tests are slow: Closed
• LLAP: TaskCommunicator should not register MultiMRInput Inputs for state updates: Closed
• LLAP: Exception is ignored if MiniLlap cluster fails to start: Closed
• LLAP: Merge master into branch: Resolved
• LLAP: Rerunning MiniLlap does not start as web app port is used: Resolved
• LLAP: Fix failing tests in MiniLlapCluster: Closed
• LLAP: Test failures - file.splitmetainfo file not found: Closed
• LLAP: Fix tests related to stats difference: Closed
• LLAP: Analyze and fix vectorization tests: Closed
• LLAP: Generate join0.q golden file for java 8: Resolved
• LLAP: Fix unionDistinct_1.q and union_view.q: Closed
• LLAP: constprog_dpp.q,explainuser_1.q showing explain diff: Closed
• LLAP: Remove fileformat_mix.q.out as it is not part of tez test suite: Closed
• LLAP: Partition column value different for identity_project_remove_skip.q: Closed
• LLAP: Plan is different for insert1.q: Closed
• LLAP: Fix auto_join_nulls.q,auto_sortmerge_join_6.q test diffs: Closed
• LLAP: Update errata.txt for HIVE-11871: Closed
• LLAP: Fix ordering difference in acid_vectorization_partition.q: Closed
• LLAP: Fix discrepancies with metadata_only_queries_with_filters.q: Closed
• LLAP: Remove unused/old golden files: Closed
• LLAP: merge master into branch: Closed
• LLAP: Merge master into branch (for HIVE-11860): Resolved
• LLAP: Update llap golden files after master merge: Closed
• LLAP: Typo in webapps causing initialization error for MiniLlap tests: Closed
• LLAP: Update golden files for orc_llap and llapdecider tests: Closed
• LLAP: Merge branch to hive-2.0: Closed
• LLAP: merge master into branch: Closed
• LLAP: merge master into branch: Closed
• Llap: Disable web app for mini llap tests: Closed
• LLAP: Merge master to branch: Closed
• LLAP: merge master into branch: Closed
• LLAP: Merge master into branch: Closed
• LLAP: merge master into branch: Closed
• LLAP: merge master into branch: Closed
• UGI instances being used in IO elevator threads are incorrect: Closed
• LLAP: merge master into branch: Closed
• Add javadoc for methods added to public APIs: Closed
• LLAP: merge master into branch: Closed
• LLAP: Make explainuser_1.q test consistent: Closed
• LLAP: merge master into branch: Closed
• LLAP: update errata for bad branch commits: Closed
• LLAP: Tez counters for LLAP 2: Resolved