[HIVE-1557] increase concurrency

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.7.0
• Component/s: Query Processor
• Assignee: Unassigned
• Reporter: Namit Jain
• Created: 18/Aug/10 17:16
• Updated: 14/Feb/11 21:02

Description

Copying Joy's comment from https://issues.apache.org/jira/browse/HIVE-1293

a little bummed that locks need to be held for entire query execution. that could mean a writer blocking readers for hours.
hive's query plans seem to be of two distinct stages:
1. read a bunch of stuff, compute intermediate/final data
2. move final data into output locations

ie. - a single query never reads what it writes (into a final output location). even if #1 and #2 are mingled today - they can easily be put in order.

in that sense - we only need to get shared locks for all read entities involved in #1 to begin with. once phase #1 is done, we can drop all the read locks and get the exclusive locks for all the write entities in #2, perform #2 and quit. that way exclusive locks are held for a very short duration. i think this scheme is similarly deadlock free (now there are two independent lock acquire/release phases - and each of them can lock stuff in lex. order).

Issue Links

• requires: HIVE-1293 Concurrency Model for Hive