[HIVE-11228] Mutation API should use semi-shared locks.

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 2.0.0
• Component/s: HCatalog
• Labels: streaming_api transaction transactions
• Target Version/s: 1.3.0
• Hadoop Flags: Incompatible change
• Release Note: Streaming mutation API uses semi-shared locks.
• Flags: Patch
• Assignee: Elliot West
• Reporter: Elliot West
• Created: 10/Jul/15 13:59
• Updated: 16/Feb/16 23:52
• Resolved: 13/Jul/15 16:45

Description

An issue was identified by ekoifman in the streaming mutation API (HIVE-10165) where an insufficiently restrictive lock was being used when issuing updates and deletes to ACID tables and partitions. A shared lock was being used where in fact a semi-shared lock is required. Additionally, the current lock scope targets the entire table, whereas in theory if the table is partitioned, then only the affected partitions are required to participate in the semi-shared lock. However, there are a couple of technical challenges that prevent the locks currently being applied on a per-partition basis:

• It is expected that the affected partitions are not known in advance so individual partition locks would need to be acquired as needed.
• The API is expected to execute in a clustered environment and so acquiring these locks as on an ‘as needed’ basis presents a risk that the meta store may become overwhelmed. This is expected to be less of an problem when an HBase based meta store is introduced (HIVE-9452).
• My understanding is that multiple fine grained lock acquisitions for a single transaction are not possible at present. When they are available they’ll introduce the possibility that deadlocks can occur. This should be better handled when HIVE-9675 is complete.

Therefore, as advised, at this time the system will obtain a semi-shared lock on participating tables. Although this will prevent other concurrent writes, it will preserve snapshot isolation when reading.

Issue Links

• relates to: HIVE-10165 Improve hive-hcatalog-streaming extensibility and support updates and deletes.