Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
-
None
Description
When hive.support.concurrency is enabled if you launch a query that reads data from a partition and writes data into another partition of the same table,
it creates a deadlock.
The worse part is that once the deadlock is active, you can't query the table until it times out.
- How to reproduce :
CREATE TABLE test_table (id INT)
PARTITIONED BY (part STRING)
;
INSERT INTO TABLE test_table PARTITION (part="test")
VALUES (1), (2), (3), (4)
;
INSERT OVERWRITE TABLE test_table PARTITION (part="test2")
SELECT id FROM test_table WHERE part="test1";
Nothing happens, and when doing a SHOW LOCKS in another terminal we get :
| lockid | database | table | partition | lock_state | lock_type | transaction_id | last_heartbeat | acquired_at |
| 3765 | default | test_table | NULL | WAITING | SHARED_READ | NULL | 1440603633148 | NULL |
| 3765 | default | test_table | part=test2 | WAITING | EXCLUSIVE | NULL | 1440603633148 | NULL |
This was tested on Hive 1.1.0-cdh5.4.2 but I believe the bug is still presents in 1.2.1.
I could not reproduce it easily locally because it requires a pseudo-distributed setup with zookeeper to have concurrency enabled.
From looking at the code I believe the problem comes from the EmbeddedLockManager method
public List<HiveLock> lock(List<HiveLockObj> objs, int numRetriesForLock, long sleepTime)
that keeps trying to acquire two incompatible locks, and ends up failing after
hive.lock.numretries*hive.lock.sleep.between.retries which by defaut is 100*60s = 100 minutes.