Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
2.2.0
-
None
Description
Pre-commit jobs are failing currently due to out of disk space. The issue is happening due to huge size of hive.log when TestDbTxnManager test fails or times-out. When this test fails or times-out Ptest tries to persist these logs for debugging. Since this test has been timing out frequently, this accumulates a lot of these log files and eventually Ptest server runs of disk space. Each run of TestDbTxnManager is generating ~30 GB of hive.log. I tried to run it locally and it quickly reached 7 GB until I had to cancel it.
The issue seems to be coming from this code block in TxnHandler.java
if(LOG.isDebugEnabled()) { LOG.debug("Locks to check(full): "); for(LockInfo info : locks) { LOG.debug(" " + info); } }
We should either change it trace or change the log mode of this test to INFO so that it generates smaller log files.
Attachments
Attachments
Issue Links
- is broken by
-
HIVE-13335 tune TxnHandler.performTimeOuts() batch size
-
- Resolved
-