[KUDU-2456] TransactionTracker logs too much when there are many pending transactions

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.8.0
• Fix Version/s: 1.9.0
• Component/s: tablet
• Labels: newbie
• Code Review: https://gerrit.cloudera.org/c/12240
• Assignee: Bankim Bhavsar
• Reporter: Adar Dembo
• Created: 25/May/18 23:13
• Updated: 28/Jan/19 18:50
• Resolved: 28/Jan/19 18:50

Description

It's possible for a tablet to have hundreds or even thousands of pending transactions, at which point the logging of all txns in TransactionTracker::WaitForAllToFinish is wasteful. Here's a snippet I saw in one user's cluster in the wild:

W0521 22:38:05.430140 23532 transaction_tracker.cc:227] TransactionTracker waiting for 1832 outstanding transactions to complete now for 6406.481s
W0521 22:38:33.133323 23550 transaction_tracker.cc:227] TransactionTracker waiting for 1812 outstanding transactions to complete now for 6917.103s
W0521 22:39:09.462368 23539 transaction_tracker.cc:227] TransactionTracker waiting for 1798 outstanding transactions to complete now for 6662.420s
W0521 22:40:43.066598 23542 transaction_tracker.cc:227] TransactionTracker waiting for 3016 outstanding transactions to complete now for 9483.333s
W0521 22:41:39.838529 23545 transaction_tracker.cc:227] TransactionTracker waiting for 3390 outstanding transactions to complete now for 9484.259s
W0521 22:42:12.304265 23549 transaction_tracker.cc:227] TransactionTracker waiting for 1837 outstanding transactions to complete now for 6661.056s
W0521 22:42:21.666013 23532 transaction_tracker.cc:227] TransactionTracker waiting for 1832 outstanding transactions to complete now for 6662.716s
W0521 22:42:49.326387 23550 transaction_tracker.cc:227] TransactionTracker waiting for 1812 outstanding transactions to complete now for 7173.297s
W0521 22:43:25.660738 23539 transaction_tracker.cc:227] TransactionTracker waiting for 1798 outstanding transactions to complete now for 6918.618s
W0521 22:44:59.375663 23542 transaction_tracker.cc:227] TransactionTracker waiting for 3016 outstanding transactions to complete now for 9739.642s

Besides the spamming the log, it's not clear that these enormous lists of transactions are at all useful after accounting for column redaction. We should apply some kind of truncation here; maybe only log the first 10 transactions or something.