[KUDU-2083] MaintenanceManager running_op_ count not decremented if MaintenanceOp::Prepare() fails

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 1.3.2, 1.5.0, 1.2.1, 1.4.1
• Target Version/s: 1.3.2, 1.5.0, 1.2.1, 1.4.1
• Code Review: https://gerrit.cloudera.org/#/c/7610/
• Assignee: David Alves
• Reporter: Samuel Okrent
• Created: 27/Jul/17 22:09
• Updated: 12/Oct/17 20:59
• Resolved: 12/Oct/17 20:59

Description

In MaintenanceManager::RunSchedulerThread(), an op gets selected, running_ops_ is incremented, and Prepare() is called on the op. If Prepare() returns false, the op isn't run, so running_ops_ never gets decremented. If Prepare() ever fails, then this could be a problem, as the maintenance manager compares running_ops_ to the number of operation threads to determine whether or not it can run another operation. Prepare generally doesn't fail, but if Tablet::AlterSchema() is called in between FlushMRSOp::UpdateStats() and FlushMRSOp::Prepare(), that is one instance where Prepare() could potentially fail.

To fix, decrement running_ops_ in the codepath that follows from Prepare() failing.

Issue Links

• is duplicated by: IMPALA-5770 test_kudu_insert failed with "Remote error: Service unavailable: Soft memory limit exceeded"