[KUDU-1477] Pending COMMIT message for failed write operation can prevent tablet startup

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.9.0
• Fix Version/s: 0.9.1, 0.10.0
• Component/s: tablet
• Target Version/s: 1.0.0
• Code Review: http://gerrit.cloudera.org:8080/3321
• Assignee: Todd Lipcon
• Reporter: Todd Lipcon
• Created: 03/Jun/16 16:10
• Updated: 12/Aug/16 04:51
• Resolved: 12/Aug/16 04:51

Description

I had a workload running which generated a lot of writes for duplicate keys. In this case, a commit message only has bad statuses, and no mutated stores. This triggers the following code path:

      if (entry.second->commit().op_type() == WRITE_OP &&
          !AreAnyStoresActive(entry.second->commit())) {
        DumpReplayStateToLog(state);
        TabletSuperBlockPB super;
        WARN_NOT_OK(meta_->ToSuperBlock(&super), "Couldn't build TabletSuperBlockPB.");
        return Status::Corruption(Substitute("CommitMsg was pending but it did not refer "
            "to any active memory stores. Commit: $0. TabletMetadata: $1",
            entry.second->commit().ShortDebugString(), super.ShortDebugString()));
      }