Details
Major Description
During the minor compaction, the zookeeper lock for the tablet server is double-checked prior to updating the METADATA table information. In one unlucky moment, the zookeeper connection was lost during this check. The tablet server failed the check, but the lock was not lost. As a result, the root tablet remained hosted for another 4 weeks, but did not flush mutations to disk. When memory filled, the operator noticed a long hold time and killed the server. This caused a log recovery of 98 1G of logs, some of which were very old.