[IMPALA-12666] Statestore topic metrics are wrong when impalads are shutdown

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Impala 4.4.0
• Component/s: Backend
• Epic Color: ghx-label-8
• Assignee: Quanlong Huang
• Reporter: Quanlong Huang
• Created: 02/Jan/24 01:14
• Updated: 22/Jan/24 12:51
• Resolved: 22/Jan/24 12:51

Description

While looking into the statestore codes, I found we incorrectly update the metrics in Statestore::Topic::DeleteIfVersionsMatch()

void Statestore::Topic::DeleteIfVersionsMatch(TopicEntry::Version version,
    const Statestore::TopicEntryKey& key) {
  // Acquire exclusive lock - we are modifying the topic.
  lock_guard<shared_mutex> write_lock(lock_);
  TopicEntryMap::iterator entry_it = entries_.find(key);
  if (entry_it != entries_.end() && entry_it->second.version() == version) {
    // Add a new entry with the the version history for this deletion and remove the old
    // entry
    topic_update_log_.erase(version);
    topic_update_log_.emplace(++last_version_, key);
    value_size_metric_->Increment(entry_it->second.value().size()); // We shouldn't bump this since nothing is added into the entrics_ map.
    topic_size_metric_->Increment(entry_it->second.value().size()); // Same here.
    entry_it->second.SetDeleted(true);
    entry_it->second.SetVersion(last_version_);
  }
}

value_size_metric_ and topic_size_metric_ are shown in the "/metrics" page as "statestore.total-value-size-bytes" and "statestore.total-topic-size-bytes".

Each topic has two counters: total_key_size_bytes_ and total_value_size_bytes_.
value_size_metric_ tracks the sum of total_value_size_bytes_ across all topics. We shouldn't bump the metric here since we just mark a topic entry as deleted but not removing it. Note that in other places, we will always bump total_value_size_bytes_ and value_size_metric_. This is the only place that we just bump value_size_metric_ without total_value_size_bytes_ so it's wrong.