[IMPALA-6816] Statestore spends a lot of time in GetMinSubscriberTopicVersion()

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: Impala 3.0, Impala 2.12.0
• Fix Version/s: Impala 2.13.0, Impala 3.1.0
• Component/s: Distributed Exec
• Labels: admission-control statestore
• Target Version: Impala 3.1.0
• Epic Color: ghx-label-1
• Assignee: Tim Armstrong
• Reporter: Tim Armstrong
• Created: 05/Apr/18 21:54
• Updated: 25/Feb/19 07:57
• Resolved: 27/Jun/18 01:06

Description

Samples: 13K of event 'cycles', Event count (approx.): 1200870513
  20.23%      statestored  impalad              [.] impala::Statestore::GetMinSubscriberTopicVersion(std::string const&, std::string*)
   7.68%      statestored  [kernel.kallsyms]    [k] find_busiest_group
   3.46%      statestored  impalad              [.] impala::Statestore::Subscriber::LastTopicVersionProcessed(std::string const&) const
   3.26%      statestored  libc-2.12.so         [.] __memcmp_sse4_1
   1.41%      statestored  [kernel.kallsyms]    [k] find_next_bit
   1.40%      statestored  [kernel.kallsyms]    [k] cpumask_next_and
   1.21%      statestored  libpthread-2.12.so   [.] pthread_mutex_lock
   1.04%      statestored  libc-2.12.so         [.] memcpy
   1.01%      statestored  [kernel.kallsyms]    [k] _spin_lock
   0.98%      statestored  impalad              [.] 0x000000000088f903
   0.93%      statestored  impalad              [.] 0x000000000088f8f5
   0.91%      statestored  impalad              [.] 0x000000000088f8ea
   0.85%      statestored  [kernel.kallsyms]    [k] ixgbe_xmit_frame_ring
   0.77%      statestored  impalad              [.] 0x000000000088f8e3
   0.75%      statestored  impalad              [.] 0x000000000088f900
   0.75%      statestored  impalad              [.] impala::Statestore::IsPrioritizedTopic(std::string const&)
   0.73%      statestored  impalad              [.] 0x000000000088f8fa
   0.72%      statestored  impalad              [.] operator new[](unsigned long)
   0.68%      statestored  [kernel.kallsyms]    [k] tcp_recvmsg
   0.67%      statestored  impalad              [.] 0x000000000088f8fd
   0.66%      statestored  impalad              [.] impala::Statestore::Topic::BuildDelta(std::string const&, long, impala::TTopicDelta*)
   0.61%      statestored  [kernel.kallsyms]    [k] thread_return
   0.60%      statestored  impalad              [.] 0x000000000088f8f2
   0.60%      statestored  libstdc++.so.6       [.] std::string::compare(std::string const&) const
   0.59%      statestored  impalad              [.] 0x000000000088f8e6
   0.56%      statestored  impalad              [.] 0x000000000088f8ee
   0.56%      statestored  libcrypto.so.1.0.1e  [.] aesni_encrypt
   0.55%      statestored  impalad              [.] 0x000000000088f8e0
   0.55%      statestored  [kernel.kallsyms]    [k] tcp_transmit_skb
   0.53%      statestored  [kernel.kallsyms]    [k] fget_light
   0.51%      statestored  impalad              [.] std::_Rb_tree<void*, std::pair<void* const, std::shared_ptr<impala::ThriftClientImpl> >, std::_Select1st<std::pair<void* const, std::shared_ptr<impala::ThriftCl
   0.50%      statestored  impalad              [.] apache::thrift::transport::TVirtualTransport<apache::thrift::transport::TBufferedTransport, apache::thrift::transport::TBufferBase>::readAll_virt(unsigned char*
   0.50%      statestored  impalad              [.] impala::Statestore::DoSubscriberUpdate(impala::Statestore::UpdateKind, int, impala::Statestore::ScheduledSubscriberUpdate const&)
   0.49%      statestored  libssl.so.1.0.1e     [.] tls1_enc
   0.48%      statestored  libssl.so.1.0.1e     [.] ssl3_read_bytes

We are spending most of our time computing this for non-catalog topics, where it's not even used.

There are a couple of ways we could fix this that I can think of:

• Avoid including this information for topics where we're not interested in it
• Cache or precompute the value somehow to avoid iterating over all subscribers every time