[MESOS-920] Set GLOG_drop_log_memory=false in environment prior to logging initialization.

• Type: Improvement
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.15.0, 0.16.0
• Fix Version/s: 0.24.2, 0.25.1, 0.26.1, 0.27.0
• Labels: tech-debt
• Target Version/s: 0.27.0
• Sprint: Mesosphere Sprint 27
• Story Points: 2
• Assignee: Kapil Arya
• Reporter: Benjamin Mahler
• Shepherd: Benjamin Mahler
• Created: 17/Jan/14 19:09
• Updated: 27/Mar/19 14:57
• Resolved: 25/Jan/16 06:05

Description

We've observed issues where the masters are slow to respond. Two perf traces collected while the masters were slow to respond:

 25.84%  [kernel]                [k] default_send_IPI_mask_sequence_phys
 20.44%  [kernel]                [k] native_write_msr_safe
  4.54%  [kernel]                [k] _raw_spin_lock
  2.95%  libc-2.5.so             [.] _int_malloc
  1.82%  libc-2.5.so             [.] malloc
  1.55%  [kernel]                [k] apic_timer_interrupt
  1.36%  libc-2.5.so             [.] _int_free

 29.03%  [kernel]                [k] default_send_IPI_mask_sequence_phys
  9.64%  [kernel]                [k] _raw_spin_lock
  7.38%  [kernel]                [k] native_write_msr_safe
  2.43%  libc-2.5.so             [.] _int_malloc
  2.05%  libc-2.5.so             [.] _int_free
  1.67%  [kernel]                [k] apic_timer_interrupt
  1.58%  libc-2.5.so             [.] malloc

These have been found to be attributed to the posix_fadvise calls made by glog. We can disable these via the environment:

GLOG_DEFINE_bool(drop_log_memory, true, "Drop in-memory buffers of log contents. "
                 "Logs can grow very quickly and they are rarely read before they "
                 "need to be evicted from memory. Instead, drop them from memory "
                 "as soon as they are flushed to disk.");

    if (FLAGS_drop_log_memory) {
      if (file_length_ >= logging::kPageSize) {
        // don't evict the most recent page
        uint32 len = file_length_ & ~(logging::kPageSize - 1);
        posix_fadvise(fileno(file_), 0, len, POSIX_FADV_DONTNEED);
      }
    }

We should set GLOG_drop_log_memory=false prior to making our call to google::InitGoogleLogging, to avoid others running into this issue.

Issue Links

• is related to: IMPALA-7252 Backport rate limiting of fadvise calls into toolchain glog
• is related to: MESOS-1375 Log rotation capable
• is related to: MESOS-9680 Remove automatic disablement of GLOG_drop_log_memory.