Uploaded image for project: 'Mesos'
  1. Mesos
  2. MESOS-920

Set GLOG_drop_log_memory=false in environment prior to logging initialization.

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.15.0, 0.16.0
    • Fix Version/s: 0.24.2, 0.25.1, 0.26.1, 0.27.0
    • Component/s: None
    • Labels:
    • Target Version/s:
    • Sprint:
      Mesosphere Sprint 27
    • Story Points:
      2

      Description

      We've observed issues where the masters are slow to respond. Two perf traces collected while the masters were slow to respond:

       25.84%  [kernel]                [k] default_send_IPI_mask_sequence_phys
       20.44%  [kernel]                [k] native_write_msr_safe
        4.54%  [kernel]                [k] _raw_spin_lock
        2.95%  libc-2.5.so             [.] _int_malloc
        1.82%  libc-2.5.so             [.] malloc
        1.55%  [kernel]                [k] apic_timer_interrupt
        1.36%  libc-2.5.so             [.] _int_free
      
       29.03%  [kernel]                [k] default_send_IPI_mask_sequence_phys
        9.64%  [kernel]                [k] _raw_spin_lock
        7.38%  [kernel]                [k] native_write_msr_safe
        2.43%  libc-2.5.so             [.] _int_malloc
        2.05%  libc-2.5.so             [.] _int_free
        1.67%  [kernel]                [k] apic_timer_interrupt
        1.58%  libc-2.5.so             [.] malloc
      

      These have been found to be attributed to the posix_fadvise calls made by glog. We can disable these via the environment:

      GLOG_DEFINE_bool(drop_log_memory, true, "Drop in-memory buffers of log contents. "
                       "Logs can grow very quickly and they are rarely read before they "
                       "need to be evicted from memory. Instead, drop them from memory "
                       "as soon as they are flushed to disk.");
      
      
          if (FLAGS_drop_log_memory) {
            if (file_length_ >= logging::kPageSize) {
              // don't evict the most recent page
              uint32 len = file_length_ & ~(logging::kPageSize - 1);
              posix_fadvise(fileno(file_), 0, len, POSIX_FADV_DONTNEED);
            }
          }
      

      We should set GLOG_drop_log_memory=false prior to making our call to google::InitGoogleLogging, to avoid others running into this issue.

        Issue Links

          Activity

          Hide
          bmahler Benjamin Mahler added a comment -

          Looks like a fix was upstreamed, possibly making it safe to use use the drop_log_memory flag:
          https://github.com/google/glog/pull/145

          Show
          bmahler Benjamin Mahler added a comment - Looks like a fix was upstreamed, possibly making it safe to use use the drop_log_memory flag: https://github.com/google/glog/pull/145
          Hide
          neilc Neil Conway added a comment -

          FYI, I opened https://github.com/google/glog/issues/84 for this issue with glog upstream.

          Show
          neilc Neil Conway added a comment - FYI, I opened https://github.com/google/glog/issues/84 for this issue with glog upstream.
          Hide
          karya Kapil Arya added a comment -

          commit 363b0b059bdc7742b2258a33ebfe430fd03f4311
          Author: Kapil Arya <kapil@mesosphere.io>
          Date: Mon Jan 25 00:41:17 2016 -0500

          Fixed non-linux build involving glog drop_log_meory flag.

          The variable "FLAGS_drop_log_memory" is available only on Linux.

          Review: https://reviews.apache.org/r/42704

          Show
          karya Kapil Arya added a comment - commit 363b0b059bdc7742b2258a33ebfe430fd03f4311 Author: Kapil Arya <kapil@mesosphere.io> Date: Mon Jan 25 00:41:17 2016 -0500 Fixed non-linux build involving glog drop_log_meory flag. The variable "FLAGS_drop_log_memory" is available only on Linux. Review: https://reviews.apache.org/r/42704
          Hide
          qiujian Jian Qiu added a comment -

          It works for me, thanks!

          Show
          qiujian Jian Qiu added a comment - It works for me, thanks!
          Hide
          neilc Neil Conway added a comment -

          That patch fixes the build issue for me.

          Show
          neilc Neil Conway added a comment - That patch fixes the build issue for me.
          Hide
          karya Kapil Arya added a comment - - edited

          Here is the RR: https://reviews.apache.org/r/42704/

          Jian Qiu: Can you take a look and see if this fixes the build issue? I don't have a OS X machine handy.

          Thanks!

          Show
          karya Kapil Arya added a comment - - edited Here is the RR: https://reviews.apache.org/r/42704/ Jian Qiu : Can you take a look and see if this fixes the build issue? I don't have a OS X machine handy. Thanks!
          Hide
          karya Kapil Arya added a comment -

          Looks like a platform issue. I'll create a RR right away.

          Show
          karya Kapil Arya added a comment - Looks like a platform issue. I'll create a RR right away.
          Hide
          qiujian Jian Qiu added a comment -

          When I run make check on OSX, it appears a error message.

          ./mesos-tests
          dyld: Symbol not found: __ZN3fLB21FLAGS_drop_log_memoryE
            Referenced from: /Users/qiujian/Documents/mesos/build/src/.libs/libmesos-0.27.0.dylib
            Expected in: flat namespace
           in /Users/qiujian/Documents/mesos/build/src/.libs/libmesos-0.27.0.dylib
          make[3]: *** [check-local] Trace/BPT trap: 5
          make[2]: *** [check-am] Error 2
          make[1]: *** [check] Error 2
          make: *** [check-recursive] Error 1
          

          Seems to be related to this ticket?

          Show
          qiujian Jian Qiu added a comment - When I run make check on OSX, it appears a error message. ./mesos-tests dyld: Symbol not found: __ZN3fLB21FLAGS_drop_log_memoryE Referenced from: /Users/qiujian/Documents/mesos/build/src/.libs/libmesos-0.27.0.dylib Expected in: flat namespace in /Users/qiujian/Documents/mesos/build/src/.libs/libmesos-0.27.0.dylib make[3]: *** [check-local] Trace/BPT trap: 5 make[2]: *** [check-am] Error 2 make[1]: *** [check] Error 2 make: *** [check-recursive] Error 1 Seems to be related to this ticket?
          Hide
          karya Kapil Arya added a comment -

          commit 9ca66cab1251d58dd202ea11884d557aa10e47c1
          Author: Kapil Arya <kapil@mesosphere.io>
          Date: Tue Jan 19 15:08:26 2016 -0500

          Disabled "drop_log_memory" flag for glog.

          When this flag is set, glog makes several hundred `posix_fadvise(...,
          POSIX_FADV_DONTNEED)` calls per second to advise the kernel to drop
          in-memory buffers related to log contents.

          Review: https://reviews.apache.org/r/42518

          Show
          karya Kapil Arya added a comment - commit 9ca66cab1251d58dd202ea11884d557aa10e47c1 Author: Kapil Arya <kapil@mesosphere.io> Date: Tue Jan 19 15:08:26 2016 -0500 Disabled "drop_log_memory" flag for glog. When this flag is set, glog makes several hundred `posix_fadvise(..., POSIX_FADV_DONTNEED)` calls per second to advise the kernel to drop in-memory buffers related to log contents. Review: https://reviews.apache.org/r/42518
          Hide
          neilc Neil Conway added a comment -

          Can we report this issue to glog upstream?

          Show
          neilc Neil Conway added a comment - Can we report this issue to glog upstream?
          Hide
          karya Kapil Arya added a comment -
          Show
          karya Kapil Arya added a comment - Here is the RR: https://reviews.apache.org/r/42518/

            People

            • Assignee:
              karya Kapil Arya
              Reporter:
              bmahler Benjamin Mahler
              Shepherd:
              Benjamin Mahler
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development

                  Agile