ZooKeeper
  1. ZooKeeper
  2. ZOOKEEPER-545

investigate use of realtime gc as the recommened default for server vm

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Critical Critical
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: 3.6.0
    • Component/s: server
    • Labels:

      Description

      We currently don't recommend that ppl use the realtime gc when running the server, we probably should.

      Before we do so we need to verify that it works.
      We should make it the default for all our tests.
      concurrent vs "g2" or whatever it's called (new in 1.6_15 or something?)
      Update all scripts to specify this option
      update documentation to include this option and add section in the dev/ops docs detailing it's benefits (in particular latency effects of gc)

      Also, -server option? any benefit for us to recommend this as well?

        Activity

        Hide
        Jean-Daniel Cryans added a comment -

        Patrick,

        Here's the configuration we currently ship with HBase to solve most of the basic problems seen by new users:

        -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode
        

        Also since 1.6u14 and with 1.7 beta, Java ships with the new G1 GC (paper http://www.research.sun.com/jtech/pubs/04-g1-paper-ismm.pdf) which get rids of JVM pauses. You can use the following options to enable it:

        -XX:+UnlockExperimentalVMOptions -XX:+UseG1GC
        
        Show
        Jean-Daniel Cryans added a comment - Patrick, Here's the configuration we currently ship with HBase to solve most of the basic problems seen by new users: -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode Also since 1.6u14 and with 1.7 beta, Java ships with the new G1 GC (paper http://www.research.sun.com/jtech/pubs/04-g1-paper-ismm.pdf ) which get rids of JVM pauses. You can use the following options to enable it: -XX:+UnlockExperimentalVMOptions -XX:+UseG1GC
        Hide
        Patrick Hunt added a comment -

        I've had alot of luck recently using the latest 1.6.0 jvm (_17) along with CMS/incremental.

        Question is, what should we do, just update the docs or make changes in the source as well. For example we could change
        the tests to run with CMS/Incremental.

        Show
        Patrick Hunt added a comment - I've had alot of luck recently using the latest 1.6.0 jvm (_17) along with CMS/incremental. Question is, what should we do, just update the docs or make changes in the source as well. For example we could change the tests to run with CMS/Incremental.
        Hide
        Patrick Hunt added a comment -

        Still not 100% sure on this one, pushing to 3.4.0.

        Show
        Patrick Hunt added a comment - Still not 100% sure on this one, pushing to 3.4.0.
        Hide
        Mahadev konar added a comment -

        not a blocker. Moving it out of 3.4 release.

        Show
        Mahadev konar added a comment - not a blocker. Moving it out of 3.4 release.
        Hide
        Sean Carr added a comment -

        This seems like a simple issue to resolve and then maybe improve upon later if there is uncertainty as to what the ideal options are. I recently spent a lot of time debugging some nasty production application problems that ended up being a result of long GC pauses in ZooKeeper since ConcMarkSweep is not the default (and I assumed it would be) combined with ZOOKEEPER-706.

        I haven't tested CMSIncrementalMode a bunch but making at least UseConcMarkSweepGC the default in the start script seems like an important, low-risk change.

        Show
        Sean Carr added a comment - This seems like a simple issue to resolve and then maybe improve upon later if there is uncertainty as to what the ideal options are. I recently spent a lot of time debugging some nasty production application problems that ended up being a result of long GC pauses in ZooKeeper since ConcMarkSweep is not the default (and I assumed it would be) combined with ZOOKEEPER-706 . I haven't tested CMSIncrementalMode a bunch but making at least UseConcMarkSweepGC the default in the start script seems like an important, low-risk change.
        Hide
        Patrick Hunt added a comment - - edited

        My current thinking is that we should add this:

        -XX:+UseConcMarkSweepGC –XX:+UseParNewGC
        

        with some sane default for -XX:ParallelGCThreads (we should check what the vm default is).

        These should be added as zkServer defaults via environment variables that can be easily overridden.

        Show
        Patrick Hunt added a comment - - edited My current thinking is that we should add this: -XX:+UseConcMarkSweepGC –XX:+UseParNewGC with some sane default for -XX:ParallelGCThreads (we should check what the vm default is). These should be added as zkServer defaults via environment variables that can be easily overridden.
        Hide
        Sean Carr added a comment -

        The default for -XX:ParallelGCThreads is the number of cores, which seems fairly sane. There are probably some environments where that should be adjusted but I think it's a good default for ZK.

        Show
        Sean Carr added a comment - The default for -XX:ParallelGCThreads is the number of cores, which seems fairly sane. There are probably some environments where that should be adjusted but I think it's a good default for ZK.
        Hide
        Patrick Hunt added a comment -

        That sounds reasonable to me. PPl can always override the defaults using JVMFlags or whatnot.

        Show
        Patrick Hunt added a comment - That sounds reasonable to me. PPl can always override the defaults using JVMFlags or whatnot.
        Hide
        nijel added a comment -

        Users will face challenge in setting other GC options also (like mewSize, mx, gcInterval, ms).

        Can we have a profile based GC options ?

        We can ship a gc-opts file with multiple profiles (low,medium and high). Each profile will have set of GC options. The profiles are named in the order of memory configured. Like high can have mx as 4 gb and related configurations. Low can have 512 as mx and related configurations.

        So user can just configure the profile name.

        As we are getting more improvements, comments can update these values in the file.

        Show
        nijel added a comment - Users will face challenge in setting other GC options also (like mewSize, mx, gcInterval, ms). Can we have a profile based GC options ? We can ship a gc-opts file with multiple profiles (low,medium and high). Each profile will have set of GC options. The profiles are named in the order of memory configured. Like high can have mx as 4 gb and related configurations. Low can have 512 as mx and related configurations. So user can just configure the profile name. As we are getting more improvements, comments can update these values in the file.

          People

          • Assignee:
            Unassigned
            Reporter:
            Patrick Hunt
          • Votes:
            2 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:

              Development