Cassandra
  1. Cassandra
  2. CASSANDRA-5361

Enable ThreadLocal allocation in the JVM

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Fix Version/s: 1.1.11, 1.2.4
    • Component/s: None
    • Labels:
      None
    • Environment:

      JDK 6

      Description

      Adding -XX:+UseTLAB to cassandra-env.sh yields a ~15% read performance boost.

      Works with usual stress tool but also in our production.

      It enables thread local allocation of new objects which makes sense for SEDA based systems.

        Activity

        Hide
        Ryan McGuire added a comment -

        Assigning to myself for testing.

        Show
        Ryan McGuire added a comment - Assigning to myself for testing.
        Hide
        Marcus Eriksson added a comment -

        it seems +UseTLAB is on by default

        Show
        Marcus Eriksson added a comment - it seems +UseTLAB is on by default
        Hide
        T Jake Luciani added a comment -

        Its default with -server which we don't set

        Show
        T Jake Luciani added a comment - Its default with -server which we don't set
        Hide
        Ryan McGuire added a comment -

        I've run a 4-node baseline off trunk without +UseTLAB and here is the average result:

          stress -n 2000000 -o read -i 1
        
          Averages from the middle 80% of values:
          interval_op_rate          : 21918
          interval_key_rate         : 21918
          latency median            : 0.9
          latency 95th percentile   : 2.7
          latency 99.9th percentile : 40.2
          Total operation time      : 00:01:42
          END
        

        With +UseTLAB I do indeed get better results:

          Averages from the middle 80% of values:
          interval_op_rate          : 28352
          interval_key_rate         : 28352
          latency median            : 0.7
          latency 95th percentile   : 1.4
          latency 99.9th percentile : 22.3
          Total operation time      : 00:01:21
          END
        

        Looking at interval_op_rate, that's ~30% increase in read performance. Interestingly, I see more variation in the interval_op_rate with UseTLAB - the first run I ran was much closer to the baseline at 23065. I never saw that much variation without UseTLAB, but I never did see a value that was below the baseline average, so I see it as a benefit to turn it on even though it's less consistent.

        Write performance did not noticeably change.

        Show
        Ryan McGuire added a comment - I've run a 4-node baseline off trunk without +UseTLAB and here is the average result: stress -n 2000000 -o read -i 1 Averages from the middle 80% of values: interval_op_rate : 21918 interval_key_rate : 21918 latency median : 0.9 latency 95th percentile : 2.7 latency 99.9th percentile : 40.2 Total operation time : 00:01:42 END With +UseTLAB I do indeed get better results: Averages from the middle 80% of values: interval_op_rate : 28352 interval_key_rate : 28352 latency median : 0.7 latency 95th percentile : 1.4 latency 99.9th percentile : 22.3 Total operation time : 00:01:21 END Looking at interval_op_rate, that's ~30% increase in read performance. Interestingly, I see more variation in the interval_op_rate with UseTLAB - the first run I ran was much closer to the baseline at 23065. I never saw that much variation without UseTLAB, but I never did see a value that was below the baseline average, so I see it as a benefit to turn it on even though it's less consistent. Write performance did not noticeably change.
        Hide
        Jonathan Ellis added a comment -

        Ship it!

        Show
        Jonathan Ellis added a comment - Ship it!
        Show
        T Jake Luciani added a comment - committed in https://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=commit;h=c5864a524e11842a75ac1d6c79ed137fadcb6c0e
        Hide
        Ahmed Bashir added a comment - - edited

        Incidentally we've operated with this option enabled for a few months now and have observed reduced latency as well

        Show
        Ahmed Bashir added a comment - - edited Incidentally we've operated with this option enabled for a few months now and have observed reduced latency as well

          People

          • Assignee:
            T Jake Luciani
            Reporter:
            T Jake Luciani
            Tester:
            Ryan McGuire
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development