Cassandra
  1. Cassandra
  2. CASSANDRA-5361

Enable ThreadLocal allocation in the JVM

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Fix Version/s: 1.1.11, 1.2.4
    • Component/s: None
    • Labels:
    • Environment:

      JDK 6

      Description

      Adding -XX:+UseTLAB to cassandra-env.sh yields a ~15% read performance boost.

      Works with usual stress tool but also in our production.

      It enables thread local allocation of new objects which makes sense for SEDA based systems.

        Activity

        T Jake Luciani created issue -
        T Jake Luciani made changes -
        Field Original Value New Value
        Fix Version/s 1.2.4 [ 12324157 ]
        Ryan McGuire made changes -
        Tester enigmacurry
        Ryan McGuire made changes -
        Assignee T Jake Luciani [ tjake ] Ryan McGuire [ enigmacurry ]
        Hide
        Ryan McGuire added a comment -

        Assigning to myself for testing.

        Show
        Ryan McGuire added a comment - Assigning to myself for testing.
        Hide
        Marcus Eriksson added a comment -

        it seems +UseTLAB is on by default

        Show
        Marcus Eriksson added a comment - it seems +UseTLAB is on by default
        Hide
        T Jake Luciani added a comment -

        Its default with -server which we don't set

        Show
        T Jake Luciani added a comment - Its default with -server which we don't set
        Hide
        Ryan McGuire added a comment -

        I've run a 4-node baseline off trunk without +UseTLAB and here is the average result:

          stress -n 2000000 -o read -i 1
        
          Averages from the middle 80% of values:
          interval_op_rate          : 21918
          interval_key_rate         : 21918
          latency median            : 0.9
          latency 95th percentile   : 2.7
          latency 99.9th percentile : 40.2
          Total operation time      : 00:01:42
          END
        

        With +UseTLAB I do indeed get better results:

          Averages from the middle 80% of values:
          interval_op_rate          : 28352
          interval_key_rate         : 28352
          latency median            : 0.7
          latency 95th percentile   : 1.4
          latency 99.9th percentile : 22.3
          Total operation time      : 00:01:21
          END
        

        Looking at interval_op_rate, that's ~30% increase in read performance. Interestingly, I see more variation in the interval_op_rate with UseTLAB - the first run I ran was much closer to the baseline at 23065. I never saw that much variation without UseTLAB, but I never did see a value that was below the baseline average, so I see it as a benefit to turn it on even though it's less consistent.

        Write performance did not noticeably change.

        Show
        Ryan McGuire added a comment - I've run a 4-node baseline off trunk without +UseTLAB and here is the average result: stress -n 2000000 -o read -i 1 Averages from the middle 80% of values: interval_op_rate : 21918 interval_key_rate : 21918 latency median : 0.9 latency 95th percentile : 2.7 latency 99.9th percentile : 40.2 Total operation time : 00:01:42 END With +UseTLAB I do indeed get better results: Averages from the middle 80% of values: interval_op_rate : 28352 interval_key_rate : 28352 latency median : 0.7 latency 95th percentile : 1.4 latency 99.9th percentile : 22.3 Total operation time : 00:01:21 END Looking at interval_op_rate, that's ~30% increase in read performance. Interestingly, I see more variation in the interval_op_rate with UseTLAB - the first run I ran was much closer to the baseline at 23065. I never saw that much variation without UseTLAB, but I never did see a value that was below the baseline average, so I see it as a benefit to turn it on even though it's less consistent. Write performance did not noticeably change.
        Ryan McGuire made changes -
        Assignee Ryan McGuire [ enigmacurry ] T Jake Luciani [ tjake ]
        Hide
        Jonathan Ellis added a comment -

        Ship it!

        Show
        Jonathan Ellis added a comment - Ship it!
        Show
        T Jake Luciani added a comment - committed in https://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=commit;h=c5864a524e11842a75ac1d6c79ed137fadcb6c0e
        T Jake Luciani made changes -
        Fix Version/s 1.1.11 [ 12324040 ]
        T Jake Luciani made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        Ahmed Bashir added a comment - - edited

        Incidentally we've operated with this option enabled for a few months now and have observed reduced latency as well

        Show
        Ahmed Bashir added a comment - - edited Incidentally we've operated with this option enabled for a few months now and have observed reduced latency as well
        Ryan McGuire made changes -
        Labels qa-resolved
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Resolved Resolved
        1d 20h 53m 1 T Jake Luciani 21/Mar/13 16:47

          People

          • Assignee:
            T Jake Luciani
            Reporter:
            T Jake Luciani
            Tester:
            Ryan McGuire
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development