Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Fix Version/s: 0.6.5
    • Component/s: Core
    • Labels:
      None

      Description

      An endpoint snitch that automatically and dynamically infers "distance" to other machines without having to explicitly configure rack and datacenter positions solves two problems:

      The killer feature here is adapting to things like compaction or a failing-but-not-yet-dead disk. This is important, since when we are doing reads we pick the "closest" replica for actually reading data from (and only read md5s from other replicas). This means that if the closest replica by network topology is temporarily slow due to compaction (for instance), we'll have to block for its reply even if we get the other replies much much faster.

      Not having to manually re-sync your configuration with your network topology when changes (adding machines) are made is a nice bonus.

      1. 981.txt
        27 kB
        Brandon Williams
      2. 981-0.6.txt
        26 kB
        Brandon Williams
      3. 981-fix.txt
        3 kB
        Jonathan Ellis
      4. 981-fix-v2.txt
        4 kB
        Jonathan Ellis

        Activity

        Jonathan Ellis created issue -
        Hide
        Jonathan Ellis added a comment -

        Stu suggested Vivaldi positioning (http://swtch.com/~rsc/talks/vivaldi-ccs.pdf), but that is actually solving a different problem. (How do you estimate network distance, without actually talking to the node in question?) Since we are in near-constant communication with all the nodes in the system, we should be able to leverage that for a solution that is both simpler and quicker to adapt to changing conditions.

        This problem seems similar to the sliding window we maintain of heartbeat times for failure detection, but this seems too heavyweight to apply when processing thousands of responses per second. Perhaps restricting the sampling to "at most N per second" would work, although I'm not sure if the phi algorithm can work on samples that don't occur at evenly-spaced intervals.

        Show
        Jonathan Ellis added a comment - Stu suggested Vivaldi positioning ( http://swtch.com/~rsc/talks/vivaldi-ccs.pdf ), but that is actually solving a different problem. (How do you estimate network distance, without actually talking to the node in question?) Since we are in near-constant communication with all the nodes in the system, we should be able to leverage that for a solution that is both simpler and quicker to adapt to changing conditions. This problem seems similar to the sliding window we maintain of heartbeat times for failure detection, but this seems too heavyweight to apply when processing thousands of responses per second. Perhaps restricting the sampling to "at most N per second" would work, although I'm not sure if the phi algorithm can work on samples that don't occur at evenly-spaced intervals.
        Hide
        Ryan King added a comment -

        I'm sure you thought about this, but in cases where you're using the RackAwareStrategy, the dynamic snitch will need to yield consistent results, otherwise, you could create situations where the natural endpoints for a token keep changing depending on network performance.

        Show
        Ryan King added a comment - I'm sure you thought about this, but in cases where you're using the RackAwareStrategy, the dynamic snitch will need to yield consistent results, otherwise, you could create situations where the natural endpoints for a token keep changing depending on network performance.
        Hide
        Stu Hood added a comment -

        > but that is actually solving a different problem.
        I think the key takeaway from Vivaldi is using a coordinate system, so that you don't have to store latency information for every endpoint you've ever communicated with.

        > I'm sure you thought about this, but in cases where you're using the RackAwareStrategy, the dynamic snitch will need to yield consistent results
        Would it be possible to have the snitch store its coordinates in the system table, so that during bootstrapping, it looks at gossip latency, tunes its coordinates, and then persists them forever? It rules out using endpointsnitches to dynamically adjust position based on load on a machine, but I think that might be better solved by load balancing anyway.

        Show
        Stu Hood added a comment - > but that is actually solving a different problem. I think the key takeaway from Vivaldi is using a coordinate system, so that you don't have to store latency information for every endpoint you've ever communicated with. > I'm sure you thought about this, but in cases where you're using the RackAwareStrategy, the dynamic snitch will need to yield consistent results Would it be possible to have the snitch store its coordinates in the system table, so that during bootstrapping, it looks at gossip latency, tunes its coordinates, and then persists them forever? It rules out using endpointsnitches to dynamically adjust position based on load on a machine, but I think that might be better solved by load balancing anyway.
        Hide
        Jonathan Ellis added a comment -

        I think what we want to do is have the "dynamic" part just apply to endpoint-sorting, and leave replica determination up to the strategy in terms of DC + Rack.

        A crappy alternative would be run-time loading of a new snitch (which could be hard-coded to sort whichever way).

        Show
        Jonathan Ellis added a comment - I think what we want to do is have the "dynamic" part just apply to endpoint-sorting, and leave replica determination up to the strategy in terms of DC + Rack. A crappy alternative would be run-time loading of a new snitch (which could be hard-coded to sort whichever way).
        Hide
        Vijay added a comment -

        Things can get complicated when there is a network outage and the network being smart trys to re-roue the traffic in the more expensive route, in which case the whole equation changes.... I am not sure if this sorting has to be done only once when the node comes up? even then we dont know if during bootstrapping there was a network outage.....

        Show
        Vijay added a comment - Things can get complicated when there is a network outage and the network being smart trys to re-roue the traffic in the more expensive route, in which case the whole equation changes.... I am not sure if this sorting has to be done only once when the node comes up? even then we dont know if during bootstrapping there was a network outage.....
        Hide
        Jonathan Ellis added a comment -

        Once the other nodes come back up they'll get some traffic still b/c of RR (and of course writes), so it will quickly figure out that the cheap route is back.

        Show
        Jonathan Ellis added a comment - Once the other nodes come back up they'll get some traffic still b/c of RR (and of course writes), so it will quickly figure out that the cheap route is back.
        Hide
        Ryan King added a comment -

        >I think what we want to do is have the "dynamic" part just apply to endpoint-sorting, and leave replica determination up to the strategy in terms of DC + Rack.

        I think this could be good for more than network distance, it would also help a cluster route around a node that is slow (either permanently or temporarily). In addition to being in compaction, a node that has just been restarted and therefore has empty caches should be routed around (but not completely, since it needs traffic in order to warm its caches).

        Show
        Ryan King added a comment - >I think what we want to do is have the "dynamic" part just apply to endpoint-sorting, and leave replica determination up to the strategy in terms of DC + Rack. I think this could be good for more than network distance, it would also help a cluster route around a node that is slow (either permanently or temporarily). In addition to being in compaction, a node that has just been restarted and therefore has empty caches should be routed around (but not completely, since it needs traffic in order to warm its caches).
        Hide
        Jonathan Ellis added a comment -

        Right, you want to adapt to "effective reply time" not just the network part. That is why I don't think the vivaldi stuff helps much since it's trying to solve a static network question instead.

        Show
        Jonathan Ellis added a comment - Right, you want to adapt to "effective reply time" not just the network part. That is why I don't think the vivaldi stuff helps much since it's trying to solve a static network question instead.
        Hide
        Brandon Williams added a comment -

        Wouldn't sorting the endpoints by their phi value in the FD work well since FD is already adaptive?

        Show
        Brandon Williams added a comment - Wouldn't sorting the endpoints by their phi value in the FD work well since FD is already adaptive?
        Hide
        Ryan King added a comment -

        The phi value is only effected by gossip, right? I think this would only work (or work much better) if it took into account rpc response times.

        Show
        Ryan King added a comment - The phi value is only effected by gossip, right? I think this would only work (or work much better) if it took into account rpc response times.
        Hide
        Jonathan Ellis added a comment -

        Right, we've seen lots of instances of nodes that have enough cpu to spit out gossip heartbeats, but not enough that they can handle a normal complement of requests.

        Show
        Jonathan Ellis added a comment - Right, we've seen lots of instances of nodes that have enough cpu to spit out gossip heartbeats, but not enough that they can handle a normal complement of requests.
        Hide
        Brandon Williams added a comment -

        I suspect the phi algorithm can be modified to satisfy the "at N most per second" criterion wrt RPC calls, but I'll have to experiment a bit to be certain.

        Show
        Brandon Williams added a comment - I suspect the phi algorithm can be modified to satisfy the "at N most per second" criterion wrt RPC calls, but I'll have to experiment a bit to be certain.
        Hide
        Brandon Williams added a comment -

        It looks like the phi algorithm can work for this. It's simple to replace timestamps with latency values, and then use an arbitrary value when calculating the phi score for a host. The main problem is calculating phi is less than free, and doing so for each request to determine the least latent endpoint will be too expensive. We can get around that by doing it up front when we add a sample and keeping a cache of the scores, but that means we'll have to add samples fairly infrequently, or at least batch the updates so the mean only has to be calculated once per batch.

        Show
        Brandon Williams added a comment - It looks like the phi algorithm can work for this. It's simple to replace timestamps with latency values, and then use an arbitrary value when calculating the phi score for a host. The main problem is calculating phi is less than free, and doing so for each request to determine the least latent endpoint will be too expensive. We can get around that by doing it up front when we add a sample and keeping a cache of the scores, but that means we'll have to add samples fairly infrequently, or at least batch the updates so the mean only has to be calculated once per batch.
        Hide
        Jonathan Ellis added a comment -

        right, even for something simpler than phi we'd want to sample, say, 10x per second at most.

        Show
        Jonathan Ellis added a comment - right, even for something simpler than phi we'd want to sample, say, 10x per second at most.
        Brandon Williams made changes -
        Field Original Value New Value
        Assignee Brandon Williams [ brandon.williams ]
        Hide
        Brandon Williams added a comment -

        Here is a dynamic snitch that listens to read latencies from other machines, accepts at most 10 timings per second, and once per second calculates the phi value for all hosts it knows about. Comments and testing would be most welcome.

        Show
        Brandon Williams added a comment - Here is a dynamic snitch that listens to read latencies from other machines, accepts at most 10 timings per second, and once per second calculates the phi value for all hosts it knows about. Comments and testing would be most welcome.
        Brandon Williams made changes -
        Attachment 981.txt [ 12446608 ]
        Brandon Williams made changes -
        Attachment 981.txt [ 12446608 ]
        Hide
        Brandon Williams added a comment -

        There's a problem with this snitch: once a machine sees a host and sorts it to the end, it will never read from that host again, meaning that it will never update that host's score, until such time that another host performs worse.

        Updated patch resolves this by resetting all the timing windows once per minute, giving the 'bad' hosts a chance to recover.

        Show
        Brandon Williams added a comment - There's a problem with this snitch: once a machine sees a host and sorts it to the end, it will never read from that host again, meaning that it will never update that host's score, until such time that another host performs worse. Updated patch resolves this by resetting all the timing windows once per minute, giving the 'bad' hosts a chance to recover.
        Brandon Williams made changes -
        Attachment 981.txt [ 12446626 ]
        Jonathan Ellis made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Jonathan Ellis added a comment -

        (1)

        combine

        + windows.putIfAbsent(host, new AdaptiveLatencyTracker(WINDOW_SIZE));
        + AdaptiveLatencyTracker tracker = windows.get(host);

        to

        + AdaptiveLatencyTracker tracker = windows.putIfAbsent(host, new AdaptiveLatencyTracker(WINDOW_SIZE));

        Even better: get, then putIfAbsent only if null, would avoid creating new ALT objects every time

        (2)

        receiveTiming isn't threadsafe. use AtomicInteger?

        (3)

        AdaptiveLatencyTracker doesn't look threadsafe either. definitely LBD isn't. I think just using a threadsafe queue like CLQ would work? (My fault for naming ASD a Deque, when it only really needs a Queue)

        (4)

        do we need unregister()? if not, let's drop that api

        (5)

        sortByProximity needs to return zero if both scores are null. even better, take a non-dynamic snitch and use the static topology when there is no score info yet (this would save us from sending data requests to another data center after every clear of the stats). So, rather than using DES directly in the config, maybe having a boolean for whether to wrap your regular snitch, with the dynamic one, is the way to go.

        what if reset cleared scores, instead of latencies? this would result in a more gradual aging out of both slow and fast latencies as new ones were pushed in, which would make it more tolerant of brief hiccups where a mostly fast node had a couple slow responses. Feels more how phi was meant to work, to me.

        (6)

        + if (address != FBUtilities.getLocalAddress()) // we only know about ourself
        + return addresses;

        let's change this to an assert

        (7)

        deque.offer is more idiomatic than try/except in java

        (8)

        let's use a single timer for both update and reset

        (9)

        it's not completely clear to me that the phi code designed for telling "how long has been too long to wait for updates that are supposed to arrive at the same interval," applies well to latency information that arrives in bursts up to our max per interval. can you add some tests showing that it does the right thing, given several mixes of latencies?

        Show
        Jonathan Ellis added a comment - (1) combine + windows.putIfAbsent(host, new AdaptiveLatencyTracker(WINDOW_SIZE)); + AdaptiveLatencyTracker tracker = windows.get(host); to + AdaptiveLatencyTracker tracker = windows.putIfAbsent(host, new AdaptiveLatencyTracker(WINDOW_SIZE)); Even better: get, then putIfAbsent only if null, would avoid creating new ALT objects every time (2) receiveTiming isn't threadsafe. use AtomicInteger? (3) AdaptiveLatencyTracker doesn't look threadsafe either. definitely LBD isn't. I think just using a threadsafe queue like CLQ would work? (My fault for naming ASD a Deque, when it only really needs a Queue) (4) do we need unregister()? if not, let's drop that api (5) sortByProximity needs to return zero if both scores are null. even better, take a non-dynamic snitch and use the static topology when there is no score info yet (this would save us from sending data requests to another data center after every clear of the stats). So, rather than using DES directly in the config, maybe having a boolean for whether to wrap your regular snitch, with the dynamic one, is the way to go. what if reset cleared scores, instead of latencies? this would result in a more gradual aging out of both slow and fast latencies as new ones were pushed in, which would make it more tolerant of brief hiccups where a mostly fast node had a couple slow responses. Feels more how phi was meant to work, to me. (6) + if (address != FBUtilities.getLocalAddress()) // we only know about ourself + return addresses; let's change this to an assert (7) deque.offer is more idiomatic than try/except in java (8) let's use a single timer for both update and reset (9) it's not completely clear to me that the phi code designed for telling "how long has been too long to wait for updates that are supposed to arrive at the same interval," applies well to latency information that arrives in bursts up to our max per interval. can you add some tests showing that it does the right thing, given several mixes of latencies?
        Jonathan Ellis made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Hide
        Jonathan Ellis added a comment -

        Updates:

        LBD is threadsafe. Not sure what I was smoking there.

        Big Hammer latency reset is fine for now but I would like to have more graceful aging out of latencies so if you have latencies like .|.|... and we age out the || then it sorts to its "normal" place again without having to rediscover things from scratch.

        Show
        Jonathan Ellis added a comment - Updates: LBD is threadsafe. Not sure what I was smoking there. Big Hammer latency reset is fine for now but I would like to have more graceful aging out of latencies so if you have latencies like .|.|... and we age out the || then it sorts to its "normal" place again without having to rediscover things from scratch.
        Brandon Williams made changes -
        Attachment 981.txt [ 12446626 ]
        Hide
        Brandon Williams added a comment -

        Updated patch incorporating above suggestions.

        Show
        Brandon Williams added a comment - Updated patch incorporating above suggestions.
        Brandon Williams made changes -
        Attachment 981.txt [ 12448986 ]
        Brandon Williams made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Jonathan Ellis added a comment -

        +1 after minor clean up:

        + if (conf.dynamic_snitch == false)
        + return snitch;
        + else
        + return (IEndpointSnitch) new DynamicEndpointSnitch(snitch);

        prefer Unable to render embedded object: File (conf.dynamic_snitch (and prefer positive conditions first, to avoid ") not found...else")
        the cast should be unnecessary
        personally I would go for a ?: here but if/else is ok

        + public void add(double i)
        +

        { + latencies.offer(i); + }

        indent is wrong there

        + double exponent = (-1)*(t)/mean;

        space b/t arithmetic operators here and everywhere else please

        DynamicEndpointSnitchTest: several { need to be on new lines

        Show
        Jonathan Ellis added a comment - +1 after minor clean up: + if (conf.dynamic_snitch == false) + return snitch; + else + return (IEndpointSnitch) new DynamicEndpointSnitch(snitch); prefer Unable to render embedded object: File (conf.dynamic_snitch (and prefer positive conditions first, to avoid ") not found. ..else") the cast should be unnecessary personally I would go for a ?: here but if/else is ok + public void add(double i) + { + latencies.offer(i); + } indent is wrong there + double exponent = (-1)*(t)/mean; space b/t arithmetic operators here and everywhere else please DynamicEndpointSnitchTest: several { need to be on new lines
        Hide
        Brandon Williams added a comment -

        Committed with cleanups.

        Show
        Brandon Williams added a comment - Committed with cleanups.
        Brandon Williams made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        Hudson added a comment -
        Show
        Hudson added a comment - Integrated in Cassandra #491 (See http://hudson.zones.apache.org/hudson/job/Cassandra/491/ )
        Hide
        Jonathan Ellis added a comment -

        re-opening for backport to 0.6.5

        Show
        Jonathan Ellis added a comment - re-opening for backport to 0.6.5
        Jonathan Ellis made changes -
        Resolution Fixed [ 1 ]
        Status Resolved [ 5 ] Reopened [ 4 ]
        Jonathan Ellis made changes -
        Fix Version/s 0.6.5 [ 12315220 ]
        Fix Version/s 0.7 beta 1 [ 12314533 ]
        Hide
        Brandon Williams added a comment -

        Backport for 0.6.5. Requires -Dcassandra.dynamic_snitch=true to be added to the JVM options to enable.

        Show
        Brandon Williams added a comment - Backport for 0.6.5. Requires -Dcassandra.dynamic_snitch=true to be added to the JVM options to enable.
        Brandon Williams made changes -
        Attachment 981-0.6.txt [ 12451803 ]
        Brandon Williams made changes -
        Attachment 981-0.6.txt [ 12451803 ]
        Hide
        Brandon Williams added a comment -

        Also includes improvements from CASSANDRA-1296.

        Show
        Brandon Williams added a comment - Also includes improvements from CASSANDRA-1296 .
        Brandon Williams made changes -
        Attachment 981-0.6.txt [ 12451808 ]
        Brandon Williams made changes -
        Status Reopened [ 4 ] Patch Available [ 10002 ]
        Hide
        Jonathan Ellis added a comment -

        +1

        Show
        Jonathan Ellis added a comment - +1
        Hide
        Jonathan Ellis added a comment -

        [committed by Brandon]

        Show
        Jonathan Ellis added a comment - [committed by Brandon]
        Jonathan Ellis made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        Jonathan Ellis added a comment -

        Getting this test failure in 0.6:

        [junit] Testcase: testSnitch(org.apache.cassandra.locator.DynamicEndpointSnitchTest): FAILED
        [junit] null
        [junit] junit.framework.AssertionFailedError
        [junit] at org.apache.cassandra.locator.DynamicEndpointSnitchTest.testSnitch(DynamicEndpointSnitchTest.java:70)

        Show
        Jonathan Ellis added a comment - Getting this test failure in 0.6: [junit] Testcase: testSnitch(org.apache.cassandra.locator.DynamicEndpointSnitchTest): FAILED [junit] null [junit] junit.framework.AssertionFailedError [junit] at org.apache.cassandra.locator.DynamicEndpointSnitchTest.testSnitch(DynamicEndpointSnitchTest.java:70)
        Jonathan Ellis made changes -
        Resolution Fixed [ 1 ]
        Status Resolved [ 5 ] Reopened [ 4 ]
        Hide
        Jonathan Ellis added a comment -

        fix patch attached to

        1) r/m unused imports
        2) fall back to subsnitch on score equality as well as no score at all
        3) order lower scores first in proximity list

        Show
        Jonathan Ellis added a comment - fix patch attached to 1) r/m unused imports 2) fall back to subsnitch on score equality as well as no score at all 3) order lower scores first in proximity list
        Jonathan Ellis made changes -
        Attachment 981-fix.txt [ 12452124 ]
        Jonathan Ellis made changes -
        Status Reopened [ 4 ] Patch Available [ 10002 ]
        Hide
        Jonathan Ellis added a comment -

        fix-v2 switches to EndPointSnitch (SimpleSnitch does not exist in 0.6), fixes order for "make host3 the worst" and adds missing assert.

        Show
        Jonathan Ellis added a comment - fix-v2 switches to EndPointSnitch (SimpleSnitch does not exist in 0.6), fixes order for "make host3 the worst" and adds missing assert.
        Jonathan Ellis made changes -
        Attachment 981-fix-v2.txt [ 12452125 ]
        Hide
        Brandon Williams added a comment -

        Committed to 0.6 and trunk. Also worth noting that DES has been broken in trunk due to incorrectly calculating the probability (needs to be 1 - Math.pow) which I fixed also. We missed this because the tests were wrong.

        Show
        Brandon Williams added a comment - Committed to 0.6 and trunk. Also worth noting that DES has been broken in trunk due to incorrectly calculating the probability (needs to be 1 - Math.pow) which I fixed also. We missed this because the tests were wrong.
        Brandon Williams made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        Hudson added a comment -

        Integrated in Cassandra #515 (See https://hudson.apache.org/hudson/job/Cassandra/515/)
        fall back to subsnitch on score equality as well as no score at all, fix probability calculation. Patch by jbellis and brandonwilliams, reviewed by brandonwilliams for CASSANDRA-981

        Show
        Hudson added a comment - Integrated in Cassandra #515 (See https://hudson.apache.org/hudson/job/Cassandra/515/ ) fall back to subsnitch on score equality as well as no score at all, fix probability calculation. Patch by jbellis and brandonwilliams, reviewed by brandonwilliams for CASSANDRA-981
        Hide
        Wouter de Bie added a comment -

        There seems to be a bug in the 0.6.5 DatabaseDescriptor.java. On line 629:

         String dynamic = System.getProperty("cassandra.dynamic_snitch");
         if (dynamic == null || Boolean.getBoolean(dynamic) == false)
            epSnitch = snitch;
         else
             epSnitch = new DynamicEndpointSnitch(snitch);
        

        Boolean.getBoolean(String s) checks for System.getProperty(). I think the code should read:

          if (Boolean.getBoolean('cassandra.dynamic_snitch') == false)
            epSnitch = snitch;
         else
             epSnitch = new DynamicEndpointSnitch(snitch);
        

        A workaround is to supply JVM properties like "-Dcassandra.dynamic_snitch_enabled=true -Dcassandra.dynamic_snitch=cassandra.dynamic_snitch_enabled"

        Has anyone succesfully run with 0.6.5 and the dynamic snitch?

        Show
        Wouter de Bie added a comment - There seems to be a bug in the 0.6.5 DatabaseDescriptor.java. On line 629: String dynamic = System.getProperty("cassandra.dynamic_snitch"); if (dynamic == null || Boolean.getBoolean(dynamic) == false) epSnitch = snitch; else epSnitch = new DynamicEndpointSnitch(snitch); Boolean.getBoolean(String s) checks for System.getProperty(). I think the code should read: if (Boolean.getBoolean('cassandra.dynamic_snitch') == false) epSnitch = snitch; else epSnitch = new DynamicEndpointSnitch(snitch); A workaround is to supply JVM properties like "-Dcassandra.dynamic_snitch_enabled=true -Dcassandra.dynamic_snitch=cassandra.dynamic_snitch_enabled" Has anyone succesfully run with 0.6.5 and the dynamic snitch?
        Hide
        Wouter de Bie added a comment -

        This stuff is actually more broken than I thought..

        It only works with one keyspace, since the DynamicEndpointSnitchMBean is registered per keyspace, which is not allowed.

        Next, the DynamicEndpointSnitch doesn't work with the RackAwareStrategy, since that one checks the following:

        if (!(snitch instanceof EndPointSnitch))
                    throw new IllegalArgumentException(("RackAwareStrategy requires EndPointSnitch."));
        

        And breaks, since the DynamicEndPointSnitch extends AbstractEndpointSnitch and not EndPointSnitch. RackAwareStrategy should check for the interface IEndPointSnitch.

        Show
        Wouter de Bie added a comment - This stuff is actually more broken than I thought.. It only works with one keyspace, since the DynamicEndpointSnitchMBean is registered per keyspace, which is not allowed. Next, the DynamicEndpointSnitch doesn't work with the RackAwareStrategy, since that one checks the following: if (!(snitch instanceof EndPointSnitch)) throw new IllegalArgumentException(("RackAwareStrategy requires EndPointSnitch.")); And breaks, since the DynamicEndPointSnitch extends AbstractEndpointSnitch and not EndPointSnitch. RackAwareStrategy should check for the interface IEndPointSnitch.
        Hide
        Jonathan Ellis added a comment -

        oh, that's funny, in an unfortunate way. you're right about the getBoolean. fixed in r1000877 on 0.6 branch.

        you're right about the second part too, but that was already fixed in CASSANDRA-1429 (also on 0.6 branch).

        Show
        Jonathan Ellis added a comment - oh, that's funny, in an unfortunate way. you're right about the getBoolean. fixed in r1000877 on 0.6 branch. you're right about the second part too, but that was already fixed in CASSANDRA-1429 (also on 0.6 branch).
        Gavin made changes -
        Workflow no-reopen-closed, patch-avail [ 12508487 ] patch-available, re-open possible [ 12752196 ]
        Gavin made changes -
        Workflow patch-available, re-open possible [ 12752196 ] reopen-resolved, no closed status, patch-avail, testing [ 12755236 ]

          People

          • Assignee:
            Brandon Williams
            Reporter:
            Jonathan Ellis
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development