Major Description
When doing a rolling upgrade of our services from Curator 2.10.0 to Curator 2.12.0 we noticed some of the non-upgraded services started throwing this error. The cause seems to the combination of a new field added to ServiceInstance in CURATOR-275, and the lack of an "ignore unknown properties" setting on the ObjectMapper used by Curator 2.10.0.
This makes it so clients need to be upgraded before services. But if you have a cluster where some services are also clients of other services (or of a service they are a part of) then I don't see a way to do a rolling upgrade with the way things currently are. Whatever instance you upgrade first will start announcing itself in a way that breaks instances running the previous version of the code. This doesn't seem to be configurable either, so the new "enabled" field can't be omitted from the serialized form.
One possible solution is to revert CURATOR-275, add an "ignore unknown properties" to the ObjectMapper, and then re-introduce CURATOR-275 in a future release. That'd create a "you must go through release X first to upgrade to release Y" situation, but it would at least make it possible to do rolling updates.
Another possible solution is to accept that clients always need to be upgraded before services, and accept that if you can't do this, it's impossible to update without downtime. This seems like something that'd be good to avoid though.
2017-03-23 16:22:35,652 [DruidTaskResolver[com.metamx.tranquility.druid.IndexService@37a0ec3c]] WARN c.m.t.finagle.DruidTaskResolver - Poll failed, trying again at[2017-03-23T16:23:10.899Z]. org.codehaus.jackson.map.exc.UnrecognizedPropertyException: Unrecognized field "enabled" (Class org.apache.curator.x.discovery.ServiceInstance), not marked as ignorable at [Source: [B@e9cde06; line: 1, column: 226] (through reference chain: org.apache.curator.x.discovery.ServiceInstance["enabled"]) at org.codehaus.jackson.map.exc.UnrecognizedPropertyException.from(UnrecognizedPropertyException.java:53) ~[org.codehaus.jackson.jackson-mapper-asl-1.9.13.jar:1.9.13] at org.codehaus.jackson.map.deser.StdDeserializationContext.unknownFieldException(StdDeserializationContext.java:267) ~[org.codehaus.jackson.jackson-mapper-asl-1.9.13.jar:1.9.13] at org.codehaus.jackson.map.deser.std.StdDeserializer.reportUnknownProperty(StdDeserializer.java:673) ~[org.codehaus.jackson.jackson-mapper-asl-1.9.13.jar:1.9.13] at org.codehaus.jackson.map.deser.std.StdDeserializer.handleUnknownProperty(StdDeserializer.java:659) ~[org.codehaus.jackson.jackson-mapper-asl-1.9.13.jar:1.9.13] at org.codehaus.jackson.map.deser.BeanDeserializer.handleUnknownProperty(BeanDeserializer.java:1365) ~[org.codehaus.jackson.jackson-mapper-asl-1.9.13.jar:1.9.13] at org.codehaus.jackson.map.deser.BeanDeserializer._handleUnknown(BeanDeserializer.java:725) ~[org.codehaus.jackson.jackson-mapper-asl-1.9.13.jar:1.9.13] at org.codehaus.jackson.map.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:703) ~[org.codehaus.jackson.jackson-mapper-asl-1.9.13.jar:1.9.13] at org.codehaus.jackson.map.deser.BeanDeserializer.deserialize(BeanDeserializer.java:580) ~[org.codehaus.jackson.jackson-mapper-asl-1.9.13.jar:1.9.13] at org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2732) ~[org.codehaus.jackson.jackson-mapper-asl-1.9.13.jar:1.9.13] at org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1973) ~[org.codehaus.jackson.jackson-mapper-asl-1.9.13.jar:1.9.13] at org.apache.curator.x.discovery.details.JsonInstanceSerializer.deserialize(JsonInstanceSerializer.java:50) ~[org.apache.curator.curator-x-discovery-2.10.0.jar:na] at org.apache.curator.x.discovery.details.ServiceCacheImpl.addInstance(ServiceCacheImpl.java:193) ~[org.apache.curator.curator-x-discovery-2.10.0.jar:na] at org.apache.curator.x.discovery.details.ServiceCacheImpl.start(ServiceCacheImpl.java:96) ~[org.apache.curator.curator-x-discovery-2.10.0.jar:na]
Attachments
Issue Links
- links to
