Sanjay> Moving to Avro IDL will change many data types in our servers and cannot be pluggable.
Sanjay> I don't think we can split the HDFS protocols as they use many common data types (block-id, block-token, etc).
I agree that it may not be easy to do this incrementally, but I think it would be much easier than trying to do it in a branch. For example, existing datatypes like Block that are used in many protocols could be made to implement Avro's SpecificRecord interface so that both IDL and reflect-based code can use them. Then we could more easily consider porting protocols one-by-one. Once we've ported all protocols that use Block, then we could, as a single patch, replace it with an IDL-generated version.
Eric> If we can get to agreement that the current RPC remains the default and that IDL work happens in a branch and that only after we have a reasonably complete backwards compat solution do we change to AVRO RPC by default, then I think we are good.
If Avro reflect-based RPC passes all tests, including performance, reliability, etc., why wouldn't we switch to it? Even without using an IDL, this would let client and server versions differ. We certainly don't want to encourage independent 3rd party implementations of protocols until we're happy that the protocols are what we intend to support long-term, but I don't yet see a reason not to switch once Avro-based RPC is functionally equivalent.
Eric> My concern is not with the already committed plugability, but with the change of the default RPC and incremental change over of the protocols. Once that work starts, we are committed to finishing it and we can not do that in the timeframe of the next release IMO.
I don't follow. If we proceed incrementally, thoroughly testing changes before committing them, then we should be able to release at any time, no?
Eric> Would we keep the plugable API once we make the transition?
I see no reason to keep it.
Folks are welcome to start IDL-based branch(es) if they prefer to operate that way. In that case, I will cease work on support for an incremental approach, as it would be redundant. I fear that, since such branches would be long-lived that they'll prove very painful to maintain, especially if they make substantial changes to protocols. We should not prohibit trunk changes to the HDFS and MapReduce protocols.