Details

    • Type: Sub-task Sub-task
    • Status: Open
    • Priority: Critical Critical
    • Resolution: Unresolved
    • Affects Version/s: 0.99.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Basic codec negotiation:

      There should be a default codec used for cell encoding over the RPC connection. This should be configurable in the site file.

      The client can optionally send a message, a manufactured "call" that would otherwise be invalid in some way, to the server asking for a list of supported cell codecs. An older server should simply send back an error because the request is invalid except to servers supporting this feature. A server supporting this feature should send back the requested information or an error indication if something went wrong.

      The client can optionally send a message, a manufactured "call" that would otherwise be invalid in some way, to the server asking for it to use a given codec for all further communication. Otherwise the server will continue to use the default codec. The server will send back a call response acknowledging the change or an error indication if the request cannot be honored.

      Server configuration should support mappings from one codec type to another. We need to handle the case where the server has a codec available that extends the requested type but overrides some behavior in the base class, and this is what should be used in lieu of the base type. It must also be possible to choose an alternate default codec which stands in for the default codec, is compatible with client expectations, but changes the server side behavior as needed in the absence of negotiation.

        Activity

        Hide
        Andrew Purtell added a comment -

        Moving out of 0.98

        Show
        Andrew Purtell added a comment - Moving out of 0.98
        Hide
        Andrew Purtell added a comment -

        Where is this issue ramkrishna.s.vasudevan or Anoop Sam John? Planning on branching 0.98 Sunday 12/1.

        Show
        Andrew Purtell added a comment - Where is this issue ramkrishna.s.vasudevan or Anoop Sam John ? Planning on branching 0.98 Sunday 12/1.
        Hide
        Anoop Sam John added a comment -

        So here basically using a different Encoder at server side, we will or will not serialize the tags back to the client. In case of scan where we use Codec based serialization of Result cells in the PayloadCarryingRpcController's payload, this different Codec will work. But what about pure PB based request, response? In this case the cells at server side will be converted to PB cells and returned. In the current code this conversion is not considering cell tags so we are not serializing back the tags as of now. For get request, there will be PB based return.

        Show
        Anoop Sam John added a comment - So here basically using a different Encoder at server side, we will or will not serialize the tags back to the client. In case of scan where we use Codec based serialization of Result cells in the PayloadCarryingRpcController's payload, this different Codec will work. But what about pure PB based request, response? In this case the cells at server side will be converted to PB cells and returned. In the current code this conversion is not considering cell tags so we are not serializing back the tags as of now. For get request, there will be PB based return.
        Hide
        ramkrishna.s.vasudevan added a comment -

        So it will be like the server will always NOT write back the cell tags?

        Currently it is based on the mapping that we provide and no context is available with the client.
        For export tool we may have to see how to over come the problem.
        When we have any type of negotiation
        0.96 clients talking to 0.98 servers expecting them to act like 0.96 servers would be another tricky part because the we cannot use a new codec here as the 0.96 client is not aware of it.

        Show
        ramkrishna.s.vasudevan added a comment - So it will be like the server will always NOT write back the cell tags? Currently it is based on the mapping that we provide and no context is available with the client. For export tool we may have to see how to over come the problem. When we have any type of negotiation 0.96 clients talking to 0.98 servers expecting them to act like 0.96 servers would be another tricky part because the we cannot use a new codec here as the 0.96 client is not aware of it.
        Hide
        Anoop Sam John added a comment - - edited

        Ram
        So it will be like the server will always NOT write back the cell tags? Or that also based on some context information?
        There is Export tool which uses MR based scan. In this case the server should serialize back the tags also?

        Show
        Anoop Sam John added a comment - - edited Ram So it will be like the server will always NOT write back the cell tags? Or that also based on some context information? There is Export tool which uses MR based scan. In this case the server should serialize back the tags also?
        Hide
        Elliott Clark added a comment -

        Yeah that can work too(I'm not sure which would be cleaner) as long as anything that uses the mapping is a new codec so there's no possibility of older client's being confused.

        Show
        Elliott Clark added a comment - Yeah that can work too(I'm not sure which would be cleaner) as long as anything that uses the mapping is a new codec so there's no possibility of older client's being confused.
        Hide
        ramkrishna.s.vasudevan added a comment -

        The impl that I have does the same but in a bit different way. So I would add a configuration say a mapping between two codecs.
        Say CodecA is related with CodecB. Client would say it would be using CodecA on the header and server would pick up CodecB from the above mapping configuration(if found).
        The CodecA.getEncoder and CodecB.getDecoder would work in sync (for client to server communication).
        The CodecB.getEncoder and CodecA.getDecoder() would work in sycn (for server to client communication).

        Show
        ramkrishna.s.vasudevan added a comment - The impl that I have does the same but in a bit different way. So I would add a configuration say a mapping between two codecs. Say CodecA is related with CodecB. Client would say it would be using CodecA on the header and server would pick up CodecB from the above mapping configuration(if found). The CodecA.getEncoder and CodecB.getDecoder would work in sync (for client to server communication). The CodecB.getEncoder and CodecA.getDecoder() would work in sycn (for server to client communication).
        Hide
        ramkrishna.s.vasudevan added a comment -

        Correct.. Agree. So here the mapping is we have a codec say the ClientSideCodec and the ServerSideCodec for the same type of base codec.
        The encoder and decoder will be called based on the process that calls it.. (either the client or the server).
        We have considered this way also. Let me see what changes would be needed for getting this.

        Show
        ramkrishna.s.vasudevan added a comment - Correct.. Agree. So here the mapping is we have a codec say the ClientSideCodec and the ServerSideCodec for the same type of base codec. The encoder and decoder will be called based on the process that calls it.. (either the client or the server). We have considered this way also. Let me see what changes would be needed for getting this.
        Hide
        Elliott Clark added a comment -

        This will have the two codecs that would be needed by the client and server?

        I was thinking that would be the sentinel that the following two classes are used.

        However I'm still a little slow on picking up when you'd need the client to specify more than one codec. It seems like in your example having the client specify one codec and having the codec have some idea of where it's running would make this easier.

        Client sending to Server

        1. Client calls getCodec.
        2. Client get Codec1
        3. Tells that codec that it's on the client
        4. Client then calls getEncoder https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/codec/Codec.html#getEncoder(java.io.OutputStream)
          • This call returns Codec1.ClientEncoder
        5. send the kv's to the codec's encoder who encodes the key values placing the tags at the end of the byte stream
        6. Client sends that encoded byte stream to the server after specifying the codec on the header.
        7. Server then looks up the same codec codec by calling getCodec
        8. Server gets Codec1
        9. Server then tells the codec that it's on the server
        10. Server gets the decoder https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/codec/Codec.html#getDecoder(java.io.InputStream)
          • This call returns Codec1.ServerDecoder
        11. Server feeds the decoder the bytes who preforms the exact inverse of what was done by the client's encoder.

        Server sending to Client

        1. Server then looks up the codec specified in the connection header
        2. Server gets Codec1
        3. Server then tells the codec that it's on the server
        4. Server gets the encoder https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/codec/Codec.html#getDecoder(java.io.InputStream)
          • This call returns Codec1.ServerEncoder
        5. Server feeds the encoder the cells. It stips any system tags and then encodes the cells by moving the tags to the back end (or whatever is needed)
        6. Server sends these bytes
        7. Client then gets the Call result with the encoded bytes.
        8. Notices they are encoded so callse getCodec
        9. The client gets Codec1
        10. The client then informs Codec1 that it's on the client
        11. The client gets the decoder
          • This call returns Codec1.ClientDecoder
        12. This decoder knows nothing about system tags so it only preforms the inverse of moving the tags to the end of the byte stream.
        Show
        Elliott Clark added a comment - This will have the two codecs that would be needed by the client and server? I was thinking that would be the sentinel that the following two classes are used. However I'm still a little slow on picking up when you'd need the client to specify more than one codec. It seems like in your example having the client specify one codec and having the codec have some idea of where it's running would make this easier. Client sending to Server Client calls getCodec. Client get Codec1 Tells that codec that it's on the client Client then calls getEncoder https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/codec/Codec.html#getEncoder(java.io.OutputStream ) This call returns Codec1.ClientEncoder send the kv's to the codec's encoder who encodes the key values placing the tags at the end of the byte stream Client sends that encoded byte stream to the server after specifying the codec on the header. Server then looks up the same codec codec by calling getCodec Server gets Codec1 Server then tells the codec that it's on the server Server gets the decoder https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/codec/Codec.html#getDecoder(java.io.InputStream ) This call returns Codec1.ServerDecoder Server feeds the decoder the bytes who preforms the exact inverse of what was done by the client's encoder. Server sending to Client Server then looks up the codec specified in the connection header Server gets Codec1 Server then tells the codec that it's on the server Server gets the encoder https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/codec/Codec.html#getDecoder(java.io.InputStream ) This call returns Codec1.ServerEncoder Server feeds the encoder the cells. It stips any system tags and then encodes the cells by moving the tags to the back end (or whatever is needed) Server sends these bytes Client then gets the Call result with the encoded bytes. Notices they are encoded so callse getCodec The client gets Codec1 The client then informs Codec1 that it's on the client The client gets the decoder This call returns Codec1.ClientDecoder This decoder knows nothing about system tags so it only preforms the inverse of moving the tags to the end of the byte stream.
        Hide
        ramkrishna.s.vasudevan added a comment -

        Just to clarify

        Something like "org.apache.hadoop.hbase.codecs.SpecifyEachDirectionCodec"

        This will have the two codecs that would be needed by the client and server?

        Show
        ramkrishna.s.vasudevan added a comment - Just to clarify Something like "org.apache.hadoop.hbase.codecs.SpecifyEachDirectionCodec" This will have the two codecs that would be needed by the client and server?
        Hide
        Elliott Clark added a comment -

        That sounds more like an in and an out codec. Why not just add each way.

        eg.

        // This is sent on connection setup after the connection preamble is sent.
        message ConnectionHeader {
          optional UserInformation user_info = 1;
          optional string service_name = 2;
          // Cell block codec we will use sending over optional cell blocks.  Server throws exception
          // if cannot deal.  Null means no codec'ing going on so we are pb all the time (SLOW!!!)
          optional string cell_block_codec_class = 3;
        
          // This one is already there. 
          // For 0.98 this is set to a class that's only present in 0.98+
          // Something like "org.apache.hadoop.hbase.codecs.SpecifyEachDirectionCodec"
          // Obviously the name can use some work.
          optional string cell_block_compressor_class = 4;
        
          // This will be used on all key values being sent from this client to the sever
          optional string cell_block_compressor_from_client_class = 5;
          // This will be used on all all key values being sent to the client from the server.
          optional string cell_block_compressor_to_client_class = 6;
        }
        

        Older servers will throw an exception if they don't know the full duplex codec. So the client will fall back to a 0.96 codec.

        Servers after 0.98 will throw an exception if they can't accept the codec's specified for each direction (preferably an info carrying exception with a list of codecs that are accepted). If it's successful then we just use each the negotiated codec.

        With this there's no extra round trips on the default case (server and client both support the same codecs). On the worst case there's one extra round trip.

        Show
        Elliott Clark added a comment - That sounds more like an in and an out codec. Why not just add each way. eg. // This is sent on connection setup after the connection preamble is sent. message ConnectionHeader { optional UserInformation user_info = 1; optional string service_name = 2; // Cell block codec we will use sending over optional cell blocks. Server throws exception // if cannot deal. Null means no codec'ing going on so we are pb all the time (SLOW!!!) optional string cell_block_codec_class = 3; // This one is already there. // For 0.98 this is set to a class that's only present in 0.98+ // Something like "org.apache.hadoop.hbase.codecs.SpecifyEachDirectionCodec" // Obviously the name can use some work. optional string cell_block_compressor_class = 4; // This will be used on all key values being sent from this client to the sever optional string cell_block_compressor_from_client_class = 5; // This will be used on all all key values being sent to the client from the server. optional string cell_block_compressor_to_client_class = 6; } Older servers will throw an exception if they don't know the full duplex codec. So the client will fall back to a 0.96 codec. Servers after 0.98 will throw an exception if they can't accept the codec's specified for each direction (preferably an info carrying exception with a list of codecs that are accepted). If it's successful then we just use each the negotiated codec. With this there's no extra round trips on the default case (server and client both support the same codecs). On the worst case there's one extra round trip.
        Hide
        ramkrishna.s.vasudevan added a comment -

        When codec is getting used say KeyValueCodec we would be using the codec in in Puts also. Means the server would use the codec to decode the KVs and add it to the RS.
        The same codec will be used by the server while the client issues a scan or read operation. This means that the codec is uniform in the client and the server.
        For example, take the case of tags, the client should be able to add the tags through the codec while doing Puts and the server should decode the tags. Where as while doing the scan the server should suppress the tags using a different codec and the client should be able to decode KVs using that codec.
        Using one codec would be applicable for pure scan cases but when we have read and writes then both client and server should be aware of its own local and the remote codec on the other side.
        Either we could have a mapping in the server side for the codec specified on the client side or we should have specific api in the encoder and decoders like getEncoderForClient, getEncoderForServer.
        Anyway this mapping ( we can say as the intelligence added to server to decide the codec) is always context specific.

        Show
        ramkrishna.s.vasudevan added a comment - When codec is getting used say KeyValueCodec we would be using the codec in in Puts also. Means the server would use the codec to decode the KVs and add it to the RS. The same codec will be used by the server while the client issues a scan or read operation. This means that the codec is uniform in the client and the server. For example, take the case of tags, the client should be able to add the tags through the codec while doing Puts and the server should decode the tags. Where as while doing the scan the server should suppress the tags using a different codec and the client should be able to decode KVs using that codec. Using one codec would be applicable for pure scan cases but when we have read and writes then both client and server should be aware of its own local and the remote codec on the other side. Either we could have a mapping in the server side for the codec specified on the client side or we should have specific api in the encoder and decoders like getEncoderForClient, getEncoderForServer. Anyway this mapping ( we can say as the intelligence added to server to decide the codec) is always context specific.
        Hide
        stack added a comment -

        Give me example of a mapping mighty ramkrishna.s.vasudevan so I can follow along better. Rather than one mapping for clients and another for export, would it better to have this switch be done on what permissions the client has?

        Show
        stack added a comment - Give me example of a mapping mighty ramkrishna.s.vasudevan so I can follow along better. Rather than one mapping for clients and another for export, would it better to have this switch be done on what permissions the client has?
        Hide
        ramkrishna.s.vasudevan added a comment -

        Regarding the mapping, how should the server decide on the mapping? Should it be from a configuration? If it is from a configuration then it is not generic because this may be context based.
        One reason why the mapping is important because in some cases the client may ask for a codec but it may not be right for the server to send out all the information say for example some information stored in the tags. So in that case the server should apply the context based knowledge on what codec it should use. This is where the mapping comes into picture.
        In an internal discussion, there is a chance that for some cases like Export the server still needs to send out information in tags too. So in that case the mapping should deal this in a different way.
        So overall this mapping and usage of the codecs on the server side is more context based.
        When we have sytem tags in future the above argument applies (generally here it is better to suppress the tags from the clients).
        So the connection negotiation part is a protocol that we can decide but the mapping is one which is not very generic. Can try to make this mapping a pluggable one?

        Show
        ramkrishna.s.vasudevan added a comment - Regarding the mapping, how should the server decide on the mapping? Should it be from a configuration? If it is from a configuration then it is not generic because this may be context based. One reason why the mapping is important because in some cases the client may ask for a codec but it may not be right for the server to send out all the information say for example some information stored in the tags. So in that case the server should apply the context based knowledge on what codec it should use. This is where the mapping comes into picture. In an internal discussion, there is a chance that for some cases like Export the server still needs to send out information in tags too. So in that case the mapping should deal this in a different way. So overall this mapping and usage of the codecs on the server side is more context based. When we have sytem tags in future the above argument applies (generally here it is better to suppress the tags from the clients). So the connection negotiation part is a protocol that we can decide but the mapping is one which is not very generic. Can try to make this mapping a pluggable one?
        Hide
        Andrew Purtell added a comment -

        Was just trying to save having to do fancy-dancing when may not be necessary (just-yet)

        I thought we could invest a little extra for the future while working in this area, but if I'm the only one who thinks this particular proposal is worthy, then we can just do the part Ram mentioned and address what actually needs doing right now.

        Show
        Andrew Purtell added a comment - Was just trying to save having to do fancy-dancing when may not be necessary (just-yet) I thought we could invest a little extra for the future while working in this area, but if I'm the only one who thinks this particular proposal is worthy, then we can just do the part Ram mentioned and address what actually needs doing right now.
        Hide
        stack added a comment -

        Don't you think we could use something like this for a 1.0 for future-proofing?

        Sure. Was just trying to save having to do fancy-dancing when may not be necessary (just-yet)

        Show
        stack added a comment - Don't you think we could use something like this for a 1.0 for future-proofing? Sure. Was just trying to save having to do fancy-dancing when may not be necessary (just-yet)
        Hide
        Andrew Purtell added a comment - - edited

        Small clarification - why does there have to be an invalid message? If it's a separate method/message, when the new client attempts to send it to an old server, it will get an easily recognizable exception back, right?

        For the old server it's an invalid message. Ok, there I agree that's over specified, I had one thing in mind but it's not the only way.

        Edit: Fix formatting

        Show
        Andrew Purtell added a comment - - edited Small clarification - why does there have to be an invalid message? If it's a separate method/message, when the new client attempts to send it to an old server, it will get an easily recognizable exception back, right? For the old server it's an invalid message. Ok, there I agree that's over specified, I had one thing in mind but it's not the only way. Edit: Fix formatting
        Hide
        Sergey Shelukhin added a comment -

        Small clarification - why does there have to be an invalid message?
        If it's a separate method/message, when the new client attempts to send it to an old server, it will get an easily recognizable exception back, right? (Thrift has exception for no-such-method, at least; don't recall for PB)

        Show
        Sergey Shelukhin added a comment - Small clarification - why does there have to be an invalid message? If it's a separate method/message, when the new client attempts to send it to an old server, it will get an easily recognizable exception back, right? (Thrift has exception for no-such-method, at least; don't recall for PB)
        Hide
        Andrew Purtell added a comment -

        On the over-engineering comment, if the proposal is ugly, that's one thing, but implementing coprocessors just to put in security was arguably over-engineering at the time too.

        Show
        Andrew Purtell added a comment - On the over-engineering comment, if the proposal is ugly, that's one thing, but implementing coprocessors just to put in security was arguably over-engineering at the time too.
        Hide
        Andrew Purtell added a comment -

        [Stack] Is this over-engineering?

        I wanted to put this idea out there and make a concrete proposal rather than a two line JIRA. We can wait on most of this until there are mixed major version deployments or when we have different codec flavors to choose from. Don't you think we could use something like this for a 1.0 for future-proofing?

        [Ram] The most important thing would be how we change the default behaviour of the base class in the client and in the server. We need a mapping for this and this mapping should be configurable.

        What Ram said.

        Show
        Andrew Purtell added a comment - [Stack] Is this over-engineering? I wanted to put this idea out there and make a concrete proposal rather than a two line JIRA. We can wait on most of this until there are mixed major version deployments or when we have different codec flavors to choose from. Don't you think we could use something like this for a 1.0 for future-proofing? [Ram] The most important thing would be how we change the default behaviour of the base class in the client and in the server. We need a mapping for this and this mapping should be configurable. What Ram said.
        Hide
        ramkrishna.s.vasudevan added a comment -

        The most important thing would be how we change the default behaviour of the base class in the client and in the server. We need a mapping for this and this mapping should be configurable.

        Show
        ramkrishna.s.vasudevan added a comment - The most important thing would be how we change the default behaviour of the base class in the client and in the server. We need a mapping for this and this mapping should be configurable.
        Hide
        stack added a comment -

        Is this over-engineering? What if we have it that server never drops support for any one codec (at least, within major versions). Then there is no need of negotiation? Client just says what they want? Server fails connection if can't give client want it wants. I'm guessing client doesn't want 'fallback'. If it does, it can do it itself by catching the UnsupportedCodecException and then go again w/ a different codec. See 'J.4.3.1. CellBlock Codecs' in http://hbase.apache.org/book.html#d0e14417

        Show
        stack added a comment - Is this over-engineering? What if we have it that server never drops support for any one codec (at least, within major versions). Then there is no need of negotiation? Client just says what they want? Server fails connection if can't give client want it wants. I'm guessing client doesn't want 'fallback'. If it does, it can do it itself by catching the UnsupportedCodecException and then go again w/ a different codec. See 'J.4.3.1. CellBlock Codecs' in http://hbase.apache.org/book.html#d0e14417
        Hide
        Andrew Purtell added a comment - - edited

        So to modify the OP with Elliot's suggestion:

        The client can optionally send a message, a manufactured "call" that would otherwise be invalid in some way, to the server asking for a list of supported cell codecs, and also optionally a preference ordered list of the codecs it wants. An older server should simply send back an error because the request is invalid except to servers supporting this feature.

        If the preference ordered list was provided, the server should enable the most preferred codec available for all further communication on the connection and send back a call response acknowledging the change and indicating which codec is in use, or an error indication if something went wrong.

        If a preference ordered list was not provided, or if no codec on the client's list was available, the server should send back a list of available codecs. The client can optionally send another message, a manufactured "call" that would otherwise be invalid in some way, to the server asking for it to use one of the codecs from the server supplied list for all further communication. Otherwise the server will continue to use the default codec. The server will send back a call response acknowledging the change or an error indication if the request cannot be honored.

        Suggestions for error handling:

        • Once codec negotiation is initiated by the client, if the client and server can't agree on a mutually intelligible codec then the connection setup should fail with a big fat ERROR in the log. I think this decision will be up to the client, since it is driving the negotiation. It would be good if the client can also send its decision to the server so the server can print an ERROR in its logs also.
        • We should not do anything about incompatible default codecs if codec negotiation is disabled (either due to older servers, or older clients, or a configuration toggle). Connection setup should proceed, but there will be failures later.
        Show
        Andrew Purtell added a comment - - edited So to modify the OP with Elliot's suggestion: The client can optionally send a message, a manufactured "call" that would otherwise be invalid in some way, to the server asking for a list of supported cell codecs, and also optionally a preference ordered list of the codecs it wants. An older server should simply send back an error because the request is invalid except to servers supporting this feature. If the preference ordered list was provided, the server should enable the most preferred codec available for all further communication on the connection and send back a call response acknowledging the change and indicating which codec is in use, or an error indication if something went wrong. If a preference ordered list was not provided, or if no codec on the client's list was available, the server should send back a list of available codecs. The client can optionally send another message, a manufactured "call" that would otherwise be invalid in some way, to the server asking for it to use one of the codecs from the server supplied list for all further communication. Otherwise the server will continue to use the default codec. The server will send back a call response acknowledging the change or an error indication if the request cannot be honored. Suggestions for error handling: Once codec negotiation is initiated by the client, if the client and server can't agree on a mutually intelligible codec then the connection setup should fail with a big fat ERROR in the log. I think this decision will be up to the client, since it is driving the negotiation. It would be good if the client can also send its decision to the server so the server can print an ERROR in its logs also. We should not do anything about incompatible default codecs if codec negotiation is disabled (either due to older servers, or older clients, or a configuration toggle). Connection setup should proceed, but there will be failures later.
        Hide
        Andrew Purtell added a comment -

        That sounds like a lot of round trips. Can we have the client send a preference ordered list of codec it wants.

        +1

        Show
        Andrew Purtell added a comment - That sounds like a lot of round trips. Can we have the client send a preference ordered list of codec it wants. +1
        Hide
        Elliott Clark added a comment -

        The client can optionally send a message, a manufactured "call" that would otherwise be invalid in some way, to the server asking for a list of supported cell codecs.

        That sounds like a lot of round trips. Can we have the client send a preference ordered list of codec it wants.

        Show
        Elliott Clark added a comment - The client can optionally send a message, a manufactured "call" that would otherwise be invalid in some way, to the server asking for a list of supported cell codecs. That sounds like a lot of round trips. Can we have the client send a preference ordered list of codec it wants.
        Hide
        ramkrishna.s.vasudevan added a comment -

        Sure..

        Sounds like a nasty hack Ram.

        I have tried removing this way of getting the reponse back and forth. Will post a design doc sooner.
        I am working on a cleaner way of doing rather than making the header negative.

        Show
        ramkrishna.s.vasudevan added a comment - Sure.. Sounds like a nasty hack Ram. I have tried removing this way of getting the reponse back and forth. Will post a design doc sooner. I am working on a cleaner way of doing rather than making the header negative.
        Hide
        Andrew Purtell added a comment -

        And the header of which can be forcefully made negative and the client/server understands the new response/request if the header size is negative?

        Sounds like a nasty hack Ram.

        So a 0.96 client won't barf on this new request/response message you propose? How is that a different approach than my OP on this issue? Perhaps you can put up a design doc explaining.

        Show
        Andrew Purtell added a comment - And the header of which can be forcefully made negative and the client/server understands the new response/request if the header size is negative? Sounds like a nasty hack Ram. So a 0.96 client won't barf on this new request/response message you propose? How is that a different approach than my OP on this issue? Perhaps you can put up a design doc explaining.
        Hide
        ramkrishna.s.vasudevan added a comment -

        We can do this with configuration or decorators. Either we add a new site configuration option where a comma separated list of codec classes

        Currently using this as configuration. We can use reflections to get the Codec related class paths too. I can later work on that part.

        I suppose it could be done with new optional fields in the connection header and response instead but this would need to support:

        Yes, adding new fields should work. Working on it. I would post a WIP patch on this. I feel we may need some new request/response too to make this 2 way communication. And the header of which can be forcefully made negative and the client/server understands the new response/request if the header size is negative?

        Show
        ramkrishna.s.vasudevan added a comment - We can do this with configuration or decorators. Either we add a new site configuration option where a comma separated list of codec classes Currently using this as configuration. We can use reflections to get the Codec related class paths too. I can later work on that part. I suppose it could be done with new optional fields in the connection header and response instead but this would need to support: Yes, adding new fields should work. Working on it. I would post a WIP patch on this. I feel we may need some new request/response too to make this 2 way communication. And the header of which can be forcefully made negative and the client/server understands the new response/request if the header size is negative?
        Hide
        Andrew Purtell added a comment -

        The connection header is the place where the codec and compression is determined i.e once per connection.

        My thinking is this is where the initial default codec for the connection would established like it is done now for 0.96. Then an additional round of negotiation would optionally happen, hence the idea for using 'invalid' calls after the initial connection header processing. I suppose it could be done with new optional fields in the connection header and response instead but this would need to support:

        • 0.96 clients talking to 0.98 servers expecting them to act like 0.96 servers
        • 0.98 clients talking to 0.96 servers
        • 0.98 clients talking to 0.98 servers, with codec negotiation, and server side substitution

        Need to introduce a connection response that on getting the option from the client to return the list of codecs. So on what basis should the server return them? Check it from the classpath?

        We can do this with configuration or decorators. Either we add a new site configuration option where a comma separated list of codec classes could go, or we create a new decorator type that marks codecs and scan the classpath for the decorator at server startup and add the discovered classes to a list.

        Show
        Andrew Purtell added a comment - The connection header is the place where the codec and compression is determined i.e once per connection. My thinking is this is where the initial default codec for the connection would established like it is done now for 0.96. Then an additional round of negotiation would optionally happen, hence the idea for using 'invalid' calls after the initial connection header processing. I suppose it could be done with new optional fields in the connection header and response instead but this would need to support: 0.96 clients talking to 0.98 servers expecting them to act like 0.96 servers 0.98 clients talking to 0.96 servers 0.98 clients talking to 0.98 servers, with codec negotiation, and server side substitution Need to introduce a connection response that on getting the option from the client to return the list of codecs. So on what basis should the server return them? Check it from the classpath? We can do this with configuration or decorators. Either we add a new site configuration option where a comma separated list of codec classes could go, or we create a new decorator type that marks codecs and scan the classpath for the decorator at server startup and add the discovered classes to a list.
        Hide
        ramkrishna.s.vasudevan added a comment -

        Currently we have a connectionheader, request header and request response on the RPC communication side.
        The connection header is the place where the codec and compression is determined i.e once per connection. Once the connection header is processed the remaining normal Request headers are processed.
        So as to introduce the basic negotiaion protocol
        1) Need to introduce a connection response that on getting the option from the client to return the list of codecs. So on what basis should the server return them? Check it from the classpath?
        2)This response to be interpreted by the client and select one codec from the list and send that codec as a new response from the client to the server.
        3) To this response, the server tries to use the codec given by the client and again returns a success/failure msg if it was able to use the specified codec.
        4)The mapping of individual codecs and the extended functionality of codecs that needs specific behaviour on the server side am yet to work on that.

        Show
        ramkrishna.s.vasudevan added a comment - Currently we have a connectionheader, request header and request response on the RPC communication side. The connection header is the place where the codec and compression is determined i.e once per connection. Once the connection header is processed the remaining normal Request headers are processed. So as to introduce the basic negotiaion protocol 1) Need to introduce a connection response that on getting the option from the client to return the list of codecs. So on what basis should the server return them? Check it from the classpath? 2)This response to be interpreted by the client and select one codec from the list and send that codec as a new response from the client to the server. 3) To this response, the server tries to use the codec given by the client and again returns a success/failure msg if it was able to use the specified codec. 4)The mapping of individual codecs and the extended functionality of codecs that needs specific behaviour on the server side am yet to work on that.
        Hide
        ramkrishna.s.vasudevan added a comment -

        Correct we need something like this. Where the client is able to tell what it would send and what the server should use.
        The server should respond with what server will be using from the mapping.

        Show
        ramkrishna.s.vasudevan added a comment - Correct we need something like this. Where the client is able to tell what it would send and what the server should use. The server should respond with what server will be using from the mapping.

          People

          • Assignee:
            ramkrishna.s.vasudevan
            Reporter:
            Andrew Purtell
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:

              Development