Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-1696

Kafka should be able to generate Hadoop delegation tokens

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: 1.1.0
    • Component/s: security
    • Labels:
      None

      Description

      For access from MapReduce/etc jobs run on behalf of a user.

        Issue Links

          Activity

          Hide
          omkreddy Manikumar added a comment -

          Guozhang Wang This got delayed due to some internal works. Will raise the first version PR by this month end.

          Show
          omkreddy Manikumar added a comment - Guozhang Wang This got delayed due to some internal works. Will raise the first version PR by this month end.
          Hide
          guozhang Guozhang Wang added a comment -

          Manikumar Ashish Singh Parth Brahmbhatt Are you still working on this JIRA (KIP-48) right now?

          Show
          guozhang Guozhang Wang added a comment - Manikumar Ashish Singh Parth Brahmbhatt Are you still working on this JIRA (KIP-48) right now?
          Hide
          omkreddy Manikumar added a comment -

          Thanks for creating sub-tasks. About authentication, we are still discussing about credentials of
          SCRAM mechanism. As mentioned in the mailing list, I am planning to pass tokenID, hmac as username and
          password. Now we need to store SCRAM credentials in ZK along with token details. Pl check if this
          fits into your design. This works depends on KAFKA-3751.

          Show
          omkreddy Manikumar added a comment - Thanks for creating sub-tasks. About authentication, we are still discussing about credentials of SCRAM mechanism. As mentioned in the mailing list, I am planning to pass tokenID, hmac as username and password. Now we need to store SCRAM credentials in ZK along with token details. Pl check if this fits into your design. This works depends on KAFKA-3751 .
          Hide
          singhashish Ashish Singh added a comment -

          Manikumar I had to convert this JIRA from sub-task to a new feature JIRA, to be able to create sub-tasks. Also created some sub-tasks. I have left most of them open, except the first one, for which I plan to submit a PR shortly. Also, I have partially looked into authentication via tokens, basically add a TokenAuthenticator, so if you are Ok with me taking up that task, I will be happy to.

          Show
          singhashish Ashish Singh added a comment - Manikumar I had to convert this JIRA from sub-task to a new feature JIRA, to be able to create sub-tasks. Also created some sub-tasks. I have left most of them open, except the first one, for which I plan to submit a PR shortly. Also, I have partially looked into authentication via tokens, basically add a TokenAuthenticator, so if you are Ok with me taking up that task, I will be happy to.
          Hide
          omkreddy Manikumar added a comment -

          Ashish Singh Yes, Pl go ahead and create sub-jiras and PRs , So that we avoid any duplicate efforts.

          Show
          omkreddy Manikumar added a comment - Ashish Singh Yes, Pl go ahead and create sub-jiras and PRs , So that we avoid any duplicate efforts.
          Hide
          singhashish Ashish Singh added a comment -

          Manikumar as I indicated on discuss list, I have made some progress along this to put together a POC. I would love if my work can help us speed up here. Do you think it is OK, if I propose some sub-jiras and add PRs over there. There is a lot still left to do, but I can quickly throw some initial pieces that will help us with start. Makes sense?

          Show
          singhashish Ashish Singh added a comment - Manikumar as I indicated on discuss list, I have made some progress along this to put together a POC. I would love if my work can help us speed up here. Do you think it is OK, if I propose some sub-jiras and add PRs over there. There is a lot still left to do, but I can quickly throw some initial pieces that will help us with start. Makes sense?
          Hide
          omkreddy Manikumar added a comment -

          .Ashish Singh Sorry for the late reply. I reinitiated the KIP discussion. Pl review the KIP. I will the divide the work into multiple JIRAs.

          Show
          omkreddy Manikumar added a comment - . Ashish Singh Sorry for the late reply. I reinitiated the KIP discussion. Pl review the KIP. I will the divide the work into multiple JIRAs.
          Hide
          singhashish Ashish Singh added a comment -

          Manikumar would you be able to provide an update on what progress have you made. As sriharsha chintalapani suggested, we can collaborate to help things move faster.

          Show
          singhashish Ashish Singh added a comment - Manikumar would you be able to provide an update on what progress have you made. As sriharsha chintalapani suggested, we can collaborate to help things move faster.
          Hide
          sriharsha Sriharsha Chintalapani added a comment -

          Ashish Singh Manikumar is working on it. I think its best to break this down into multiple JIRAs and distribute the work.

          Show
          sriharsha Sriharsha Chintalapani added a comment - Ashish Singh Manikumar is working on it. I think its best to break this down into multiple JIRAs and distribute the work.
          Hide
          singhashish Ashish Singh added a comment -

          It has been a while, since any progress has been made on this. It is becoming increasingly important for a lot of users. I am assuming that it is OK, if I assign this JIRA to myself and start working on it. Multiple people have asked for updated here and on associated discuss list. I will wait for a day before assigning the JIRA to myself.

          Show
          singhashish Ashish Singh added a comment - It has been a while, since any progress has been made on this. It is becoming increasingly important for a lot of users. I am assuming that it is OK, if I assign this JIRA to myself and start working on it. Multiple people have asked for updated here and on associated discuss list. I will wait for a day before assigning the JIRA to myself.
          Hide
          chtyim Terence Yim added a comment -

          May I ask for any update about this feature? Any idea in which Kafka version will have this?

          Show
          chtyim Terence Yim added a comment - May I ask for any update about this feature? Any idea in which Kafka version will have this?
          Hide
          singhashish Ashish Singh added a comment -

          Good to know sriharsha chintalapani! Thanks for the quick response.

          Show
          singhashish Ashish Singh added a comment - Good to know sriharsha chintalapani ! Thanks for the quick response.
          Hide
          sriharsha Sriharsha Chintalapani added a comment -

          Ashish Singh Yes we are actively working on it.

          Show
          sriharsha Sriharsha Chintalapani added a comment - Ashish Singh Yes we are actively working on it.
          Hide
          singhashish Ashish Singh added a comment -

          Parth Brahmbhatt are you still working on this?

          Show
          singhashish Ashish Singh added a comment - Parth Brahmbhatt are you still working on this?
          Hide
          eronwright Eron Wright added a comment -

          I'd like clarification on whether renewal is possible using the delegation token for authentication, and whether an infinite expiration will be possible (with the appropriate configuration).

          I'm thinking of the scenario of a production-level Flink streaming job, consuming a topic in perpetuity. The client that submits the job should obtain a delegation token using their Kerberos credential, then hand the delegation token to the running job. The job should periodically renew the token(s). Ideally the delegation token may be used to authenticate the renewal request. It doesn't seem easy to have Flink use a Kerberos credential to renew it, but may be possible with a service principal of some kind.

          The notion that the token eventually expires seems incompatible with long-running jobs. A key purpose of delegation tokens is to avoid distributing keytabs, but how does that reconcile with expiration?

          Show
          eronwright Eron Wright added a comment - I'd like clarification on whether renewal is possible using the delegation token for authentication, and whether an infinite expiration will be possible (with the appropriate configuration). I'm thinking of the scenario of a production-level Flink streaming job, consuming a topic in perpetuity. The client that submits the job should obtain a delegation token using their Kerberos credential, then hand the delegation token to the running job. The job should periodically renew the token(s). Ideally the delegation token may be used to authenticate the renewal request. It doesn't seem easy to have Flink use a Kerberos credential to renew it, but may be possible with a service principal of some kind. The notion that the token eventually expires seems incompatible with long-running jobs. A key purpose of delegation tokens is to avoid distributing keytabs, but how does that reconcile with expiration?
          Hide
          gwenshap Gwen Shapira added a comment -

          OK, sounds good. Thanks.
          It does make this issue depend on KIP-43 getting in.

          Show
          gwenshap Gwen Shapira added a comment - OK, sounds good. Thanks. It does make this issue depend on KIP-43 getting in.
          Hide
          sriharsha Sriharsha Chintalapani added a comment -

          Gwen Shapira Port is not relevant to this JIRA. That discussion is part of KIP-43. And we decided to single port for SASL. Each mechanism inside SASL shouldn't have its own port.

          Show
          sriharsha Sriharsha Chintalapani added a comment - Gwen Shapira Port is not relevant to this JIRA. That discussion is part of KIP-43. And we decided to single port for SASL. Each mechanism inside SASL shouldn't have its own port.
          Hide
          parth.brahmbhatt Parth Brahmbhatt added a comment -

          Gwen Shapira I think that discussion is happening as part of another KIP and no matter what we chose I don't think it affects the delegation token design.

          Show
          parth.brahmbhatt Parth Brahmbhatt added a comment - Gwen Shapira I think that discussion is happening as part of another KIP and no matter what we chose I don't think it affects the delegation token design.
          Hide
          singhashish Ashish Singh added a comment -

          I think posting a discuss thread makes sense. Discuss thread will help get
          more audience and suggestions. If we choose to go the coordinator route, we
          can always adopt KIP while discuss is going on. However, it would be nice
          to mention that this discussion has mentioned on the JIRA. Or maybe just
          copy paste the content from here to avoid duplicating points.

          On Thursday, February 25, 2016, Parth Brahmbhatt (JIRA) <jira@apache.org>


          Ashish 🎤h

          Show
          singhashish Ashish Singh added a comment - I think posting a discuss thread makes sense. Discuss thread will help get more audience and suggestions. If we choose to go the coordinator route, we can always adopt KIP while discuss is going on. However, it would be nice to mention that this discussion has mentioned on the JIRA. Or maybe just copy paste the content from here to avoid duplicating points. On Thursday, February 25, 2016, Parth Brahmbhatt (JIRA) <jira@apache.org> – Ashish 🎤h
          Hide
          gwenshap Gwen Shapira added a comment -

          One thing I can't find in the KIP:

          • Will we support SASL/Kerberos and SASL/Digest on same port?
          Show
          gwenshap Gwen Shapira added a comment - One thing I can't find in the KIP: Will we support SASL/Kerberos and SASL/Digest on same port?
          Hide
          gwenshap Gwen Shapira added a comment -

          I don't have specific opinions on the KIP yet, I have not read it...

          I was responding to Ashish who said: "moving away from ZK, tends to be the general direction"
          This is incorrect, and I want to correct this impression.

          Show
          gwenshap Gwen Shapira added a comment - I don't have specific opinions on the KIP yet, I have not read it... I was responding to Ashish who said: "moving away from ZK, tends to be the general direction" This is incorrect, and I want to correct this impression.
          Hide
          parth.brahmbhatt Parth Brahmbhatt added a comment -

          Gwen Shapira Sriharsha Chintalapani Please share your preference here so I can update the KIP accordingly and start a discussion thread. I wanted to try SASL over MD5 to ensure that this design will work as is but I don't see myself writing that prototype soon so might as well start the KIP discussion.

          Show
          parth.brahmbhatt Parth Brahmbhatt added a comment - Gwen Shapira Sriharsha Chintalapani Please share your preference here so I can update the KIP accordingly and start a discussion thread. I wanted to try SASL over MD5 to ensure that this design will work as is but I don't see myself writing that prototype soon so might as well start the KIP discussion.
          Hide
          gwenshap Gwen Shapira added a comment -

          Just to clarify - the move away from ZK is for clients, not for brokers.
          The suggestions to make Kafka metadata store pluggable were never accepted and are far from being in consensus.

          I would not base any broker-side design on the assumption that ZK is going anywhere (just on the assumption that clients shouldn't have to depend on it).

          Show
          gwenshap Gwen Shapira added a comment - Just to clarify - the move away from ZK is for clients, not for brokers. The suggestions to make Kafka metadata store pluggable were never accepted and are far from being in consensus. I would not base any broker-side design on the assumption that ZK is going anywhere (just on the assumption that clients shouldn't have to depend on it).
          Hide
          singhashish Ashish Singh added a comment -

          Parth Brahmbhatt thanks for the detailed analysis, as you have mentioned the disadvantages are not really big ones here. Moreover moving away from ZK, tends to be the general direction. KIP-4 is WIP and suggestions have been made to make Kafka metadata store pluggable. I am more in favor of going via controller. I do agree it will be more work and this work is kind of in high demand. Probably having it in 0.10.0.0 would be ideal. I am willing to help out, if needed.

          Show
          singhashish Ashish Singh added a comment - Parth Brahmbhatt thanks for the detailed analysis, as you have mentioned the disadvantages are not really big ones here. Moreover moving away from ZK, tends to be the general direction. KIP-4 is WIP and suggestions have been made to make Kafka metadata store pluggable. I am more in favor of going via controller. I do agree it will be more work and this work is kind of in high demand. Probably having it in 0.10.0.0 would be ideal. I am willing to help out, if needed.
          Hide
          parth.brahmbhatt Parth Brahmbhatt added a comment - - edited

          So here is how that request path would work in my mind:

          • Client sends request for token acquisition to any broker.
          • Broker forwards the request to the controller.
          • Controller generates the token and pushes the tokens to all brokers. (Will need a new API)
          • Controller responds back to original broker with the token.
          • Broker responds back to client with the token.

          Renewal is pretty much the same.

          The race condition you are describing can still happen in the above case during renewal because controller may have pushed the renewal information to a subset of broker and die. The clients depending on which broker it connects to may get an exception or success. I do agree though that given controller would not have responded back with success the original renew request should be retried and most likely the scenario can be avoided.

          If the above steps seems right , here are the advantages of this approach:

          Advantage:

          • Token generation/renewal will not involve zookeeper. I am not too worried about the load on zookeeper added due to this but it definitely seems more secure and follows the Hadoop model more closely. However zookeeper needs to be secure for lot of other things in kafka so not sure if this should really be a concern.
          • Clients will get better consistency.

          Disadvantage:

          • We will have to add new APIs to support controller pushing tokens to brokers on top of the minimal APIs that are currently proposed. I like the publicly available APIs to be minimal and I like them to be something that we expect clients to use + this adds more development complexity. Overall this seems like a more philosophical thing so depending on who you ask they may see this as disadvantage or not.
          • We will also have to add APIs to support the bootstrapping case. What I mean is , when a new broker comes up it will have to get all delegation tokens from the controller so we will again need to add new APIs like getAllTokens. Again some of us may see that as disadvantage and some may not.
          • In catastrophic failures where all brokers go down, the tokens will be lost even if servers are restarted as tokens are not persisted anywhere. Granted if something like this happens customer has bigger things to worry about but if they don't have to regenerate/redistribute tokens that is one less thing.

          I don't see strong reasons to go one way or another so I would still like to go with zookeeper but don't really feel strongly about it. If you think I have mischaracterized what you were proposing feel free to add more details or list and other advantages/disadvantages.

          Show
          parth.brahmbhatt Parth Brahmbhatt added a comment - - edited So here is how that request path would work in my mind: Client sends request for token acquisition to any broker. Broker forwards the request to the controller. Controller generates the token and pushes the tokens to all brokers. (Will need a new API) Controller responds back to original broker with the token. Broker responds back to client with the token. Renewal is pretty much the same. The race condition you are describing can still happen in the above case during renewal because controller may have pushed the renewal information to a subset of broker and die. The clients depending on which broker it connects to may get an exception or success. I do agree though that given controller would not have responded back with success the original renew request should be retried and most likely the scenario can be avoided. If the above steps seems right , here are the advantages of this approach: Advantage: Token generation/renewal will not involve zookeeper. I am not too worried about the load on zookeeper added due to this but it definitely seems more secure and follows the Hadoop model more closely. However zookeeper needs to be secure for lot of other things in kafka so not sure if this should really be a concern. Clients will get better consistency. Disadvantage: We will have to add new APIs to support controller pushing tokens to brokers on top of the minimal APIs that are currently proposed. I like the publicly available APIs to be minimal and I like them to be something that we expect clients to use + this adds more development complexity. Overall this seems like a more philosophical thing so depending on who you ask they may see this as disadvantage or not. We will also have to add APIs to support the bootstrapping case. What I mean is , when a new broker comes up it will have to get all delegation tokens from the controller so we will again need to add new APIs like getAllTokens. Again some of us may see that as disadvantage and some may not. In catastrophic failures where all brokers go down, the tokens will be lost even if servers are restarted as tokens are not persisted anywhere. Granted if something like this happens customer has bigger things to worry about but if they don't have to regenerate/redistribute tokens that is one less thing. I don't see strong reasons to go one way or another so I would still like to go with zookeeper but don't really feel strongly about it. If you think I have mischaracterized what you were proposing feel free to add more details or list and other advantages/disadvantages.
          Hide
          singhashish Ashish Singh added a comment -

          Parth Brahmbhatt I am in favor of keeping this token information as much contained within brokers, not on ZK. Infact, how does the idea of limiting the delegation token generation to coordinator sounds to you? So, delegationToken req goes to co-ordinator and it is responsible of passing the token to client after it has been distributed to other brokers. This will avoid a race condition of brokers not updated with delegation token before clients try using it, which is possible in current design suggestion.

          Show
          singhashish Ashish Singh added a comment - Parth Brahmbhatt I am in favor of keeping this token information as much contained within brokers, not on ZK. Infact, how does the idea of limiting the delegation token generation to coordinator sounds to you? So, delegationToken req goes to co-ordinator and it is responsible of passing the token to client after it has been distributed to other brokers. This will avoid a race condition of brokers not updated with delegation token before clients try using it, which is possible in current design suggestion.
          Hide
          parth.brahmbhatt Parth Brahmbhatt added a comment -

          Ashish Singh Thanks for taking the time to review. I will add the errorcode/exception to the KIP and sample CLI.

          For point 2, can you elaborate why it might be a better idea to communicate via coordinator? When I was thinking about distributing the tokens I felt the coordinator will just add extra complexity and wont buy us much. We already have multiple things ACLs/Configs for which we rely on zookeeper watchers so it seemed like a natural choice.

          Show
          parth.brahmbhatt Parth Brahmbhatt added a comment - Ashish Singh Thanks for taking the time to review. I will add the errorcode/exception to the KIP and sample CLI. For point 2, can you elaborate why it might be a better idea to communicate via coordinator? When I was thinking about distributing the tokens I felt the coordinator will just add extra complexity and wont buy us much. We already have multiple things ACLs/Configs for which we rely on zookeeper watchers so it seemed like a natural choice.
          Hide
          singhashish Ashish Singh added a comment -

          Parth Brahmbhatt thanks for working out the design. Great write up. I think it is in a decent shape to initiate a discuss thread on the KIP. I have following questions/ concerns.

          1. We probably need new error-codes to indicate token renewal failure, etc. I will probably be a good idea to list them as well.
          2. Probably use coordinator to communicate token changes from one broker to other brokers. So, a broker generating token will add to ZK, coordinator watches for new or changed tokens and communicates to other brokers in cluster.
          3. May be provide an outline on the cli tool for creating/renewing/deleting delegation tokens.

          Show
          singhashish Ashish Singh added a comment - Parth Brahmbhatt thanks for working out the design. Great write up. I think it is in a decent shape to initiate a discuss thread on the KIP. I have following questions/ concerns. 1. We probably need new error-codes to indicate token renewal failure, etc. I will probably be a good idea to list them as well. 2. Probably use coordinator to communicate token changes from one broker to other brokers. So, a broker generating token will add to ZK, coordinator watches for new or changed tokens and communicates to other brokers in cluster. 3. May be provide an outline on the cli tool for creating/renewing/deleting delegation tokens.
          Hide
          parth.brahmbhatt Parth Brahmbhatt added a comment -

          Gwen Shapira sriharsha chintalapani Ashish Singh I posted an initial KIP Draft https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka. I haven't yet opened a Discuss thread as I need to verify some assumptions I have made.

          Show
          parth.brahmbhatt Parth Brahmbhatt added a comment - Gwen Shapira sriharsha chintalapani Ashish Singh I posted an initial KIP Draft https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka . I haven't yet opened a Discuss thread as I need to verify some assumptions I have made.
          Hide
          gwenshap Gwen Shapira added a comment -

          sriharsha chintalapani, you are following Rajini Sivaram's KIP on new SASL mechanisms, right?

          Hadoop implements Delegation Tokens as a SASL mechanism and I was wondering if you were planning on doing the same? If so, you may have an opinion on how the mechanism is determined. If you are not planning on hooking into SASL, then never mind

          Show
          gwenshap Gwen Shapira added a comment - sriharsha chintalapani , you are following Rajini Sivaram 's KIP on new SASL mechanisms, right? Hadoop implements Delegation Tokens as a SASL mechanism and I was wondering if you were planning on doing the same? If so, you may have an opinion on how the mechanism is determined. If you are not planning on hooking into SASL, then never mind
          Hide
          sriharsha Sriharsha Chintalapani added a comment -

          Ashish Singh We are working on it.Will post the KIP to wiki.

          Show
          sriharsha Sriharsha Chintalapani added a comment - Ashish Singh We are working on it.Will post the KIP to wiki.
          Hide
          singhashish Ashish Singh added a comment -

          Parth Brahmbhatt just curious if you have the KIP draft up for review. Willing to help, if you need any. Thanks!

          Show
          singhashish Ashish Singh added a comment - Parth Brahmbhatt just curious if you have the KIP draft up for review. Willing to help, if you need any. Thanks!
          Hide
          singhashish Ashish Singh added a comment -

          Thanks Gwen Shapira, Parth Brahmbhatt! Looking forward to the KIP proposal.

          Show
          singhashish Ashish Singh added a comment - Thanks Gwen Shapira , Parth Brahmbhatt ! Looking forward to the KIP proposal.
          Hide
          parth.brahmbhatt Parth Brahmbhatt added a comment -

          I will assign it to my self and file a KIP

          Show
          parth.brahmbhatt Parth Brahmbhatt added a comment - I will assign it to my self and file a KIP
          Hide
          gwenshap Gwen Shapira added a comment -

          I am not working on it, so you guys figure out ownership

          Show
          gwenshap Gwen Shapira added a comment - I am not working on it, so you guys figure out ownership
          Hide
          singhashish Ashish Singh added a comment -

          sriharsha chintalapani Parth Brahmbhatt Gwen Shapira seems like we have similar needs and would like to know if someone is working on this. If required I can pitch in.

          Show
          singhashish Ashish Singh added a comment - sriharsha chintalapani Parth Brahmbhatt Gwen Shapira seems like we have similar needs and would like to know if someone is working on this. If required I can pitch in.
          Hide
          parth.brahmbhatt Parth Brahmbhatt added a comment -

          Gwen Shapira Are you still working on this? We have some customers that needs this feature and if you have not started the design work I would like to take this over.

          Show
          parth.brahmbhatt Parth Brahmbhatt added a comment - Gwen Shapira Are you still working on this? We have some customers that needs this feature and if you have not started the design work I would like to take this over.
          Hide
          sriharsha Sriharsha Chintalapani added a comment -

          Gwen Shapira sounds good to me. Since it was unassigned I assigned it to myself. Feel free to reassign. I'll work on putting together my thoughts on this.

          Show
          sriharsha Sriharsha Chintalapani added a comment - Gwen Shapira sounds good to me. Since it was unassigned I assigned it to myself. Feel free to reassign. I'll work on putting together my thoughts on this.
          Hide
          gwenshap Gwen Shapira added a comment -

          I started on the design doc, but I'll admit that I was not in a hurry since this is blocked on the Kerberos support anyway.

          If you have your own thoughts and want to collaborate on the design, I'll be happy to work together. We can figure out who is doing the coding when it becomes more relevant.

          Show
          gwenshap Gwen Shapira added a comment - I started on the design doc, but I'll admit that I was not in a hurry since this is blocked on the Kerberos support anyway. If you have your own thoughts and want to collaborate on the design, I'll be happy to work together. We can figure out who is doing the coding when it becomes more relevant.
          Hide
          sriharsha Sriharsha Chintalapani added a comment -

          Gwen Shapira Incase if you not started working on it I am interested in taking it up. Thanks.

          Show
          sriharsha Sriharsha Chintalapani added a comment - Gwen Shapira Incase if you not started working on it I am interested in taking it up. Thanks.
          Hide
          gwenshap Gwen Shapira added a comment -

          Since the delegation tokens depend on SASL/MD5-DIGEST, it will be easier to implement after the SASL/GSS (Kerberos) is in. We'll want to re-use a lot of the same code in both cases.

          Also, there's a good chance we'll use ZK to store the broker secrets, so being able to authenticate with secure ZK is a dependency.

          Show
          gwenshap Gwen Shapira added a comment - Since the delegation tokens depend on SASL/MD5-DIGEST, it will be easier to implement after the SASL/GSS (Kerberos) is in. We'll want to re-use a lot of the same code in both cases. Also, there's a good chance we'll use ZK to store the broker secrets, so being able to authenticate with secure ZK is a dependency.
          Hide
          gwenshap Gwen Shapira added a comment -

          The way Hadoop currently works, MapReduce tasks (and similarly anything in YARN containers) does not have access to Kerberos tickets. Instead, it uses DIGEST-MD5 authentication scheme. This is explained in great detail here: http://carfield.com.hk/document/distributed/hadoop-security-design.pdf

          The gist is that when the job is started, the job client obtains a signed "delegation token" and secret "token authenticator" from the service (typically NN or JT). These are distributed to the mappers (or containers) through a credentials cache. The server keeps part of the secret (typically in NN/JT memory) and uses that to authenticate the clients.

          In order for Kafka to support this authentication scheme, we need to:
          1) Be able to generate the token
          2) Be able to store the Broker half of the secret in a way that is secured but accessible to all brokers (probably ZK)
          3) Be able to authenticate clients using the tokens

          Other goals I see are:
          1) Reuse Hadoop code to avoid re-inventing the wheel, but otherwise minimize dependencies
          2) If at all possible, without patching Hadoop, we want this to be transparent to existing MR jobs (especially Camus). If not possible, we should provide an easy-to-use authentication API to minimize the required changes.

          Open question: Do we need to support token renewal?

          I'm working on a design doc for this part, but it will probably only happen after Strata.

          Show
          gwenshap Gwen Shapira added a comment - The way Hadoop currently works, MapReduce tasks (and similarly anything in YARN containers) does not have access to Kerberos tickets. Instead, it uses DIGEST-MD5 authentication scheme. This is explained in great detail here: http://carfield.com.hk/document/distributed/hadoop-security-design.pdf The gist is that when the job is started, the job client obtains a signed "delegation token" and secret "token authenticator" from the service (typically NN or JT). These are distributed to the mappers (or containers) through a credentials cache. The server keeps part of the secret (typically in NN/JT memory) and uses that to authenticate the clients. In order for Kafka to support this authentication scheme, we need to: 1) Be able to generate the token 2) Be able to store the Broker half of the secret in a way that is secured but accessible to all brokers (probably ZK) 3) Be able to authenticate clients using the tokens Other goals I see are: 1) Reuse Hadoop code to avoid re-inventing the wheel, but otherwise minimize dependencies 2) If at all possible, without patching Hadoop, we want this to be transparent to existing MR jobs (especially Camus). If not possible, we should provide an easy-to-use authentication API to minimize the required changes. Open question: Do we need to support token renewal? I'm working on a design doc for this part, but it will probably only happen after Strata.

            People

            • Assignee:
              parth.brahmbhatt Parth Brahmbhatt
              Reporter:
              jkreps Jay Kreps
            • Votes:
              5 Vote for this issue
              Watchers:
              35 Start watching this issue

              Dates

              • Created:
                Updated:

                Development