HBase
  1. HBase
  2. HBASE-3615

Implement token based DIGEST-MD5 authentication for MapReduce tasks

    Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.92.0
    • Component/s: IPC/RPC, security
    • Labels:
      None

      Description

      HBase security currently supports Kerberos authentication for clients, but this isn't sufficient for map-reduce interoperability, where tasks execute without Kerberos credentials. In order to fully interoperate with map-reduce clients, we will need to provide our own token authentication mechanism, mirroring the Hadoop token authentication mechanisms. This will require obtaining an HBase authentication token for the user when the job is submitted, serializing it to a secure location, and then, at task execution, having the client or task code de-serialize the stored authentication token and use that in the HBase client authentication process.

      A detailed implementation proposal is sketched out on the wiki:
      http://wiki.apache.org/hadoop/Hbase/HBaseTokenAuthentication

      1. HBASE-3615.patch
        118 kB
        Gary Helmling

        Issue Links

          Activity

          Hide
          Andrew Purtell added a comment -

          Bringing into 0.92.

          Show
          Andrew Purtell added a comment - Bringing into 0.92.
          Hide
          stack added a comment -

          I took a read off the wiki page Gary. The goals seem pretty impossible though I'm sure you'll pull it off.

          will need to persist somewhere (zookeeper?) to allow for master restarts and failover

          ZK seems good for storing a small thing that does not change. Will the key be generally available if its in zk?

          Is there a 'secure' location in hdfs?

          could be on region checkin/heartbeats, though stack is removing those

          Yes. Trying.

          but do we really need to do token renewal

          When would you need this?

          Yes, as you note, for case of multiple masters, on failover, new master will need to be able to pick up the token.

          FileSystem.getCanonicalServiceName() is used as the alias for HDFS delegation tokens, what should we use?

          Whats this? We need name for cluster instance? I suppose we can't use master ip plus port because could change with time. The zk ensemble string plus the zk rootdir? This is ugly – zk1,zk2,zk3,zk4,zk5/hbase3 where hbase3 is the homedir in zk of the third cluster running against this ensemble – in that what do you do if you are passed zk1,zk2,zk3 and zk1,zk2,zk3,zk4,zk5? Are they the same ensemble. I think so.

          I can't comment on how secure it all is Gary. I'm clueless on that stuff.

          Doc. looks great.

          Show
          stack added a comment - I took a read off the wiki page Gary. The goals seem pretty impossible though I'm sure you'll pull it off. will need to persist somewhere (zookeeper?) to allow for master restarts and failover ZK seems good for storing a small thing that does not change. Will the key be generally available if its in zk? Is there a 'secure' location in hdfs? could be on region checkin/heartbeats, though stack is removing those Yes. Trying. but do we really need to do token renewal When would you need this? Yes, as you note, for case of multiple masters, on failover, new master will need to be able to pick up the token. FileSystem.getCanonicalServiceName() is used as the alias for HDFS delegation tokens, what should we use? Whats this? We need name for cluster instance? I suppose we can't use master ip plus port because could change with time. The zk ensemble string plus the zk rootdir? This is ugly – zk1,zk2,zk3,zk4,zk5/hbase3 where hbase3 is the homedir in zk of the third cluster running against this ensemble – in that what do you do if you are passed zk1,zk2,zk3 and zk1,zk2,zk3,zk4,zk5? Are they the same ensemble. I think so. I can't comment on how secure it all is Gary. I'm clueless on that stuff. Doc. looks great.
          Hide
          Gary Helmling added a comment -

          ZK seems good for storing a small thing that does not change. Will the key be generally available if its in zk?

          This will rely on securing ZK with kerberos auth (Eugene has a patch for ZOOKEEPER-938) and setting up ACLs. We already need ZK secured as we use it to broadcast changes in ACLs to the RSs, so this seems to fit with that too. Goes without saying, but using ZK security will be optional config as well from the HBase standpoint.

          We'll want to periodically roll master keys and communicate updates to RSs. Hadoop rolls the "current" key every 24 hrs and keeps the last 7, so ZK again seems a good fit to communicate the changes. I considered just storing the key IDs in ZK for change notification and storing key data in HDFS using file permissions for security, but that's just another piece that can break when we're securing ZK anyway.

          When would you need this? [token renewal]

          Hadoop again does this. I think the jobtracker is designed as the token "renewer" and then it pings the NN to keep it live for up to 7 days. In that case, each token has a "max date", but expiration is computed separately as current time + some window. Expiration is a bit fuzzy in that implementation though, as the renewer can still resurrect expired tokens if the current time < the "max date" in the token. In theory, it limits the window during which token disclosure allows impersonating the user. If the token expires in 24 hours without renewal, and the MR job completes in less than that time, then a disclosure of the token 25 hrs after issue, when the token has expired, and the JT has not needed to renew it, will not allow the token to be re-used to impersonate the user.

          However, this doesn't really close the loop if you can somehow trick the JT (the designated "renewer") into resurrecting the expired token for you. Also, we can't use the built in JT renewal as it only works for Hadoop DelegationTokens, so something else would have to handle it for the duration of a job execution. And it's not clear to me that it's a meaningful enhancement in security. So I've ignored the expiration/max date distinction and just made it expire date.

          On failure-over, the master would read in the current master keys from ZK and repopulate the valid tokens in memory as they're used, validating that they use an existing master key and haven't expired, err.. "maxed out", yet.

          Whats this? We need name for cluster instance? I suppose we can't use master ip plus port because could change with time. The zk ensemble string plus the zk rootdir?

          Yeah, this part is a bit tricky. I hadn't thought of cluster ensemble subsets. I was going to ping JD on if replication had anything to use for similar purposes – uniquely identify clusters to prevent replication loops, say. Talking with Andy, he suggested generating a UUID on initial FS setup and adding it to hbase.version. From there master could pop it up in ZK on startup? Maybe I should open a separate JIRA for discussing that bit?

          Thanks for the comments! I suppose I should clarify some of these bits on the wiki page.

          Show
          Gary Helmling added a comment - ZK seems good for storing a small thing that does not change. Will the key be generally available if its in zk? This will rely on securing ZK with kerberos auth (Eugene has a patch for ZOOKEEPER-938 ) and setting up ACLs. We already need ZK secured as we use it to broadcast changes in ACLs to the RSs, so this seems to fit with that too. Goes without saying, but using ZK security will be optional config as well from the HBase standpoint. We'll want to periodically roll master keys and communicate updates to RSs. Hadoop rolls the "current" key every 24 hrs and keeps the last 7, so ZK again seems a good fit to communicate the changes. I considered just storing the key IDs in ZK for change notification and storing key data in HDFS using file permissions for security, but that's just another piece that can break when we're securing ZK anyway. When would you need this? [token renewal] Hadoop again does this. I think the jobtracker is designed as the token "renewer" and then it pings the NN to keep it live for up to 7 days. In that case, each token has a "max date", but expiration is computed separately as current time + some window. Expiration is a bit fuzzy in that implementation though, as the renewer can still resurrect expired tokens if the current time < the "max date" in the token. In theory, it limits the window during which token disclosure allows impersonating the user. If the token expires in 24 hours without renewal, and the MR job completes in less than that time, then a disclosure of the token 25 hrs after issue, when the token has expired, and the JT has not needed to renew it, will not allow the token to be re-used to impersonate the user. However, this doesn't really close the loop if you can somehow trick the JT (the designated "renewer") into resurrecting the expired token for you. Also, we can't use the built in JT renewal as it only works for Hadoop DelegationTokens, so something else would have to handle it for the duration of a job execution. And it's not clear to me that it's a meaningful enhancement in security. So I've ignored the expiration/max date distinction and just made it expire date. On failure-over, the master would read in the current master keys from ZK and repopulate the valid tokens in memory as they're used, validating that they use an existing master key and haven't expired, err.. "maxed out", yet. Whats this? We need name for cluster instance? I suppose we can't use master ip plus port because could change with time. The zk ensemble string plus the zk rootdir? Yeah, this part is a bit tricky. I hadn't thought of cluster ensemble subsets. I was going to ping JD on if replication had anything to use for similar purposes – uniquely identify clusters to prevent replication loops, say. Talking with Andy, he suggested generating a UUID on initial FS setup and adding it to hbase.version. From there master could pop it up in ZK on startup? Maybe I should open a separate JIRA for discussing that bit? Thanks for the comments! I suppose I should clarify some of these bits on the wiki page.
          Hide
          Todd Lipcon added a comment -

          Looks pretty good to me. One question - how are we going to get the build to work? For most of the security stuff we've been able to use reflection, but since we need to add annotations to our RPC protocols, it might get kind of messy. Do we have to have a long running branch?

          Show
          Todd Lipcon added a comment - Looks pretty good to me. One question - how are we going to get the build to work? For most of the security stuff we've been able to use reflection, but since we need to add annotations to our RPC protocols, it might get kind of messy. Do we have to have a long running branch?
          Hide
          Gary Helmling added a comment -

          Ah, good catch on the protocol annotations! Yes, those definitely leak out the secure Hadoop classes. The token auth stuff will be handling in a secure RPC engine enabled via the pluggable RPC engines configuration. So the rest of the implementation will be separated out from the standard HBaseClient/Server. We can duplicate the annotation interfaces – I'll look into the implications of this for any lower level class dependencies.

          For build, I'm thinking we have a separate optional build step (I guess practically this means a separate maven module?) with an isolated dependency on secure Hadoop. The module would separate out source code for the secure RPC engine and AccessController coprocessor and generate a separate jar for these two security products. (Both are already configured in via class names in hbase-site.xml and use established interfaces to prevent any direct dependencies from core HBase code). It sounds workable to me, but I'm too much of a maven noob to anticipate how difficult it'll be.

          I'd love to start working out the build details with people at the hackathon next week. But any thoughts before then are definitely welcome.

          Show
          Gary Helmling added a comment - Ah, good catch on the protocol annotations! Yes, those definitely leak out the secure Hadoop classes. The token auth stuff will be handling in a secure RPC engine enabled via the pluggable RPC engines configuration. So the rest of the implementation will be separated out from the standard HBaseClient/Server. We can duplicate the annotation interfaces – I'll look into the implications of this for any lower level class dependencies. For build, I'm thinking we have a separate optional build step (I guess practically this means a separate maven module?) with an isolated dependency on secure Hadoop. The module would separate out source code for the secure RPC engine and AccessController coprocessor and generate a separate jar for these two security products. (Both are already configured in via class names in hbase-site.xml and use established interfaces to prevent any direct dependencies from core HBase code). It sounds workable to me, but I'm too much of a maven noob to anticipate how difficult it'll be. I'd love to start working out the build details with people at the hackathon next week. But any thoughts before then are definitely welcome.
          Hide
          Gary Helmling added a comment -

          Thinking through the TokenInfo annotation issue a bit more, we could copy over the TokenInfo annotation and modify it to reference the Token "kind" from the token identifier for matching, then register the TokenSelector using the kind.

          I'm wondering if we even need to go to this trouble – we could have the implementation just register the selector class for each protocol class it handles. But I suppose it's nice to have the protocols be a bit self-documenting.

          Show
          Gary Helmling added a comment - Thinking through the TokenInfo annotation issue a bit more, we could copy over the TokenInfo annotation and modify it to reference the Token "kind" from the token identifier for matching, then register the TokenSelector using the kind. I'm wondering if we even need to go to this trouble – we could have the implementation just register the selector class for each protocol class it handles. But I suppose it's nice to have the protocols be a bit self-documenting.
          Hide
          Gary Helmling added a comment -

          Attaching a diff from the security branch (https://github.com/trendmicro/hbase/tree/security) implementing authentication tokens. This builds on top of the secure RPC layer already in place (org.apache.hadoop.hbase.ipc.SecureClient, SecureRpcEngine, SecureServer), so sorry if changes to those are out of context. If you'd like to see a fuller patch with those included, just ask.

          You can ignore the changes to HBaseClient and HBaseServer. Those are mostly fixes for subclassing of those by the secure RPC classes.

          Most relevant will be the classes in org.apache.hadoop.hbase.security.token, annotation changes for RPC interfaces, and hooks in org.apache.hadoop.hbase.security.User and TableMapReduceUtil to obtain tokens.

          Show
          Gary Helmling added a comment - Attaching a diff from the security branch ( https://github.com/trendmicro/hbase/tree/security ) implementing authentication tokens. This builds on top of the secure RPC layer already in place (org.apache.hadoop.hbase.ipc.SecureClient, SecureRpcEngine, SecureServer), so sorry if changes to those are out of context. If you'd like to see a fuller patch with those included, just ask. You can ignore the changes to HBaseClient and HBaseServer. Those are mostly fixes for subclassing of those by the secure RPC classes. Most relevant will be the classes in org.apache.hadoop.hbase.security.token, annotation changes for RPC interfaces, and hooks in org.apache.hadoop.hbase.security.User and TableMapReduceUtil to obtain tokens.
          Hide
          stack added a comment -

          Moving out of 0.92.0. Pull it back in if you think different.

          Show
          stack added a comment - Moving out of 0.92.0. Pull it back in if you think different.
          Hide
          Gary Helmling added a comment -

          Since tokens are one of the possible authentication methods during RPC, this was included as part of the change for HBASE-2742.

          Show
          Gary Helmling added a comment - Since tokens are one of the possible authentication methods during RPC, this was included as part of the change for HBASE-2742 .

            People

            • Assignee:
              Gary Helmling
              Reporter:
              Gary Helmling
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development