HBase
  1. HBase
  2. HBASE-2742

Provide strong authentication with a secure RPC engine

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.92.0
    • Component/s: IPC/RPC
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The HBase RPC code (org.apache.hadoop.hbase.ipc.*) was originally forked off of Hadoop RPC classes, with some performance tweaks added. Those optimizations have come at a cost in keeping up with Hadoop RPC changes however, both bug fixes and improvements/new features.

      In particular, this impacts how we implement security features in HBase (see HBASE-1697 and HBASE-2016). The secure Hadoop implementation (HADOOP-4487) relies heavily on RPC changes to support client authentication via kerberos and securing and mutual authentication of client/server connections via SASL. Making use of the built-in Hadoop RPC classes will gain us these pieces for free in a secure HBase.

      So, I'm proposing that we drop the HBase forked version of RPC and convert to direct use of Hadoop RPC, while working to contribute important fixes back upstream to Hadoop core. Based on a review of the HBase RPC changes, the key divergences seem to be:

      HBaseClient:

      • added use of TCP keepalive (HBASE-1754)
      • made connection retries and sleep configurable (HBASE-1815)
      • prevent NPE if socket == null due to creation failure (HBASE-2443)

      HBaseRPC:

      • mapping of method names <-> codes (removed in HBASE-2219)

      HBaseServer:

      HbaseObjectWritable:

      • allows List<> serialization
      • includes it's own class <-> code mapping (HBASE-328)

      Proposed process is:

      1. open issues with patches on Hadoop core for important fixes/adjustments from HBase RPC (HBASE-1198, HBASE-1815, HBASE-1754, HBASE-2443, plus a pluggable ObjectWritable implementation in RPC.Invocation to allow use of HbaseObjectWritable).

      2. ship a Hadoop version with RPC patches applied – ideally we should avoid another copy-n-paste code fork, subject to ability to isolate changes from impacting Hadoop internal RPC wire formats

      3. if all Hadoop core patches are applied we can drop back to a plain vanilla Hadoop version

      I realize there are many different opinions on how to proceed with HBase RPC, so I'm hoping this issue will kick off a discussion on what the best approach might be. My own motivation is maximizing re-use of the authentication and connection security work that's already gone into Hadoop core. I'll put together a set of patches around #1 and #2, but obviously we need some consensus around this to move forward. If I'm missing other differences between HBase and Hadoop RPC, please list as well. Discuss!

      1. HBASE-2742_addendum.patch
        2 kB
        Gary Helmling
      2. HBASE-2742_10.patch
        255 kB
        Gary Helmling

        Issue Links

          Activity

          Hide
          Gary Helmling added a comment -

          This is a key enabler for providing client authentication and secure connections to HBase

          Show
          Gary Helmling added a comment - This is a key enabler for providing client authentication and secure connections to HBase
          Hide
          Todd Lipcon added a comment -

          In sentiment I agree. In practicality, the idea that we could get the necessary changes into Hadoop in a reasonable timeline seems unlikely. Also, we'd have to wait until Hadoop 0.22 to get these in, which won't be out til 2011, so in the meantime we'd still be maintaining copy-paste. Lastly, Hadoop is hopefully moving to Avro down the line, and we should consider that too.

          I've only looked a bit at the secure RPC code in hadoop trunk, but IIRC the RPC part of security is reasonably straightforward (most of the work was in adding delegation tokens, etc, which are hadoop-layer, not IPC layer). How hard would it be to put similar security into, say, Avro, which we should be thinking about anyway?

          Show
          Todd Lipcon added a comment - In sentiment I agree. In practicality, the idea that we could get the necessary changes into Hadoop in a reasonable timeline seems unlikely. Also, we'd have to wait until Hadoop 0.22 to get these in, which won't be out til 2011, so in the meantime we'd still be maintaining copy-paste. Lastly, Hadoop is hopefully moving to Avro down the line, and we should consider that too. I've only looked a bit at the secure RPC code in hadoop trunk, but IIRC the RPC part of security is reasonably straightforward (most of the work was in adding delegation tokens, etc, which are hadoop-layer, not IPC layer). How hard would it be to put similar security into, say, Avro, which we should be thinking about anyway?
          Hide
          stack added a comment -

          Adding link to HBASE-2425. If we do this issue, then it subsumes need to do hbase-2425.

          Gary, since you've been looking, are any of hbase fixes already up in hadoop rpc? Or, is it that the ones you call out are in hbase rpc but not in hadoop rpc?

          I agree w/ Todd on the not till 2011.... (none of the pieces we'd need to override are overrideable IIRC – hence the copy local)

          On AVRO, what you reckon Gary? Seems like you have the lads (T+J) to get any needs committed on AVRO front. How hard would it be to do? It'd be good for the hbase client to hbase servers. What happens then moving the credentials over to hbase<->DN+NN connections?

          For me, AVRO ain't even a contender unless it has a real transport. HTTP doesn't cut it if only because it hurts my teeth just thinking about fat http headers wrapping and perhaps even replicating info in avro headers (The latter is foggy rememberance from emails long ago – world has probably changed since but HTTP for transport seems horribly profligate at least in hbase case).

          Any timelines for when the double-turbo-netty (or whatever its going to be done on) avro transport is going to make a showing Todd/Jeff?

          Show
          stack added a comment - Adding link to HBASE-2425 . If we do this issue, then it subsumes need to do hbase-2425. Gary, since you've been looking, are any of hbase fixes already up in hadoop rpc? Or, is it that the ones you call out are in hbase rpc but not in hadoop rpc? I agree w/ Todd on the not till 2011.... (none of the pieces we'd need to override are overrideable IIRC – hence the copy local) On AVRO, what you reckon Gary? Seems like you have the lads (T+J) to get any needs committed on AVRO front. How hard would it be to do? It'd be good for the hbase client to hbase servers. What happens then moving the credentials over to hbase<->DN+NN connections? For me, AVRO ain't even a contender unless it has a real transport. HTTP doesn't cut it if only because it hurts my teeth just thinking about fat http headers wrapping and perhaps even replicating info in avro headers (The latter is foggy rememberance from emails long ago – world has probably changed since but HTTP for transport seems horribly profligate at least in hbase case). Any timelines for when the double-turbo-netty (or whatever its going to be done on) avro transport is going to make a showing Todd/Jeff?
          Hide
          Gary Helmling added a comment -

          On the issues listed:
          HBASE-2443 looks to be already present in Hadoop trunk. HBASE-1198 (OOME) is really about the RS not getting notified of an OOME happening in the RPC layer. Seems like bad practice to me to swallow the exception and an important fix for us, but it may not be an issue or be handled differently by the Hadoop procs? The other 2 are not applied, but maybe not high priority for Hadoop core:

          • HBASE-1815: configurable max retries and sleep for connection timeouts
          • HBASE-1754: allow use of TCP keep alive

          @Todd
          Understood about the long timeline to get any of these changes committed. Based on that and discussion on IRC, maybe a reasonable alternative would be to just update our copy-paste fork with the security enabled code. I can still file the issues against core, as they seem generally useful, and in the future, either core integrates them or we move to our own alternate RPC (avro), either way potentially allowing us to drop the fork later on.

          @Stack
          I'm generally pro alternate rpc for HBase internal purposes. But I'm hesitant to make the security implementation (already a big project) depend on replacing internal rpc with avro (another big project). I just envision too many complications with one bleeding into the other. I also think that piggy-backing off the client user authentication and SASL setup code already in the secure Hadoop RPC will get us a good way started on secure HBase.

          In any case, for a full security implementation, we will need hooks into the new Avro connector, Thrift, Stargate/REST, etc. in order to secure all endpoints. But it seems wiser to leave those as separate tasks and off the critical path. And work done there could then be used to enable an Avro-based internal RPC.

          For the HBase authentication implementation, the plan for the first stage is to have HBase authenticate clients and perform access control, but have HBase interact with HDFS as a single server principal – so all of /hbase is owned by an "hbase" user, say. There are some issues to work out there with bulk loading mechanisms, but it's much simpler than passing ownership/credentials all the way down to HDFS. We've discussed a future stage mapping users to file ownership for isolation all the way down, but it's not clear that we will wind up going there or that it even provides better security to do so. So passing client credentials through to NN may not be necessary at all.

          Based on all of the above, I guess I'm leaning towards updating our copy-paste fork as part of the security implementation, at least as a first step. The original proposal to ship a patched Hadoop seems a non-starter if the changes won't be quickly integrated and we want to support multiple versions.

          Show
          Gary Helmling added a comment - On the issues listed: HBASE-2443 looks to be already present in Hadoop trunk. HBASE-1198 (OOME) is really about the RS not getting notified of an OOME happening in the RPC layer. Seems like bad practice to me to swallow the exception and an important fix for us, but it may not be an issue or be handled differently by the Hadoop procs? The other 2 are not applied, but maybe not high priority for Hadoop core: HBASE-1815 : configurable max retries and sleep for connection timeouts HBASE-1754 : allow use of TCP keep alive @Todd Understood about the long timeline to get any of these changes committed. Based on that and discussion on IRC, maybe a reasonable alternative would be to just update our copy-paste fork with the security enabled code. I can still file the issues against core, as they seem generally useful, and in the future, either core integrates them or we move to our own alternate RPC (avro), either way potentially allowing us to drop the fork later on. @Stack I'm generally pro alternate rpc for HBase internal purposes. But I'm hesitant to make the security implementation (already a big project) depend on replacing internal rpc with avro (another big project). I just envision too many complications with one bleeding into the other. I also think that piggy-backing off the client user authentication and SASL setup code already in the secure Hadoop RPC will get us a good way started on secure HBase. In any case, for a full security implementation, we will need hooks into the new Avro connector, Thrift, Stargate/REST, etc. in order to secure all endpoints. But it seems wiser to leave those as separate tasks and off the critical path. And work done there could then be used to enable an Avro-based internal RPC. For the HBase authentication implementation, the plan for the first stage is to have HBase authenticate clients and perform access control, but have HBase interact with HDFS as a single server principal – so all of /hbase is owned by an "hbase" user, say. There are some issues to work out there with bulk loading mechanisms, but it's much simpler than passing ownership/credentials all the way down to HDFS. We've discussed a future stage mapping users to file ownership for isolation all the way down, but it's not clear that we will wind up going there or that it even provides better security to do so. So passing client credentials through to NN may not be necessary at all. Based on all of the above, I guess I'm leaning towards updating our copy-paste fork as part of the security implementation, at least as a first step. The original proposal to ship a patched Hadoop seems a non-starter if the changes won't be quickly integrated and we want to support multiple versions.
          Hide
          Todd Lipcon added a comment -

          +1 for updating our copypaste with security.
          Regarding the OOME swallowing, it's done so that sending a malformed packet to a server doesn't crash it. It's way too easy to craft a packet (on purpose or by accident) that has a 100GB "length" header in it, and cause OOME.

          Show
          Todd Lipcon added a comment - +1 for updating our copypaste with security. Regarding the OOME swallowing, it's done so that sending a malformed packet to a server doesn't crash it. It's way too easy to craft a packet (on purpose or by accident) that has a 100GB "length" header in it, and cause OOME.
          Hide
          Gary Helmling added a comment -

          Update title to reflect discussed approach.

          Show
          Gary Helmling added a comment - Update title to reflect discussed approach.
          Hide
          Gary Helmling added a comment -

          Token-based DIGEST-MD5 is just one of the authentication methods supported in secure RPC, so it's subsumed by this issue.

          Show
          Gary Helmling added a comment - Token-based DIGEST-MD5 is just one of the authentication methods supported in secure RPC, so it's subsumed by this issue.
          Hide
          Gary Helmling added a comment -

          Changing title for clarity.

          Show
          Gary Helmling added a comment - Changing title for clarity.
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/1991/
          -----------------------------------------------------------

          Review request for hbase.

          Summary
          -------

          This patch creates a new secure RPC engine for HBase, which provides Kerberos based authentication of clients, and a token-based authentication mechanism for mapreduce jobs. Primary components of the patch are:

          • a new maven profile for secure Hadoop/HBase: hadoop-0.20S
          • Secure Hadoop dependent classes are separated under a pseudo-module in the security/ directory. These source and test directories are only including if building the secure Hadoop profile
          • Currently the security classes get packaged with the regular HBase build artifacts. We need a way to at least override project.version, so we can append something like a "-security" suffix indicating the additional security components.
          • The pseudo-module here is really a half-step forward. It enables the security code to be optionally included in the build for now, and sets up the structure for a security module. But we still will want to pursue full modularization (see HBASE-4336), which will allow packing the security code in a separate build artifact.
          • a new RPC engine providing kerberos and token-based authentication: org.apache.hadoop.hbase.ipc.SecureRpcEngine
          • implementation under security/src/main/java/org/apache/hadoop/hbase/ipc/
          • The implementation classes extend the existing HBaseClient and HBaseServer to share as much of the RPC code as possible. The main override is of the connection classes to allow control over the SASL negotiation of secure connections
          • existing RPC changes
          • The existing HBaseClient and HBaseServer have been modified to make subclassing possible
          • All references to Hadoop UserGroupInformation have been replaced with org.apache.hadoop.hbase.security.User to insulate from future dependencies on specific Hadoop versions
          • a coprocessor endpoint for obtaining new authentication tokens: TokenProvider, and supporting classes for token generation and synchronization (incorporating HBASE-3615)
          • implementation is under security/src/main/java/org/apache/hadoop/hbase/security/token/
          • Secret keys for token generation and verification are synchronized throughout the cluster in zookeeper, under /hbase/tokenauth/keys

          To enable secure RPC, add the following configuration to hbase-site.xml:

          <property>
          <name>hadoop.security.authorization</name>
          <value>true</value>
          </property>
          <property>
          <name>hadoop.security.authentication</name>
          <value>kerberos</value>
          </property>
          <property>
          <name>hbase.rpc.engine</name>
          <value>org.apache.hadoop.hbase.ipc.SecureRpcEngine</value>
          </property>
          <property>
          <name>hbase.coprocessor.region.classes</name>
          <value>org.apache.hadoop.hbase.security.token.TokenProvider</value>
          </property>

          In addition, the master and regionserver processes must be configured for kerberos authentication using the properties:

          • hbase.(master|regionserver).keytab.file
          • hbase.(master|regionserver).kerberos.principal
          • hbase.(master|regionserver).kerberos.https.principal

          This addresses bug HBASE-2742.
          https://issues.apache.org/jira/browse/HBASE-2742

          Diffs


          conf/hbase-policy.xml PRE-CREATION
          pom.xml 241973c
          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureConnectionHeader.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/ipc/Status.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationKey.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationProtocol.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenIdentifier.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSelector.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/token/TokenProvider.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/token/ZKLeaderManager.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/token/ZKSecretWatcher.java PRE-CREATION
          security/src/test/java/org/apache/hadoop/hbase/security/token/TestTokenAuthentication.java PRE-CREATION
          security/src/test/java/org/apache/hadoop/hbase/security/token/TestZKSecretWatcher.java PRE-CREATION
          src/main/java/org/apache/hadoop/hbase/HServerAddress.java 94977e0
          src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 5afaedf
          src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java 904c107
          src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 4621109
          src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java 18310d6
          src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 337da78
          src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 36b0560
          src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java a069400
          src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java 965102c
          src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 3679c02
          src/main/java/org/apache/hadoop/hbase/ipc/RequestContext.java PRE-CREATION
          src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java 1b5629a
          src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java 60a9248
          src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java db07ed1
          src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java ad88b76
          src/main/java/org/apache/hadoop/hbase/master/HMaster.java 06bf814
          src/main/java/org/apache/hadoop/hbase/security/KerberosInfo.java PRE-CREATION
          src/main/java/org/apache/hadoop/hbase/security/TokenInfo.java PRE-CREATION
          src/main/java/org/apache/hadoop/hbase/security/User.java d90f2c7
          src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java 699a5f5
          src/main/resources/hbase-default.xml 2c8f44b
          src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java 7194c02
          src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java 3982eff

          Diff: https://reviews.apache.org/r/1991/diff

          Testing
          -------

          A full test suite run showed failures in TestMasterFailover (timeout) and TestAvroServer. I'll look into those.

          Cluster testing (1 master + 3 slaves in ec2) with Kerberos authentication and map reduce token authentication, using YCSB, RowCounter, PerformanceEvaluation.

          Thanks,

          Gary

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1991/ ----------------------------------------------------------- Review request for hbase. Summary ------- This patch creates a new secure RPC engine for HBase, which provides Kerberos based authentication of clients, and a token-based authentication mechanism for mapreduce jobs. Primary components of the patch are: a new maven profile for secure Hadoop/HBase: hadoop-0.20S Secure Hadoop dependent classes are separated under a pseudo-module in the security/ directory. These source and test directories are only including if building the secure Hadoop profile Currently the security classes get packaged with the regular HBase build artifacts. We need a way to at least override project.version, so we can append something like a "-security" suffix indicating the additional security components. The pseudo-module here is really a half-step forward. It enables the security code to be optionally included in the build for now, and sets up the structure for a security module. But we still will want to pursue full modularization (see HBASE-4336 ), which will allow packing the security code in a separate build artifact. a new RPC engine providing kerberos and token-based authentication: org.apache.hadoop.hbase.ipc.SecureRpcEngine implementation under security/src/main/java/org/apache/hadoop/hbase/ipc/ The implementation classes extend the existing HBaseClient and HBaseServer to share as much of the RPC code as possible. The main override is of the connection classes to allow control over the SASL negotiation of secure connections existing RPC changes The existing HBaseClient and HBaseServer have been modified to make subclassing possible All references to Hadoop UserGroupInformation have been replaced with org.apache.hadoop.hbase.security.User to insulate from future dependencies on specific Hadoop versions a coprocessor endpoint for obtaining new authentication tokens: TokenProvider, and supporting classes for token generation and synchronization (incorporating HBASE-3615 ) implementation is under security/src/main/java/org/apache/hadoop/hbase/security/token/ Secret keys for token generation and verification are synchronized throughout the cluster in zookeeper, under /hbase/tokenauth/keys To enable secure RPC, add the following configuration to hbase-site.xml: <property> <name>hadoop.security.authorization</name> <value>true</value> </property> <property> <name>hadoop.security.authentication</name> <value>kerberos</value> </property> <property> <name>hbase.rpc.engine</name> <value>org.apache.hadoop.hbase.ipc.SecureRpcEngine</value> </property> <property> <name>hbase.coprocessor.region.classes</name> <value>org.apache.hadoop.hbase.security.token.TokenProvider</value> </property> In addition, the master and regionserver processes must be configured for kerberos authentication using the properties: hbase.(master|regionserver).keytab.file hbase.(master|regionserver).kerberos.principal hbase.(master|regionserver).kerberos.https.principal This addresses bug HBASE-2742 . https://issues.apache.org/jira/browse/HBASE-2742 Diffs conf/hbase-policy.xml PRE-CREATION pom.xml 241973c security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/SecureConnectionHeader.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/Status.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationKey.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationProtocol.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenIdentifier.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSelector.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/TokenProvider.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/ZKLeaderManager.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/ZKSecretWatcher.java PRE-CREATION security/src/test/java/org/apache/hadoop/hbase/security/token/TestTokenAuthentication.java PRE-CREATION security/src/test/java/org/apache/hadoop/hbase/security/token/TestZKSecretWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/HServerAddress.java 94977e0 src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 5afaedf src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java 904c107 src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 4621109 src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java 18310d6 src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 337da78 src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 36b0560 src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java a069400 src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java 965102c src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 3679c02 src/main/java/org/apache/hadoop/hbase/ipc/RequestContext.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java 1b5629a src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java 60a9248 src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java db07ed1 src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java ad88b76 src/main/java/org/apache/hadoop/hbase/master/HMaster.java 06bf814 src/main/java/org/apache/hadoop/hbase/security/KerberosInfo.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/security/TokenInfo.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/security/User.java d90f2c7 src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java 699a5f5 src/main/resources/hbase-default.xml 2c8f44b src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java 7194c02 src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java 3982eff Diff: https://reviews.apache.org/r/1991/diff Testing ------- A full test suite run showed failures in TestMasterFailover (timeout) and TestAvroServer. I'll look into those. Cluster testing (1 master + 3 slaves in ec2) with Kerberos authentication and map reduce token authentication, using YCSB, RowCounter, PerformanceEvaluation. Thanks, Gary
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/1991/#review2013
          -----------------------------------------------------------

          This is what we are running in production, incorporating additional bug fixes currently in staging, and ported to trunk. Looks ok but I found a few details worth looking at.

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java
          <https://reviews.apache.org/r/1991/#comment4539>

          Didn't we move the serialization of the call and params out of the synchronized block? Or, we should.

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java
          <https://reviews.apache.org/r/1991/#comment4540>

          Let's fix the spelling error here finally

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java
          <https://reviews.apache.org/r/1991/#comment4541>

          Logs tend to fill up with this, make it DEBUG?

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java
          <https://reviews.apache.org/r/1991/#comment4542>

          Likewise

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java
          <https://reviews.apache.org/r/1991/#comment4543>

          Likewise, but maybe this should be TRACE

          security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java
          <https://reviews.apache.org/r/1991/#comment4544>

          Should be "hbase.rpc.protection"

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java
          <https://reviews.apache.org/r/1991/#comment4545>

          Writes are synchronized but not this read

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java
          <https://reviews.apache.org/r/1991/#comment4546>

          Could conceivably get a CME here too right?

          src/main/java/org/apache/hadoop/hbase/HServerAddress.java
          <https://reviews.apache.org/r/1991/#comment4547>

          Could use a comment here explaining why it is important.

          • Andrew

          On 2011-09-21 05:54:32, Gary Helmling wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/1991/

          -----------------------------------------------------------

          (Updated 2011-09-21 05:54:32)

          Review request for hbase.

          Summary

          -------

          This patch creates a new secure RPC engine for HBase, which provides Kerberos based authentication of clients, and a token-based authentication mechanism for mapreduce jobs. Primary components of the patch are:

          - a new maven profile for secure Hadoop/HBase: hadoop-0.20S

          - Secure Hadoop dependent classes are separated under a pseudo-module in the security/ directory. These source and test directories are only including if building the secure Hadoop profile

          - Currently the security classes get packaged with the regular HBase build artifacts. We need a way to at least override project.version, so we can append something like a "-security" suffix indicating the additional security components.

          - The pseudo-module here is really a half-step forward. It enables the security code to be optionally included in the build for now, and sets up the structure for a security module. But we still will want to pursue full modularization (see HBASE-4336), which will allow packing the security code in a separate build artifact.

          - a new RPC engine providing kerberos and token-based authentication: org.apache.hadoop.hbase.ipc.SecureRpcEngine

          - implementation under security/src/main/java/org/apache/hadoop/hbase/ipc/

          - The implementation classes extend the existing HBaseClient and HBaseServer to share as much of the RPC code as possible. The main override is of the connection classes to allow control over the SASL negotiation of secure connections

          - existing RPC changes

          - The existing HBaseClient and HBaseServer have been modified to make subclassing possible

          - All references to Hadoop UserGroupInformation have been replaced with org.apache.hadoop.hbase.security.User to insulate from future dependencies on specific Hadoop versions

          - a coprocessor endpoint for obtaining new authentication tokens: TokenProvider, and supporting classes for token generation and synchronization (incorporating HBASE-3615)

          - implementation is under security/src/main/java/org/apache/hadoop/hbase/security/token/

          - Secret keys for token generation and verification are synchronized throughout the cluster in zookeeper, under /hbase/tokenauth/keys

          To enable secure RPC, add the following configuration to hbase-site.xml:

          <property>

          <name>hadoop.security.authorization</name>

          <value>true</value>

          </property>

          <property>

          <name>hadoop.security.authentication</name>

          <value>kerberos</value>

          </property>

          <property>

          <name>hbase.rpc.engine</name>

          <value>org.apache.hadoop.hbase.ipc.SecureRpcEngine</value>

          </property>

          <property>

          <name>hbase.coprocessor.region.classes</name>

          <value>org.apache.hadoop.hbase.security.token.TokenProvider</value>

          </property>

          In addition, the master and regionserver processes must be configured for kerberos authentication using the properties:

          * hbase.(master|regionserver).keytab.file

          * hbase.(master|regionserver).kerberos.principal

          * hbase.(master|regionserver).kerberos.https.principal

          This addresses bug HBASE-2742.

          https://issues.apache.org/jira/browse/HBASE-2742

          Diffs

          -----

          conf/hbase-policy.xml PRE-CREATION

          pom.xml 241973c

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureConnectionHeader.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/ipc/Status.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationKey.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationProtocol.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenIdentifier.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSelector.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/TokenProvider.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/ZKLeaderManager.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/ZKSecretWatcher.java PRE-CREATION

          security/src/test/java/org/apache/hadoop/hbase/security/token/TestTokenAuthentication.java PRE-CREATION

          security/src/test/java/org/apache/hadoop/hbase/security/token/TestZKSecretWatcher.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/HServerAddress.java 94977e0

          src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 5afaedf

          src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java 904c107

          src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 4621109

          src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java 18310d6

          src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 337da78

          src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 36b0560

          src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java a069400

          src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java 965102c

          src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 3679c02

          src/main/java/org/apache/hadoop/hbase/ipc/RequestContext.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java 1b5629a

          src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java 60a9248

          src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java db07ed1

          src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java ad88b76

          src/main/java/org/apache/hadoop/hbase/master/HMaster.java 06bf814

          src/main/java/org/apache/hadoop/hbase/security/KerberosInfo.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/security/TokenInfo.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/security/User.java d90f2c7

          src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java 699a5f5

          src/main/resources/hbase-default.xml 2c8f44b

          src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java 7194c02

          src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java 3982eff

          Diff: https://reviews.apache.org/r/1991/diff

          Testing

          -------

          A full test suite run showed failures in TestMasterFailover (timeout) and TestAvroServer. I'll look into those.

          Cluster testing (1 master + 3 slaves in ec2) with Kerberos authentication and map reduce token authentication, using YCSB, RowCounter, PerformanceEvaluation.

          Thanks,

          Gary

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1991/#review2013 ----------------------------------------------------------- This is what we are running in production, incorporating additional bug fixes currently in staging, and ported to trunk. Looks ok but I found a few details worth looking at. security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java < https://reviews.apache.org/r/1991/#comment4539 > Didn't we move the serialization of the call and params out of the synchronized block? Or, we should. security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java < https://reviews.apache.org/r/1991/#comment4540 > Let's fix the spelling error here finally security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java < https://reviews.apache.org/r/1991/#comment4541 > Logs tend to fill up with this, make it DEBUG? security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java < https://reviews.apache.org/r/1991/#comment4542 > Likewise security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java < https://reviews.apache.org/r/1991/#comment4543 > Likewise, but maybe this should be TRACE security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java < https://reviews.apache.org/r/1991/#comment4544 > Should be "hbase.rpc.protection" security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java < https://reviews.apache.org/r/1991/#comment4545 > Writes are synchronized but not this read security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java < https://reviews.apache.org/r/1991/#comment4546 > Could conceivably get a CME here too right? src/main/java/org/apache/hadoop/hbase/HServerAddress.java < https://reviews.apache.org/r/1991/#comment4547 > Could use a comment here explaining why it is important. Andrew On 2011-09-21 05:54:32, Gary Helmling wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1991/ ----------------------------------------------------------- (Updated 2011-09-21 05:54:32) Review request for hbase. Summary ------- This patch creates a new secure RPC engine for HBase, which provides Kerberos based authentication of clients, and a token-based authentication mechanism for mapreduce jobs. Primary components of the patch are: - a new maven profile for secure Hadoop/HBase: hadoop-0.20S - Secure Hadoop dependent classes are separated under a pseudo-module in the security/ directory. These source and test directories are only including if building the secure Hadoop profile - Currently the security classes get packaged with the regular HBase build artifacts. We need a way to at least override project.version, so we can append something like a "-security" suffix indicating the additional security components. - The pseudo-module here is really a half-step forward. It enables the security code to be optionally included in the build for now, and sets up the structure for a security module. But we still will want to pursue full modularization (see HBASE-4336 ), which will allow packing the security code in a separate build artifact. - a new RPC engine providing kerberos and token-based authentication: org.apache.hadoop.hbase.ipc.SecureRpcEngine - implementation under security/src/main/java/org/apache/hadoop/hbase/ipc/ - The implementation classes extend the existing HBaseClient and HBaseServer to share as much of the RPC code as possible. The main override is of the connection classes to allow control over the SASL negotiation of secure connections - existing RPC changes - The existing HBaseClient and HBaseServer have been modified to make subclassing possible - All references to Hadoop UserGroupInformation have been replaced with org.apache.hadoop.hbase.security.User to insulate from future dependencies on specific Hadoop versions - a coprocessor endpoint for obtaining new authentication tokens: TokenProvider, and supporting classes for token generation and synchronization (incorporating HBASE-3615 ) - implementation is under security/src/main/java/org/apache/hadoop/hbase/security/token/ - Secret keys for token generation and verification are synchronized throughout the cluster in zookeeper, under /hbase/tokenauth/keys To enable secure RPC, add the following configuration to hbase-site.xml: <property> <name>hadoop.security.authorization</name> <value>true</value> </property> <property> <name>hadoop.security.authentication</name> <value>kerberos</value> </property> <property> <name>hbase.rpc.engine</name> <value>org.apache.hadoop.hbase.ipc.SecureRpcEngine</value> </property> <property> <name>hbase.coprocessor.region.classes</name> <value>org.apache.hadoop.hbase.security.token.TokenProvider</value> </property> In addition, the master and regionserver processes must be configured for kerberos authentication using the properties: * hbase.(master|regionserver).keytab.file * hbase.(master|regionserver).kerberos.principal * hbase.(master|regionserver).kerberos.https.principal This addresses bug HBASE-2742 . https://issues.apache.org/jira/browse/HBASE-2742 Diffs ----- conf/hbase-policy.xml PRE-CREATION pom.xml 241973c security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/SecureConnectionHeader.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/Status.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationKey.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationProtocol.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenIdentifier.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSelector.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/TokenProvider.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/ZKLeaderManager.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/ZKSecretWatcher.java PRE-CREATION security/src/test/java/org/apache/hadoop/hbase/security/token/TestTokenAuthentication.java PRE-CREATION security/src/test/java/org/apache/hadoop/hbase/security/token/TestZKSecretWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/HServerAddress.java 94977e0 src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 5afaedf src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java 904c107 src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 4621109 src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java 18310d6 src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 337da78 src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 36b0560 src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java a069400 src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java 965102c src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 3679c02 src/main/java/org/apache/hadoop/hbase/ipc/RequestContext.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java 1b5629a src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java 60a9248 src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java db07ed1 src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java ad88b76 src/main/java/org/apache/hadoop/hbase/master/HMaster.java 06bf814 src/main/java/org/apache/hadoop/hbase/security/KerberosInfo.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/security/TokenInfo.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/security/User.java d90f2c7 src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java 699a5f5 src/main/resources/hbase-default.xml 2c8f44b src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java 7194c02 src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java 3982eff Diff: https://reviews.apache.org/r/1991/diff Testing ------- A full test suite run showed failures in TestMasterFailover (timeout) and TestAvroServer. I'll look into those. Cluster testing (1 master + 3 slaves in ec2) with Kerberos authentication and map reduce token authentication, using YCSB, RowCounter, PerformanceEvaluation. Thanks, Gary
          Hide
          Ted Yu added a comment -

          When I ran the test suite, I got:

          initializationError(org.apache.hadoop.hbase.security.token.TestTokenAuthentication)  Time elapsed: 0.005 sec  <<< ERROR!
          java.lang.NoClassDefFoundError: org/apache/hadoop/security/token/SecretManager
                  at java.lang.ClassLoader.defineClass1(Native Method)
                  at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
                  at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
                  at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
                  at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
                  at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
          
          Show
          Ted Yu added a comment - When I ran the test suite, I got: initializationError(org.apache.hadoop.hbase.security.token.TestTokenAuthentication) Time elapsed: 0.005 sec <<< ERROR! java.lang.NoClassDefFoundError: org/apache/hadoop/security/token/SecretManager at java.lang. ClassLoader .defineClass1(Native Method) at java.lang. ClassLoader .defineClassCond( ClassLoader .java:632) at java.lang. ClassLoader .defineClass( ClassLoader .java:616) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141) at java.net.URLClassLoader.defineClass(URLClassLoader.java:283) at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/1991/
          -----------------------------------------------------------

          (Updated 2011-10-21 00:14:15.788894)

          Review request for hbase.

          Changes
          -------

          Updated patch against current trunk.

          • Cleaned out some unnecessary code duplication in SecureClient vs. HBaseClient
          • Some javadoc cleanups and additions
          • Addressed Andy's review comments

          Running tests using SecureRpcEngine requires manually editing src/test/resources/hbase-site.xml and adding:

          <property>
          <name>hbase.rpc.engine</name>
          <value>org.apache.hadoop.hbase.ipc.SecureRpcEngine</value>
          </property>

          Then run the tests with "mvn test -P hadoop-0.20S"

          I'm hacking at the pom.xml to see if I can get it to override the hbase-site.xml that's used so this isn't necessary.

          Summary
          -------

          This patch creates a new secure RPC engine for HBase, which provides Kerberos based authentication of clients, and a token-based authentication mechanism for mapreduce jobs. Primary components of the patch are:

          • a new maven profile for secure Hadoop/HBase: hadoop-0.20S
          • Secure Hadoop dependent classes are separated under a pseudo-module in the security/ directory. These source and test directories are only including if building the secure Hadoop profile
          • Currently the security classes get packaged with the regular HBase build artifacts. We need a way to at least override project.version, so we can append something like a "-security" suffix indicating the additional security components.
          • The pseudo-module here is really a half-step forward. It enables the security code to be optionally included in the build for now, and sets up the structure for a security module. But we still will want to pursue full modularization (see HBASE-4336), which will allow packing the security code in a separate build artifact.
          • a new RPC engine providing kerberos and token-based authentication: org.apache.hadoop.hbase.ipc.SecureRpcEngine
          • implementation under security/src/main/java/org/apache/hadoop/hbase/ipc/
          • The implementation classes extend the existing HBaseClient and HBaseServer to share as much of the RPC code as possible. The main override is of the connection classes to allow control over the SASL negotiation of secure connections
          • existing RPC changes
          • The existing HBaseClient and HBaseServer have been modified to make subclassing possible
          • All references to Hadoop UserGroupInformation have been replaced with org.apache.hadoop.hbase.security.User to insulate from future dependencies on specific Hadoop versions
          • a coprocessor endpoint for obtaining new authentication tokens: TokenProvider, and supporting classes for token generation and synchronization (incorporating HBASE-3615)
          • implementation is under security/src/main/java/org/apache/hadoop/hbase/security/token/
          • Secret keys for token generation and verification are synchronized throughout the cluster in zookeeper, under /hbase/tokenauth/keys

          To enable secure RPC, add the following configuration to hbase-site.xml:

          <property>
          <name>hadoop.security.authorization</name>
          <value>true</value>
          </property>
          <property>
          <name>hadoop.security.authentication</name>
          <value>kerberos</value>
          </property>
          <property>
          <name>hbase.rpc.engine</name>
          <value>org.apache.hadoop.hbase.ipc.SecureRpcEngine</value>
          </property>
          <property>
          <name>hbase.coprocessor.region.classes</name>
          <value>org.apache.hadoop.hbase.security.token.TokenProvider</value>
          </property>

          In addition, the master and regionserver processes must be configured for kerberos authentication using the properties:

          • hbase.(master|regionserver).keytab.file
          • hbase.(master|regionserver).kerberos.principal
          • hbase.(master|regionserver).kerberos.https.principal

          This addresses bug HBASE-2742.
          https://issues.apache.org/jira/browse/HBASE-2742

          Diffs (updated)


          conf/hbase-policy.xml PRE-CREATION
          pom.xml bfe8103
          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureConnectionHeader.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/ipc/Status.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationKey.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationProtocol.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenIdentifier.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSelector.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/token/TokenProvider.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/token/ZKLeaderManager.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/token/ZKSecretWatcher.java PRE-CREATION
          security/src/test/java/org/apache/hadoop/hbase/security/token/TestTokenAuthentication.java PRE-CREATION
          security/src/test/java/org/apache/hadoop/hbase/security/token/TestZKSecretWatcher.java PRE-CREATION
          src/main/java/org/apache/hadoop/hbase/HServerAddress.java 94977e0
          src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java f7fac44
          src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java 904c107
          src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 1365411
          src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java 18310d6
          src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 337da78
          src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 1db05cb
          src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java 000f99c
          src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java 965102c
          src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 1f8b629
          src/main/java/org/apache/hadoop/hbase/ipc/RequestContext.java PRE-CREATION
          src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java 1b5629a
          src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java 60a9248
          src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java db07ed1
          src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java ad88b76
          src/main/java/org/apache/hadoop/hbase/master/HMaster.java 50b49a6
          src/main/java/org/apache/hadoop/hbase/security/KerberosInfo.java PRE-CREATION
          src/main/java/org/apache/hadoop/hbase/security/TokenInfo.java PRE-CREATION
          src/main/java/org/apache/hadoop/hbase/security/User.java 00bd06d
          src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java bb67e53
          src/main/resources/hbase-default.xml a35e7c7
          src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java 7194c02
          src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java 3982eff

          Diff: https://reviews.apache.org/r/1991/diff

          Testing
          -------

          A full test suite run showed failures in TestMasterFailover (timeout) and TestAvroServer. I'll look into those.

          Cluster testing (1 master + 3 slaves in ec2) with Kerberos authentication and map reduce token authentication, using YCSB, RowCounter, PerformanceEvaluation.

          Thanks,

          Gary

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1991/ ----------------------------------------------------------- (Updated 2011-10-21 00:14:15.788894) Review request for hbase. Changes ------- Updated patch against current trunk. Cleaned out some unnecessary code duplication in SecureClient vs. HBaseClient Some javadoc cleanups and additions Addressed Andy's review comments Running tests using SecureRpcEngine requires manually editing src/test/resources/hbase-site.xml and adding: <property> <name>hbase.rpc.engine</name> <value>org.apache.hadoop.hbase.ipc.SecureRpcEngine</value> </property> Then run the tests with "mvn test -P hadoop-0.20S" I'm hacking at the pom.xml to see if I can get it to override the hbase-site.xml that's used so this isn't necessary. Summary ------- This patch creates a new secure RPC engine for HBase, which provides Kerberos based authentication of clients, and a token-based authentication mechanism for mapreduce jobs. Primary components of the patch are: a new maven profile for secure Hadoop/HBase: hadoop-0.20S Secure Hadoop dependent classes are separated under a pseudo-module in the security/ directory. These source and test directories are only including if building the secure Hadoop profile Currently the security classes get packaged with the regular HBase build artifacts. We need a way to at least override project.version, so we can append something like a "-security" suffix indicating the additional security components. The pseudo-module here is really a half-step forward. It enables the security code to be optionally included in the build for now, and sets up the structure for a security module. But we still will want to pursue full modularization (see HBASE-4336 ), which will allow packing the security code in a separate build artifact. a new RPC engine providing kerberos and token-based authentication: org.apache.hadoop.hbase.ipc.SecureRpcEngine implementation under security/src/main/java/org/apache/hadoop/hbase/ipc/ The implementation classes extend the existing HBaseClient and HBaseServer to share as much of the RPC code as possible. The main override is of the connection classes to allow control over the SASL negotiation of secure connections existing RPC changes The existing HBaseClient and HBaseServer have been modified to make subclassing possible All references to Hadoop UserGroupInformation have been replaced with org.apache.hadoop.hbase.security.User to insulate from future dependencies on specific Hadoop versions a coprocessor endpoint for obtaining new authentication tokens: TokenProvider, and supporting classes for token generation and synchronization (incorporating HBASE-3615 ) implementation is under security/src/main/java/org/apache/hadoop/hbase/security/token/ Secret keys for token generation and verification are synchronized throughout the cluster in zookeeper, under /hbase/tokenauth/keys To enable secure RPC, add the following configuration to hbase-site.xml: <property> <name>hadoop.security.authorization</name> <value>true</value> </property> <property> <name>hadoop.security.authentication</name> <value>kerberos</value> </property> <property> <name>hbase.rpc.engine</name> <value>org.apache.hadoop.hbase.ipc.SecureRpcEngine</value> </property> <property> <name>hbase.coprocessor.region.classes</name> <value>org.apache.hadoop.hbase.security.token.TokenProvider</value> </property> In addition, the master and regionserver processes must be configured for kerberos authentication using the properties: hbase.(master|regionserver).keytab.file hbase.(master|regionserver).kerberos.principal hbase.(master|regionserver).kerberos.https.principal This addresses bug HBASE-2742 . https://issues.apache.org/jira/browse/HBASE-2742 Diffs (updated) conf/hbase-policy.xml PRE-CREATION pom.xml bfe8103 security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/SecureConnectionHeader.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/Status.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationKey.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationProtocol.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenIdentifier.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSelector.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/TokenProvider.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/ZKLeaderManager.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/ZKSecretWatcher.java PRE-CREATION security/src/test/java/org/apache/hadoop/hbase/security/token/TestTokenAuthentication.java PRE-CREATION security/src/test/java/org/apache/hadoop/hbase/security/token/TestZKSecretWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/HServerAddress.java 94977e0 src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java f7fac44 src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java 904c107 src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 1365411 src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java 18310d6 src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 337da78 src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 1db05cb src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java 000f99c src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java 965102c src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 1f8b629 src/main/java/org/apache/hadoop/hbase/ipc/RequestContext.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java 1b5629a src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java 60a9248 src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java db07ed1 src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java ad88b76 src/main/java/org/apache/hadoop/hbase/master/HMaster.java 50b49a6 src/main/java/org/apache/hadoop/hbase/security/KerberosInfo.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/security/TokenInfo.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/security/User.java 00bd06d src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java bb67e53 src/main/resources/hbase-default.xml a35e7c7 src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java 7194c02 src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java 3982eff Diff: https://reviews.apache.org/r/1991/diff Testing ------- A full test suite run showed failures in TestMasterFailover (timeout) and TestAvroServer. I'll look into those. Cluster testing (1 master + 3 slaves in ec2) with Kerberos authentication and map reduce token authentication, using YCSB, RowCounter, PerformanceEvaluation. Thanks, Gary
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/1991/
          -----------------------------------------------------------

          (Updated 2011-10-26 20:23:19.711393)

          Review request for hbase.

          Changes
          -------

          This updated patch just changes the POM security profile from the previous version:

          • Renames the profile from "hadoop-0.20S" to "security"
          • Since the default hadoop profile now uses 0.20.205.0 as the build version, the hadoop version is no longer needed in the security profile. The security profile can now be used in combination with any of the Hadoop profiles (though I've only tested with 0.20.205.0, aka profile "hadoop-0.20").
          • The security profile now overrides the hbase-site.xml used for testing with one that enables SecureRpcEngine. This should make testing with the RPC changes easier, though the pom.xml changes to accommodate it are a little clunky.

          To run tests using SecureRpcEngine, just do:

          mvn clean test -P security

          Summary
          -------

          This patch creates a new secure RPC engine for HBase, which provides Kerberos based authentication of clients, and a token-based authentication mechanism for mapreduce jobs. Primary components of the patch are:

          • a new maven profile for secure Hadoop/HBase: hadoop-0.20S
          • Secure Hadoop dependent classes are separated under a pseudo-module in the security/ directory. These source and test directories are only including if building the secure Hadoop profile
          • Currently the security classes get packaged with the regular HBase build artifacts. We need a way to at least override project.version, so we can append something like a "-security" suffix indicating the additional security components.
          • The pseudo-module here is really a half-step forward. It enables the security code to be optionally included in the build for now, and sets up the structure for a security module. But we still will want to pursue full modularization (see HBASE-4336), which will allow packing the security code in a separate build artifact.
          • a new RPC engine providing kerberos and token-based authentication: org.apache.hadoop.hbase.ipc.SecureRpcEngine
          • implementation under security/src/main/java/org/apache/hadoop/hbase/ipc/
          • The implementation classes extend the existing HBaseClient and HBaseServer to share as much of the RPC code as possible. The main override is of the connection classes to allow control over the SASL negotiation of secure connections
          • existing RPC changes
          • The existing HBaseClient and HBaseServer have been modified to make subclassing possible
          • All references to Hadoop UserGroupInformation have been replaced with org.apache.hadoop.hbase.security.User to insulate from future dependencies on specific Hadoop versions
          • a coprocessor endpoint for obtaining new authentication tokens: TokenProvider, and supporting classes for token generation and synchronization (incorporating HBASE-3615)
          • implementation is under security/src/main/java/org/apache/hadoop/hbase/security/token/
          • Secret keys for token generation and verification are synchronized throughout the cluster in zookeeper, under /hbase/tokenauth/keys

          To enable secure RPC, add the following configuration to hbase-site.xml:

          <property>
          <name>hadoop.security.authorization</name>
          <value>true</value>
          </property>
          <property>
          <name>hadoop.security.authentication</name>
          <value>kerberos</value>
          </property>
          <property>
          <name>hbase.rpc.engine</name>
          <value>org.apache.hadoop.hbase.ipc.SecureRpcEngine</value>
          </property>
          <property>
          <name>hbase.coprocessor.region.classes</name>
          <value>org.apache.hadoop.hbase.security.token.TokenProvider</value>
          </property>

          In addition, the master and regionserver processes must be configured for kerberos authentication using the properties:

          • hbase.(master|regionserver).keytab.file
          • hbase.(master|regionserver).kerberos.principal
          • hbase.(master|regionserver).kerberos.https.principal

          This addresses bug HBASE-2742.
          https://issues.apache.org/jira/browse/HBASE-2742

          Diffs (updated)


          conf/hbase-policy.xml PRE-CREATION
          pom.xml 9d42e2b
          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureConnectionHeader.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/ipc/Status.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationKey.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationProtocol.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenIdentifier.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSelector.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/token/TokenProvider.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/token/ZKLeaderManager.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/token/ZKSecretWatcher.java PRE-CREATION
          security/src/test/java/org/apache/hadoop/hbase/security/token/TestTokenAuthentication.java PRE-CREATION
          security/src/test/java/org/apache/hadoop/hbase/security/token/TestZKSecretWatcher.java PRE-CREATION
          security/src/test/resources/hbase-site.xml PRE-CREATION
          src/main/java/org/apache/hadoop/hbase/HServerAddress.java f28240c
          src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 06a2312
          src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java 904c107
          src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 1365411
          src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java 18310d6
          src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 337da78
          src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 4a8918a
          src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java 000f99c
          src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java 965102c
          src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 1f8b629
          src/main/java/org/apache/hadoop/hbase/ipc/RequestContext.java PRE-CREATION
          src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java 1b5629a
          src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java 60a9248
          src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java db07ed1
          src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java ad88b76
          src/main/java/org/apache/hadoop/hbase/master/HMaster.java 3f8f929
          src/main/java/org/apache/hadoop/hbase/security/KerberosInfo.java PRE-CREATION
          src/main/java/org/apache/hadoop/hbase/security/TokenInfo.java PRE-CREATION
          src/main/java/org/apache/hadoop/hbase/security/User.java 00bd06d
          src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java bb67e53
          src/main/resources/hbase-default.xml 3785533
          src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java 7194c02
          src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java 3982eff

          Diff: https://reviews.apache.org/r/1991/diff

          Testing
          -------

          A full test suite run showed failures in TestMasterFailover (timeout) and TestAvroServer. I'll look into those.

          Cluster testing (1 master + 3 slaves in ec2) with Kerberos authentication and map reduce token authentication, using YCSB, RowCounter, PerformanceEvaluation.

          Thanks,

          Gary

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1991/ ----------------------------------------------------------- (Updated 2011-10-26 20:23:19.711393) Review request for hbase. Changes ------- This updated patch just changes the POM security profile from the previous version: Renames the profile from "hadoop-0.20S" to "security" Since the default hadoop profile now uses 0.20.205.0 as the build version, the hadoop version is no longer needed in the security profile. The security profile can now be used in combination with any of the Hadoop profiles (though I've only tested with 0.20.205.0, aka profile "hadoop-0.20"). The security profile now overrides the hbase-site.xml used for testing with one that enables SecureRpcEngine. This should make testing with the RPC changes easier, though the pom.xml changes to accommodate it are a little clunky. To run tests using SecureRpcEngine, just do: mvn clean test -P security Summary ------- This patch creates a new secure RPC engine for HBase, which provides Kerberos based authentication of clients, and a token-based authentication mechanism for mapreduce jobs. Primary components of the patch are: a new maven profile for secure Hadoop/HBase: hadoop-0.20S Secure Hadoop dependent classes are separated under a pseudo-module in the security/ directory. These source and test directories are only including if building the secure Hadoop profile Currently the security classes get packaged with the regular HBase build artifacts. We need a way to at least override project.version, so we can append something like a "-security" suffix indicating the additional security components. The pseudo-module here is really a half-step forward. It enables the security code to be optionally included in the build for now, and sets up the structure for a security module. But we still will want to pursue full modularization (see HBASE-4336 ), which will allow packing the security code in a separate build artifact. a new RPC engine providing kerberos and token-based authentication: org.apache.hadoop.hbase.ipc.SecureRpcEngine implementation under security/src/main/java/org/apache/hadoop/hbase/ipc/ The implementation classes extend the existing HBaseClient and HBaseServer to share as much of the RPC code as possible. The main override is of the connection classes to allow control over the SASL negotiation of secure connections existing RPC changes The existing HBaseClient and HBaseServer have been modified to make subclassing possible All references to Hadoop UserGroupInformation have been replaced with org.apache.hadoop.hbase.security.User to insulate from future dependencies on specific Hadoop versions a coprocessor endpoint for obtaining new authentication tokens: TokenProvider, and supporting classes for token generation and synchronization (incorporating HBASE-3615 ) implementation is under security/src/main/java/org/apache/hadoop/hbase/security/token/ Secret keys for token generation and verification are synchronized throughout the cluster in zookeeper, under /hbase/tokenauth/keys To enable secure RPC, add the following configuration to hbase-site.xml: <property> <name>hadoop.security.authorization</name> <value>true</value> </property> <property> <name>hadoop.security.authentication</name> <value>kerberos</value> </property> <property> <name>hbase.rpc.engine</name> <value>org.apache.hadoop.hbase.ipc.SecureRpcEngine</value> </property> <property> <name>hbase.coprocessor.region.classes</name> <value>org.apache.hadoop.hbase.security.token.TokenProvider</value> </property> In addition, the master and regionserver processes must be configured for kerberos authentication using the properties: hbase.(master|regionserver).keytab.file hbase.(master|regionserver).kerberos.principal hbase.(master|regionserver).kerberos.https.principal This addresses bug HBASE-2742 . https://issues.apache.org/jira/browse/HBASE-2742 Diffs (updated) conf/hbase-policy.xml PRE-CREATION pom.xml 9d42e2b security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/SecureConnectionHeader.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/Status.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationKey.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationProtocol.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenIdentifier.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSelector.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/TokenProvider.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/ZKLeaderManager.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/ZKSecretWatcher.java PRE-CREATION security/src/test/java/org/apache/hadoop/hbase/security/token/TestTokenAuthentication.java PRE-CREATION security/src/test/java/org/apache/hadoop/hbase/security/token/TestZKSecretWatcher.java PRE-CREATION security/src/test/resources/hbase-site.xml PRE-CREATION src/main/java/org/apache/hadoop/hbase/HServerAddress.java f28240c src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 06a2312 src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java 904c107 src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 1365411 src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java 18310d6 src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 337da78 src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 4a8918a src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java 000f99c src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java 965102c src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 1f8b629 src/main/java/org/apache/hadoop/hbase/ipc/RequestContext.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java 1b5629a src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java 60a9248 src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java db07ed1 src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java ad88b76 src/main/java/org/apache/hadoop/hbase/master/HMaster.java 3f8f929 src/main/java/org/apache/hadoop/hbase/security/KerberosInfo.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/security/TokenInfo.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/security/User.java 00bd06d src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java bb67e53 src/main/resources/hbase-default.xml 3785533 src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java 7194c02 src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java 3982eff Diff: https://reviews.apache.org/r/1991/diff Testing ------- A full test suite run showed failures in TestMasterFailover (timeout) and TestAvroServer. I'll look into those. Cluster testing (1 master + 3 slaves in ec2) with Kerberos authentication and map reduce token authentication, using YCSB, RowCounter, PerformanceEvaluation. Thanks, Gary
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/1991/#review3056
          -----------------------------------------------------------

          Ship it!

          • Andrew

          On 2011-10-26 20:23:19, Gary Helmling wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/1991/

          -----------------------------------------------------------

          (Updated 2011-10-26 20:23:19)

          Review request for hbase.

          Summary

          -------

          This patch creates a new secure RPC engine for HBase, which provides Kerberos based authentication of clients, and a token-based authentication mechanism for mapreduce jobs. Primary components of the patch are:

          - a new maven profile for secure Hadoop/HBase: hadoop-0.20S

          - Secure Hadoop dependent classes are separated under a pseudo-module in the security/ directory. These source and test directories are only including if building the secure Hadoop profile

          - Currently the security classes get packaged with the regular HBase build artifacts. We need a way to at least override project.version, so we can append something like a "-security" suffix indicating the additional security components.

          - The pseudo-module here is really a half-step forward. It enables the security code to be optionally included in the build for now, and sets up the structure for a security module. But we still will want to pursue full modularization (see HBASE-4336), which will allow packing the security code in a separate build artifact.

          - a new RPC engine providing kerberos and token-based authentication: org.apache.hadoop.hbase.ipc.SecureRpcEngine

          - implementation under security/src/main/java/org/apache/hadoop/hbase/ipc/

          - The implementation classes extend the existing HBaseClient and HBaseServer to share as much of the RPC code as possible. The main override is of the connection classes to allow control over the SASL negotiation of secure connections

          - existing RPC changes

          - The existing HBaseClient and HBaseServer have been modified to make subclassing possible

          - All references to Hadoop UserGroupInformation have been replaced with org.apache.hadoop.hbase.security.User to insulate from future dependencies on specific Hadoop versions

          - a coprocessor endpoint for obtaining new authentication tokens: TokenProvider, and supporting classes for token generation and synchronization (incorporating HBASE-3615)

          - implementation is under security/src/main/java/org/apache/hadoop/hbase/security/token/

          - Secret keys for token generation and verification are synchronized throughout the cluster in zookeeper, under /hbase/tokenauth/keys

          To enable secure RPC, add the following configuration to hbase-site.xml:

          <property>

          <name>hadoop.security.authorization</name>

          <value>true</value>

          </property>

          <property>

          <name>hadoop.security.authentication</name>

          <value>kerberos</value>

          </property>

          <property>

          <name>hbase.rpc.engine</name>

          <value>org.apache.hadoop.hbase.ipc.SecureRpcEngine</value>

          </property>

          <property>

          <name>hbase.coprocessor.region.classes</name>

          <value>org.apache.hadoop.hbase.security.token.TokenProvider</value>

          </property>

          In addition, the master and regionserver processes must be configured for kerberos authentication using the properties:

          * hbase.(master|regionserver).keytab.file

          * hbase.(master|regionserver).kerberos.principal

          * hbase.(master|regionserver).kerberos.https.principal

          This addresses bug HBASE-2742.

          https://issues.apache.org/jira/browse/HBASE-2742

          Diffs

          -----

          conf/hbase-policy.xml PRE-CREATION

          pom.xml 9d42e2b

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureConnectionHeader.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/ipc/Status.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationKey.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationProtocol.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenIdentifier.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSelector.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/TokenProvider.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/ZKLeaderManager.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/ZKSecretWatcher.java PRE-CREATION

          security/src/test/java/org/apache/hadoop/hbase/security/token/TestTokenAuthentication.java PRE-CREATION

          security/src/test/java/org/apache/hadoop/hbase/security/token/TestZKSecretWatcher.java PRE-CREATION

          security/src/test/resources/hbase-site.xml PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/HServerAddress.java f28240c

          src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 06a2312

          src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java 904c107

          src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 1365411

          src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java 18310d6

          src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 337da78

          src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 4a8918a

          src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java 000f99c

          src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java 965102c

          src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 1f8b629

          src/main/java/org/apache/hadoop/hbase/ipc/RequestContext.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java 1b5629a

          src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java 60a9248

          src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java db07ed1

          src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java ad88b76

          src/main/java/org/apache/hadoop/hbase/master/HMaster.java 3f8f929

          src/main/java/org/apache/hadoop/hbase/security/KerberosInfo.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/security/TokenInfo.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/security/User.java 00bd06d

          src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java bb67e53

          src/main/resources/hbase-default.xml 3785533

          src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java 7194c02

          src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java 3982eff

          Diff: https://reviews.apache.org/r/1991/diff

          Testing

          -------

          A full test suite run showed failures in TestMasterFailover (timeout) and TestAvroServer. I'll look into those.

          Cluster testing (1 master + 3 slaves in ec2) with Kerberos authentication and map reduce token authentication, using YCSB, RowCounter, PerformanceEvaluation.

          Thanks,

          Gary

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1991/#review3056 ----------------------------------------------------------- Ship it! Andrew On 2011-10-26 20:23:19, Gary Helmling wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1991/ ----------------------------------------------------------- (Updated 2011-10-26 20:23:19) Review request for hbase. Summary ------- This patch creates a new secure RPC engine for HBase, which provides Kerberos based authentication of clients, and a token-based authentication mechanism for mapreduce jobs. Primary components of the patch are: - a new maven profile for secure Hadoop/HBase: hadoop-0.20S - Secure Hadoop dependent classes are separated under a pseudo-module in the security/ directory. These source and test directories are only including if building the secure Hadoop profile - Currently the security classes get packaged with the regular HBase build artifacts. We need a way to at least override project.version, so we can append something like a "-security" suffix indicating the additional security components. - The pseudo-module here is really a half-step forward. It enables the security code to be optionally included in the build for now, and sets up the structure for a security module. But we still will want to pursue full modularization (see HBASE-4336 ), which will allow packing the security code in a separate build artifact. - a new RPC engine providing kerberos and token-based authentication: org.apache.hadoop.hbase.ipc.SecureRpcEngine - implementation under security/src/main/java/org/apache/hadoop/hbase/ipc/ - The implementation classes extend the existing HBaseClient and HBaseServer to share as much of the RPC code as possible. The main override is of the connection classes to allow control over the SASL negotiation of secure connections - existing RPC changes - The existing HBaseClient and HBaseServer have been modified to make subclassing possible - All references to Hadoop UserGroupInformation have been replaced with org.apache.hadoop.hbase.security.User to insulate from future dependencies on specific Hadoop versions - a coprocessor endpoint for obtaining new authentication tokens: TokenProvider, and supporting classes for token generation and synchronization (incorporating HBASE-3615 ) - implementation is under security/src/main/java/org/apache/hadoop/hbase/security/token/ - Secret keys for token generation and verification are synchronized throughout the cluster in zookeeper, under /hbase/tokenauth/keys To enable secure RPC, add the following configuration to hbase-site.xml: <property> <name>hadoop.security.authorization</name> <value>true</value> </property> <property> <name>hadoop.security.authentication</name> <value>kerberos</value> </property> <property> <name>hbase.rpc.engine</name> <value>org.apache.hadoop.hbase.ipc.SecureRpcEngine</value> </property> <property> <name>hbase.coprocessor.region.classes</name> <value>org.apache.hadoop.hbase.security.token.TokenProvider</value> </property> In addition, the master and regionserver processes must be configured for kerberos authentication using the properties: * hbase.(master|regionserver).keytab.file * hbase.(master|regionserver).kerberos.principal * hbase.(master|regionserver).kerberos.https.principal This addresses bug HBASE-2742 . https://issues.apache.org/jira/browse/HBASE-2742 Diffs ----- conf/hbase-policy.xml PRE-CREATION pom.xml 9d42e2b security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/SecureConnectionHeader.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/Status.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationKey.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationProtocol.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenIdentifier.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSelector.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/TokenProvider.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/ZKLeaderManager.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/ZKSecretWatcher.java PRE-CREATION security/src/test/java/org/apache/hadoop/hbase/security/token/TestTokenAuthentication.java PRE-CREATION security/src/test/java/org/apache/hadoop/hbase/security/token/TestZKSecretWatcher.java PRE-CREATION security/src/test/resources/hbase-site.xml PRE-CREATION src/main/java/org/apache/hadoop/hbase/HServerAddress.java f28240c src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 06a2312 src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java 904c107 src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 1365411 src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java 18310d6 src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 337da78 src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 4a8918a src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java 000f99c src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java 965102c src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 1f8b629 src/main/java/org/apache/hadoop/hbase/ipc/RequestContext.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java 1b5629a src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java 60a9248 src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java db07ed1 src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java ad88b76 src/main/java/org/apache/hadoop/hbase/master/HMaster.java 3f8f929 src/main/java/org/apache/hadoop/hbase/security/KerberosInfo.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/security/TokenInfo.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/security/User.java 00bd06d src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java bb67e53 src/main/resources/hbase-default.xml 3785533 src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java 7194c02 src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java 3982eff Diff: https://reviews.apache.org/r/1991/diff Testing ------- A full test suite run showed failures in TestMasterFailover (timeout) and TestAvroServer. I'll look into those. Cluster testing (1 master + 3 slaves in ec2) with Kerberos authentication and map reduce token authentication, using YCSB, RowCounter, PerformanceEvaluation. Thanks, Gary
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/1991/#review3058
          -----------------------------------------------------------

          I made a start; more to follow later.

          conf/hbase-policy.xml
          <https://reviews.apache.org/r/1991/#comment6821>

          Apache license?

          pom.xml
          <https://reviews.apache.org/r/1991/#comment6819>

          We need to change this when we ship 0.92? Can you use variable here? $

          {pom.version}

          ?

          pom.xml
          <https://reviews.apache.org/r/1991/#comment6820>

          This is probably best place for this code for now. Was thinking src/security but that gets weird when test code and main. This is pseudo-maven-modules for now.

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java
          <https://reviews.apache.org/r/1991/#comment6822>

          Minor nit: My guess is that this is not your code. Why not just a return CONDITIONS... no need of this if ... return true else return false

          • Michael

          On 2011-10-26 20:23:19, Gary Helmling wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/1991/

          -----------------------------------------------------------

          (Updated 2011-10-26 20:23:19)

          Review request for hbase.

          Summary

          -------

          This patch creates a new secure RPC engine for HBase, which provides Kerberos based authentication of clients, and a token-based authentication mechanism for mapreduce jobs. Primary components of the patch are:

          - a new maven profile for secure Hadoop/HBase: hadoop-0.20S

          - Secure Hadoop dependent classes are separated under a pseudo-module in the security/ directory. These source and test directories are only including if building the secure Hadoop profile

          - Currently the security classes get packaged with the regular HBase build artifacts. We need a way to at least override project.version, so we can append something like a "-security" suffix indicating the additional security components.

          - The pseudo-module here is really a half-step forward. It enables the security code to be optionally included in the build for now, and sets up the structure for a security module. But we still will want to pursue full modularization (see HBASE-4336), which will allow packing the security code in a separate build artifact.

          - a new RPC engine providing kerberos and token-based authentication: org.apache.hadoop.hbase.ipc.SecureRpcEngine

          - implementation under security/src/main/java/org/apache/hadoop/hbase/ipc/

          - The implementation classes extend the existing HBaseClient and HBaseServer to share as much of the RPC code as possible. The main override is of the connection classes to allow control over the SASL negotiation of secure connections

          - existing RPC changes

          - The existing HBaseClient and HBaseServer have been modified to make subclassing possible

          - All references to Hadoop UserGroupInformation have been replaced with org.apache.hadoop.hbase.security.User to insulate from future dependencies on specific Hadoop versions

          - a coprocessor endpoint for obtaining new authentication tokens: TokenProvider, and supporting classes for token generation and synchronization (incorporating HBASE-3615)

          - implementation is under security/src/main/java/org/apache/hadoop/hbase/security/token/

          - Secret keys for token generation and verification are synchronized throughout the cluster in zookeeper, under /hbase/tokenauth/keys

          To enable secure RPC, add the following configuration to hbase-site.xml:

          <property>

          <name>hadoop.security.authorization</name>

          <value>true</value>

          </property>

          <property>

          <name>hadoop.security.authentication</name>

          <value>kerberos</value>

          </property>

          <property>

          <name>hbase.rpc.engine</name>

          <value>org.apache.hadoop.hbase.ipc.SecureRpcEngine</value>

          </property>

          <property>

          <name>hbase.coprocessor.region.classes</name>

          <value>org.apache.hadoop.hbase.security.token.TokenProvider</value>

          </property>

          In addition, the master and regionserver processes must be configured for kerberos authentication using the properties:

          * hbase.(master|regionserver).keytab.file

          * hbase.(master|regionserver).kerberos.principal

          * hbase.(master|regionserver).kerberos.https.principal

          This addresses bug HBASE-2742.

          https://issues.apache.org/jira/browse/HBASE-2742

          Diffs

          -----

          conf/hbase-policy.xml PRE-CREATION

          pom.xml 9d42e2b

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureConnectionHeader.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/ipc/Status.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationKey.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationProtocol.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenIdentifier.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSelector.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/TokenProvider.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/ZKLeaderManager.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/ZKSecretWatcher.java PRE-CREATION

          security/src/test/java/org/apache/hadoop/hbase/security/token/TestTokenAuthentication.java PRE-CREATION

          security/src/test/java/org/apache/hadoop/hbase/security/token/TestZKSecretWatcher.java PRE-CREATION

          security/src/test/resources/hbase-site.xml PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/HServerAddress.java f28240c

          src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 06a2312

          src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java 904c107

          src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 1365411

          src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java 18310d6

          src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 337da78

          src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 4a8918a

          src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java 000f99c

          src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java 965102c

          src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 1f8b629

          src/main/java/org/apache/hadoop/hbase/ipc/RequestContext.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java 1b5629a

          src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java 60a9248

          src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java db07ed1

          src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java ad88b76

          src/main/java/org/apache/hadoop/hbase/master/HMaster.java 3f8f929

          src/main/java/org/apache/hadoop/hbase/security/KerberosInfo.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/security/TokenInfo.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/security/User.java 00bd06d

          src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java bb67e53

          src/main/resources/hbase-default.xml 3785533

          src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java 7194c02

          src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java 3982eff

          Diff: https://reviews.apache.org/r/1991/diff

          Testing

          -------

          A full test suite run showed failures in TestMasterFailover (timeout) and TestAvroServer. I'll look into those.

          Cluster testing (1 master + 3 slaves in ec2) with Kerberos authentication and map reduce token authentication, using YCSB, RowCounter, PerformanceEvaluation.

          Thanks,

          Gary

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1991/#review3058 ----------------------------------------------------------- I made a start; more to follow later. conf/hbase-policy.xml < https://reviews.apache.org/r/1991/#comment6821 > Apache license? pom.xml < https://reviews.apache.org/r/1991/#comment6819 > We need to change this when we ship 0.92? Can you use variable here? $ {pom.version} ? pom.xml < https://reviews.apache.org/r/1991/#comment6820 > This is probably best place for this code for now. Was thinking src/security but that gets weird when test code and main. This is pseudo-maven-modules for now. security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java < https://reviews.apache.org/r/1991/#comment6822 > Minor nit: My guess is that this is not your code. Why not just a return CONDITIONS... no need of this if ... return true else return false Michael On 2011-10-26 20:23:19, Gary Helmling wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1991/ ----------------------------------------------------------- (Updated 2011-10-26 20:23:19) Review request for hbase. Summary ------- This patch creates a new secure RPC engine for HBase, which provides Kerberos based authentication of clients, and a token-based authentication mechanism for mapreduce jobs. Primary components of the patch are: - a new maven profile for secure Hadoop/HBase: hadoop-0.20S - Secure Hadoop dependent classes are separated under a pseudo-module in the security/ directory. These source and test directories are only including if building the secure Hadoop profile - Currently the security classes get packaged with the regular HBase build artifacts. We need a way to at least override project.version, so we can append something like a "-security" suffix indicating the additional security components. - The pseudo-module here is really a half-step forward. It enables the security code to be optionally included in the build for now, and sets up the structure for a security module. But we still will want to pursue full modularization (see HBASE-4336 ), which will allow packing the security code in a separate build artifact. - a new RPC engine providing kerberos and token-based authentication: org.apache.hadoop.hbase.ipc.SecureRpcEngine - implementation under security/src/main/java/org/apache/hadoop/hbase/ipc/ - The implementation classes extend the existing HBaseClient and HBaseServer to share as much of the RPC code as possible. The main override is of the connection classes to allow control over the SASL negotiation of secure connections - existing RPC changes - The existing HBaseClient and HBaseServer have been modified to make subclassing possible - All references to Hadoop UserGroupInformation have been replaced with org.apache.hadoop.hbase.security.User to insulate from future dependencies on specific Hadoop versions - a coprocessor endpoint for obtaining new authentication tokens: TokenProvider, and supporting classes for token generation and synchronization (incorporating HBASE-3615 ) - implementation is under security/src/main/java/org/apache/hadoop/hbase/security/token/ - Secret keys for token generation and verification are synchronized throughout the cluster in zookeeper, under /hbase/tokenauth/keys To enable secure RPC, add the following configuration to hbase-site.xml: <property> <name>hadoop.security.authorization</name> <value>true</value> </property> <property> <name>hadoop.security.authentication</name> <value>kerberos</value> </property> <property> <name>hbase.rpc.engine</name> <value>org.apache.hadoop.hbase.ipc.SecureRpcEngine</value> </property> <property> <name>hbase.coprocessor.region.classes</name> <value>org.apache.hadoop.hbase.security.token.TokenProvider</value> </property> In addition, the master and regionserver processes must be configured for kerberos authentication using the properties: * hbase.(master|regionserver).keytab.file * hbase.(master|regionserver).kerberos.principal * hbase.(master|regionserver).kerberos.https.principal This addresses bug HBASE-2742 . https://issues.apache.org/jira/browse/HBASE-2742 Diffs ----- conf/hbase-policy.xml PRE-CREATION pom.xml 9d42e2b security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/SecureConnectionHeader.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/Status.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationKey.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationProtocol.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenIdentifier.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSelector.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/TokenProvider.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/ZKLeaderManager.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/ZKSecretWatcher.java PRE-CREATION security/src/test/java/org/apache/hadoop/hbase/security/token/TestTokenAuthentication.java PRE-CREATION security/src/test/java/org/apache/hadoop/hbase/security/token/TestZKSecretWatcher.java PRE-CREATION security/src/test/resources/hbase-site.xml PRE-CREATION src/main/java/org/apache/hadoop/hbase/HServerAddress.java f28240c src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 06a2312 src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java 904c107 src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 1365411 src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java 18310d6 src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 337da78 src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 4a8918a src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java 000f99c src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java 965102c src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 1f8b629 src/main/java/org/apache/hadoop/hbase/ipc/RequestContext.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java 1b5629a src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java 60a9248 src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java db07ed1 src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java ad88b76 src/main/java/org/apache/hadoop/hbase/master/HMaster.java 3f8f929 src/main/java/org/apache/hadoop/hbase/security/KerberosInfo.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/security/TokenInfo.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/security/User.java 00bd06d src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java bb67e53 src/main/resources/hbase-default.xml 3785533 src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java 7194c02 src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java 3982eff Diff: https://reviews.apache.org/r/1991/diff Testing ------- A full test suite run showed failures in TestMasterFailover (timeout) and TestAvroServer. I'll look into those. Cluster testing (1 master + 3 slaves in ec2) with Kerberos authentication and map reduce token authentication, using YCSB, RowCounter, PerformanceEvaluation. Thanks, Gary
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/1991/#review3072
          -----------------------------------------------------------

          Ship it!

          +1 on getting it in.

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java
          <https://reviews.apache.org/r/1991/#comment6843>

          Good. We're pretty far from hadoop rpc now, huh..

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java
          <https://reviews.apache.org/r/1991/#comment6844>

          This happen on every rpc? It won't show in hbase logs because our logger is up in hadoop ipc package.... that should be fine.

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java
          <https://reviews.apache.org/r/1991/#comment6845>

          This if/else seems a little fuzzy. Server-side, this is how it chooses an auth method?

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java
          <https://reviews.apache.org/r/1991/#comment6846>

          Should we set saslRpcClient to null after dispose? Is it good to just swallow the exception? Should log at least?

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java
          <https://reviews.apache.org/r/1991/#comment6847>

          Should this be configurable?

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java
          <https://reviews.apache.org/r/1991/#comment6848>

          Formatting. This copied from hadoop?

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java
          <https://reviews.apache.org/r/1991/#comment6849>

          What about the 'length' thing that we added to hbase? I don't see you processing it here. I suppose its ok because these are different client and servers and they just don't do it (and things like asynchbase are not going to do secure hase any time soon).

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java
          <https://reviews.apache.org/r/1991/#comment6850>

          In secure hbase we don't do that delayed response stuff that was recently added to insecure rpc?

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java
          <https://reviews.apache.org/r/1991/#comment6851>

          Why we need this? This in all our rpc'ing?

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java
          <https://reviews.apache.org/r/1991/#comment6852>

          Where does TRACELOG come from? Why not LOG.trace? Is it from parent? Should logging be from this class?

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java
          <https://reviews.apache.org/r/1991/#comment6853>

          An insecure client cannot talk to a secure server (which makes 'sense')

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java
          <https://reviews.apache.org/r/1991/#comment6854>

          If result is null, this will work?

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java
          <https://reviews.apache.org/r/1991/#comment6855>

          This copy/pasted from hadoop? Else its style violation (minor nit)

          security/src/main/java/org/apache/hadoop/hbase/ipc/Status.java
          <https://reviews.apache.org/r/1991/#comment6856>

          This has been added already.

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenIdentifier.java
          <https://reviews.apache.org/r/1991/#comment6857>

          These are copied from hadoop? Don't look like Gary style.

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java
          <https://reviews.apache.org/r/1991/#comment6858>

          Should a 'secret manager' have a public constructor?

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java
          <https://reviews.apache.org/r/1991/#comment6859>

          ditoo

          security/src/main/java/org/apache/hadoop/hbase/security/token/ZKLeaderManager.java
          <https://reviews.apache.org/r/1991/#comment6860>

          Is this 'generic' zk facility? Put in zk package?

          • Michael

          On 2011-10-26 20:23:19, Gary Helmling wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/1991/

          -----------------------------------------------------------

          (Updated 2011-10-26 20:23:19)

          Review request for hbase.

          Summary

          -------

          This patch creates a new secure RPC engine for HBase, which provides Kerberos based authentication of clients, and a token-based authentication mechanism for mapreduce jobs. Primary components of the patch are:

          - a new maven profile for secure Hadoop/HBase: hadoop-0.20S

          - Secure Hadoop dependent classes are separated under a pseudo-module in the security/ directory. These source and test directories are only including if building the secure Hadoop profile

          - Currently the security classes get packaged with the regular HBase build artifacts. We need a way to at least override project.version, so we can append something like a "-security" suffix indicating the additional security components.

          - The pseudo-module here is really a half-step forward. It enables the security code to be optionally included in the build for now, and sets up the structure for a security module. But we still will want to pursue full modularization (see HBASE-4336), which will allow packing the security code in a separate build artifact.

          - a new RPC engine providing kerberos and token-based authentication: org.apache.hadoop.hbase.ipc.SecureRpcEngine

          - implementation under security/src/main/java/org/apache/hadoop/hbase/ipc/

          - The implementation classes extend the existing HBaseClient and HBaseServer to share as much of the RPC code as possible. The main override is of the connection classes to allow control over the SASL negotiation of secure connections

          - existing RPC changes

          - The existing HBaseClient and HBaseServer have been modified to make subclassing possible

          - All references to Hadoop UserGroupInformation have been replaced with org.apache.hadoop.hbase.security.User to insulate from future dependencies on specific Hadoop versions

          - a coprocessor endpoint for obtaining new authentication tokens: TokenProvider, and supporting classes for token generation and synchronization (incorporating HBASE-3615)

          - implementation is under security/src/main/java/org/apache/hadoop/hbase/security/token/

          - Secret keys for token generation and verification are synchronized throughout the cluster in zookeeper, under /hbase/tokenauth/keys

          To enable secure RPC, add the following configuration to hbase-site.xml:

          <property>

          <name>hadoop.security.authorization</name>

          <value>true</value>

          </property>

          <property>

          <name>hadoop.security.authentication</name>

          <value>kerberos</value>

          </property>

          <property>

          <name>hbase.rpc.engine</name>

          <value>org.apache.hadoop.hbase.ipc.SecureRpcEngine</value>

          </property>

          <property>

          <name>hbase.coprocessor.region.classes</name>

          <value>org.apache.hadoop.hbase.security.token.TokenProvider</value>

          </property>

          In addition, the master and regionserver processes must be configured for kerberos authentication using the properties:

          * hbase.(master|regionserver).keytab.file

          * hbase.(master|regionserver).kerberos.principal

          * hbase.(master|regionserver).kerberos.https.principal

          This addresses bug HBASE-2742.

          https://issues.apache.org/jira/browse/HBASE-2742

          Diffs

          -----

          conf/hbase-policy.xml PRE-CREATION

          pom.xml 9d42e2b

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureConnectionHeader.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/ipc/Status.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationKey.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationProtocol.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenIdentifier.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSelector.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/TokenProvider.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/ZKLeaderManager.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/ZKSecretWatcher.java PRE-CREATION

          security/src/test/java/org/apache/hadoop/hbase/security/token/TestTokenAuthentication.java PRE-CREATION

          security/src/test/java/org/apache/hadoop/hbase/security/token/TestZKSecretWatcher.java PRE-CREATION

          security/src/test/resources/hbase-site.xml PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/HServerAddress.java f28240c

          src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 06a2312

          src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java 904c107

          src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 1365411

          src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java 18310d6

          src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 337da78

          src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 4a8918a

          src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java 000f99c

          src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java 965102c

          src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 1f8b629

          src/main/java/org/apache/hadoop/hbase/ipc/RequestContext.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java 1b5629a

          src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java 60a9248

          src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java db07ed1

          src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java ad88b76

          src/main/java/org/apache/hadoop/hbase/master/HMaster.java 3f8f929

          src/main/java/org/apache/hadoop/hbase/security/KerberosInfo.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/security/TokenInfo.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/security/User.java 00bd06d

          src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java bb67e53

          src/main/resources/hbase-default.xml 3785533

          src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java 7194c02

          src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java 3982eff

          Diff: https://reviews.apache.org/r/1991/diff

          Testing

          -------

          A full test suite run showed failures in TestMasterFailover (timeout) and TestAvroServer. I'll look into those.

          Cluster testing (1 master + 3 slaves in ec2) with Kerberos authentication and map reduce token authentication, using YCSB, RowCounter, PerformanceEvaluation.

          Thanks,

          Gary

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1991/#review3072 ----------------------------------------------------------- Ship it! +1 on getting it in. security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java < https://reviews.apache.org/r/1991/#comment6843 > Good. We're pretty far from hadoop rpc now, huh.. security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java < https://reviews.apache.org/r/1991/#comment6844 > This happen on every rpc? It won't show in hbase logs because our logger is up in hadoop ipc package.... that should be fine. security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java < https://reviews.apache.org/r/1991/#comment6845 > This if/else seems a little fuzzy. Server-side, this is how it chooses an auth method? security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java < https://reviews.apache.org/r/1991/#comment6846 > Should we set saslRpcClient to null after dispose? Is it good to just swallow the exception? Should log at least? security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java < https://reviews.apache.org/r/1991/#comment6847 > Should this be configurable? security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java < https://reviews.apache.org/r/1991/#comment6848 > Formatting. This copied from hadoop? security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java < https://reviews.apache.org/r/1991/#comment6849 > What about the 'length' thing that we added to hbase? I don't see you processing it here. I suppose its ok because these are different client and servers and they just don't do it (and things like asynchbase are not going to do secure hase any time soon). security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java < https://reviews.apache.org/r/1991/#comment6850 > In secure hbase we don't do that delayed response stuff that was recently added to insecure rpc? security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java < https://reviews.apache.org/r/1991/#comment6851 > Why we need this? This in all our rpc'ing? security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java < https://reviews.apache.org/r/1991/#comment6852 > Where does TRACELOG come from? Why not LOG.trace? Is it from parent? Should logging be from this class? security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java < https://reviews.apache.org/r/1991/#comment6853 > An insecure client cannot talk to a secure server (which makes 'sense') security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java < https://reviews.apache.org/r/1991/#comment6854 > If result is null, this will work? security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java < https://reviews.apache.org/r/1991/#comment6855 > This copy/pasted from hadoop? Else its style violation (minor nit) security/src/main/java/org/apache/hadoop/hbase/ipc/Status.java < https://reviews.apache.org/r/1991/#comment6856 > This has been added already. security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenIdentifier.java < https://reviews.apache.org/r/1991/#comment6857 > These are copied from hadoop? Don't look like Gary style. security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java < https://reviews.apache.org/r/1991/#comment6858 > Should a 'secret manager' have a public constructor? security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java < https://reviews.apache.org/r/1991/#comment6859 > ditoo security/src/main/java/org/apache/hadoop/hbase/security/token/ZKLeaderManager.java < https://reviews.apache.org/r/1991/#comment6860 > Is this 'generic' zk facility? Put in zk package? Michael On 2011-10-26 20:23:19, Gary Helmling wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1991/ ----------------------------------------------------------- (Updated 2011-10-26 20:23:19) Review request for hbase. Summary ------- This patch creates a new secure RPC engine for HBase, which provides Kerberos based authentication of clients, and a token-based authentication mechanism for mapreduce jobs. Primary components of the patch are: - a new maven profile for secure Hadoop/HBase: hadoop-0.20S - Secure Hadoop dependent classes are separated under a pseudo-module in the security/ directory. These source and test directories are only including if building the secure Hadoop profile - Currently the security classes get packaged with the regular HBase build artifacts. We need a way to at least override project.version, so we can append something like a "-security" suffix indicating the additional security components. - The pseudo-module here is really a half-step forward. It enables the security code to be optionally included in the build for now, and sets up the structure for a security module. But we still will want to pursue full modularization (see HBASE-4336 ), which will allow packing the security code in a separate build artifact. - a new RPC engine providing kerberos and token-based authentication: org.apache.hadoop.hbase.ipc.SecureRpcEngine - implementation under security/src/main/java/org/apache/hadoop/hbase/ipc/ - The implementation classes extend the existing HBaseClient and HBaseServer to share as much of the RPC code as possible. The main override is of the connection classes to allow control over the SASL negotiation of secure connections - existing RPC changes - The existing HBaseClient and HBaseServer have been modified to make subclassing possible - All references to Hadoop UserGroupInformation have been replaced with org.apache.hadoop.hbase.security.User to insulate from future dependencies on specific Hadoop versions - a coprocessor endpoint for obtaining new authentication tokens: TokenProvider, and supporting classes for token generation and synchronization (incorporating HBASE-3615 ) - implementation is under security/src/main/java/org/apache/hadoop/hbase/security/token/ - Secret keys for token generation and verification are synchronized throughout the cluster in zookeeper, under /hbase/tokenauth/keys To enable secure RPC, add the following configuration to hbase-site.xml: <property> <name>hadoop.security.authorization</name> <value>true</value> </property> <property> <name>hadoop.security.authentication</name> <value>kerberos</value> </property> <property> <name>hbase.rpc.engine</name> <value>org.apache.hadoop.hbase.ipc.SecureRpcEngine</value> </property> <property> <name>hbase.coprocessor.region.classes</name> <value>org.apache.hadoop.hbase.security.token.TokenProvider</value> </property> In addition, the master and regionserver processes must be configured for kerberos authentication using the properties: * hbase.(master|regionserver).keytab.file * hbase.(master|regionserver).kerberos.principal * hbase.(master|regionserver).kerberos.https.principal This addresses bug HBASE-2742 . https://issues.apache.org/jira/browse/HBASE-2742 Diffs ----- conf/hbase-policy.xml PRE-CREATION pom.xml 9d42e2b security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/SecureConnectionHeader.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/Status.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationKey.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationProtocol.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenIdentifier.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSelector.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/TokenProvider.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/ZKLeaderManager.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/ZKSecretWatcher.java PRE-CREATION security/src/test/java/org/apache/hadoop/hbase/security/token/TestTokenAuthentication.java PRE-CREATION security/src/test/java/org/apache/hadoop/hbase/security/token/TestZKSecretWatcher.java PRE-CREATION security/src/test/resources/hbase-site.xml PRE-CREATION src/main/java/org/apache/hadoop/hbase/HServerAddress.java f28240c src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 06a2312 src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java 904c107 src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 1365411 src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java 18310d6 src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 337da78 src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 4a8918a src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java 000f99c src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java 965102c src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 1f8b629 src/main/java/org/apache/hadoop/hbase/ipc/RequestContext.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java 1b5629a src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java 60a9248 src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java db07ed1 src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java ad88b76 src/main/java/org/apache/hadoop/hbase/master/HMaster.java 3f8f929 src/main/java/org/apache/hadoop/hbase/security/KerberosInfo.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/security/TokenInfo.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/security/User.java 00bd06d src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java bb67e53 src/main/resources/hbase-default.xml 3785533 src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java 7194c02 src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java 3982eff Diff: https://reviews.apache.org/r/1991/diff Testing ------- A full test suite run showed failures in TestMasterFailover (timeout) and TestAvroServer. I'll look into those. Cluster testing (1 master + 3 slaves in ec2) with Kerberos authentication and map reduce token authentication, using YCSB, RowCounter, PerformanceEvaluation. Thanks, Gary
          Hide
          stack added a comment -

          This is in way of a 0.92 release.

          Show
          stack added a comment - This is in way of a 0.92 release.
          Hide
          jiraposter@reviews.apache.org added a comment -

          On 2011-11-04 23:17:46, Michael Stack wrote:

          > pom.xml, line 1335

          > <https://reviews.apache.org/r/1991/diff/4/?file=53576#file53576line1335>

          >

          > This is probably best place for this code for now. Was thinking src/security but that gets weird when test code and main. This is pseudo-maven-modules for now.

          Yeah, my thinking was once we get to full blown modules we won't have to move the source around, just add a security pom.xml.

          On 2011-11-04 23:17:46, Michael Stack wrote:

          > conf/hbase-policy.xml, line 1

          > <https://reviews.apache.org/r/1991/diff/4/?file=53575#file53575line1>

          >

          > Apache license?

          Added.

          On 2011-11-04 23:17:46, Michael Stack wrote:

          > pom.xml, line 1320

          > <https://reviews.apache.org/r/1991/diff/4/?file=53576#file53576line1320>

          >

          > We need to change this when we ship 0.92? Can you use variable here? ${pom.version}?

          Removed this bit to use the project version. I'm trying out using finalName in the profile, which would allow us to wind up with the name "hbase-0.92.0-security". Dependent projects could then use artifactId=hbase, version=0.92.0, classifier=security. Or we could use versions:set -DnewVersion=0.92.0-security during the build, but making it more automatic seems nicer.

          On 2011-11-04 23:17:46, Michael Stack wrote:

          > security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java, line 158

          > <https://reviews.apache.org/r/1991/diff/4/?file=53577#file53577line158>

          >

          > Minor nit: My guess is that this is not your code. Why not just a return CONDITIONS... no need of this if ... return true else return false

          Yes, was copied. Changed to just do a return.

          • Gary

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/1991/#review3058
          -----------------------------------------------------------

          On 2011-10-26 20:23:19, Gary Helmling wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/1991/

          -----------------------------------------------------------

          (Updated 2011-10-26 20:23:19)

          Review request for hbase.

          Summary

          -------

          This patch creates a new secure RPC engine for HBase, which provides Kerberos based authentication of clients, and a token-based authentication mechanism for mapreduce jobs. Primary components of the patch are:

          - a new maven profile for secure Hadoop/HBase: hadoop-0.20S

          - Secure Hadoop dependent classes are separated under a pseudo-module in the security/ directory. These source and test directories are only including if building the secure Hadoop profile

          - Currently the security classes get packaged with the regular HBase build artifacts. We need a way to at least override project.version, so we can append something like a "-security" suffix indicating the additional security components.

          - The pseudo-module here is really a half-step forward. It enables the security code to be optionally included in the build for now, and sets up the structure for a security module. But we still will want to pursue full modularization (see HBASE-4336), which will allow packing the security code in a separate build artifact.

          - a new RPC engine providing kerberos and token-based authentication: org.apache.hadoop.hbase.ipc.SecureRpcEngine

          - implementation under security/src/main/java/org/apache/hadoop/hbase/ipc/

          - The implementation classes extend the existing HBaseClient and HBaseServer to share as much of the RPC code as possible. The main override is of the connection classes to allow control over the SASL negotiation of secure connections

          - existing RPC changes

          - The existing HBaseClient and HBaseServer have been modified to make subclassing possible

          - All references to Hadoop UserGroupInformation have been replaced with org.apache.hadoop.hbase.security.User to insulate from future dependencies on specific Hadoop versions

          - a coprocessor endpoint for obtaining new authentication tokens: TokenProvider, and supporting classes for token generation and synchronization (incorporating HBASE-3615)

          - implementation is under security/src/main/java/org/apache/hadoop/hbase/security/token/

          - Secret keys for token generation and verification are synchronized throughout the cluster in zookeeper, under /hbase/tokenauth/keys

          To enable secure RPC, add the following configuration to hbase-site.xml:

          <property>

          <name>hadoop.security.authorization</name>

          <value>true</value>

          </property>

          <property>

          <name>hadoop.security.authentication</name>

          <value>kerberos</value>

          </property>

          <property>

          <name>hbase.rpc.engine</name>

          <value>org.apache.hadoop.hbase.ipc.SecureRpcEngine</value>

          </property>

          <property>

          <name>hbase.coprocessor.region.classes</name>

          <value>org.apache.hadoop.hbase.security.token.TokenProvider</value>

          </property>

          In addition, the master and regionserver processes must be configured for kerberos authentication using the properties:

          * hbase.(master|regionserver).keytab.file

          * hbase.(master|regionserver).kerberos.principal

          * hbase.(master|regionserver).kerberos.https.principal

          This addresses bug HBASE-2742.

          https://issues.apache.org/jira/browse/HBASE-2742

          Diffs

          -----

          conf/hbase-policy.xml PRE-CREATION

          pom.xml 9d42e2b

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureConnectionHeader.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/ipc/Status.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationKey.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationProtocol.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenIdentifier.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSelector.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/TokenProvider.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/ZKLeaderManager.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/ZKSecretWatcher.java PRE-CREATION

          security/src/test/java/org/apache/hadoop/hbase/security/token/TestTokenAuthentication.java PRE-CREATION

          security/src/test/java/org/apache/hadoop/hbase/security/token/TestZKSecretWatcher.java PRE-CREATION

          security/src/test/resources/hbase-site.xml PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/HServerAddress.java f28240c

          src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 06a2312

          src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java 904c107

          src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 1365411

          src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java 18310d6

          src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 337da78

          src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 4a8918a

          src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java 000f99c

          src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java 965102c

          src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 1f8b629

          src/main/java/org/apache/hadoop/hbase/ipc/RequestContext.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java 1b5629a

          src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java 60a9248

          src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java db07ed1

          src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java ad88b76

          src/main/java/org/apache/hadoop/hbase/master/HMaster.java 3f8f929

          src/main/java/org/apache/hadoop/hbase/security/KerberosInfo.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/security/TokenInfo.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/security/User.java 00bd06d

          src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java bb67e53

          src/main/resources/hbase-default.xml 3785533

          src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java 7194c02

          src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java 3982eff

          Diff: https://reviews.apache.org/r/1991/diff

          Testing

          -------

          A full test suite run showed failures in TestMasterFailover (timeout) and TestAvroServer. I'll look into those.

          Cluster testing (1 master + 3 slaves in ec2) with Kerberos authentication and map reduce token authentication, using YCSB, RowCounter, PerformanceEvaluation.

          Thanks,

          Gary

          Show
          jiraposter@reviews.apache.org added a comment - On 2011-11-04 23:17:46, Michael Stack wrote: > pom.xml, line 1335 > < https://reviews.apache.org/r/1991/diff/4/?file=53576#file53576line1335 > > > This is probably best place for this code for now. Was thinking src/security but that gets weird when test code and main. This is pseudo-maven-modules for now. Yeah, my thinking was once we get to full blown modules we won't have to move the source around, just add a security pom.xml. On 2011-11-04 23:17:46, Michael Stack wrote: > conf/hbase-policy.xml, line 1 > < https://reviews.apache.org/r/1991/diff/4/?file=53575#file53575line1 > > > Apache license? Added. On 2011-11-04 23:17:46, Michael Stack wrote: > pom.xml, line 1320 > < https://reviews.apache.org/r/1991/diff/4/?file=53576#file53576line1320 > > > We need to change this when we ship 0.92? Can you use variable here? ${pom.version}? Removed this bit to use the project version. I'm trying out using finalName in the profile, which would allow us to wind up with the name "hbase-0.92.0-security". Dependent projects could then use artifactId=hbase, version=0.92.0, classifier=security. Or we could use versions:set -DnewVersion=0.92.0-security during the build, but making it more automatic seems nicer. On 2011-11-04 23:17:46, Michael Stack wrote: > security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java, line 158 > < https://reviews.apache.org/r/1991/diff/4/?file=53577#file53577line158 > > > Minor nit: My guess is that this is not your code. Why not just a return CONDITIONS... no need of this if ... return true else return false Yes, was copied. Changed to just do a return. Gary ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1991/#review3058 ----------------------------------------------------------- On 2011-10-26 20:23:19, Gary Helmling wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1991/ ----------------------------------------------------------- (Updated 2011-10-26 20:23:19) Review request for hbase. Summary ------- This patch creates a new secure RPC engine for HBase, which provides Kerberos based authentication of clients, and a token-based authentication mechanism for mapreduce jobs. Primary components of the patch are: - a new maven profile for secure Hadoop/HBase: hadoop-0.20S - Secure Hadoop dependent classes are separated under a pseudo-module in the security/ directory. These source and test directories are only including if building the secure Hadoop profile - Currently the security classes get packaged with the regular HBase build artifacts. We need a way to at least override project.version, so we can append something like a "-security" suffix indicating the additional security components. - The pseudo-module here is really a half-step forward. It enables the security code to be optionally included in the build for now, and sets up the structure for a security module. But we still will want to pursue full modularization (see HBASE-4336 ), which will allow packing the security code in a separate build artifact. - a new RPC engine providing kerberos and token-based authentication: org.apache.hadoop.hbase.ipc.SecureRpcEngine - implementation under security/src/main/java/org/apache/hadoop/hbase/ipc/ - The implementation classes extend the existing HBaseClient and HBaseServer to share as much of the RPC code as possible. The main override is of the connection classes to allow control over the SASL negotiation of secure connections - existing RPC changes - The existing HBaseClient and HBaseServer have been modified to make subclassing possible - All references to Hadoop UserGroupInformation have been replaced with org.apache.hadoop.hbase.security.User to insulate from future dependencies on specific Hadoop versions - a coprocessor endpoint for obtaining new authentication tokens: TokenProvider, and supporting classes for token generation and synchronization (incorporating HBASE-3615 ) - implementation is under security/src/main/java/org/apache/hadoop/hbase/security/token/ - Secret keys for token generation and verification are synchronized throughout the cluster in zookeeper, under /hbase/tokenauth/keys To enable secure RPC, add the following configuration to hbase-site.xml: <property> <name>hadoop.security.authorization</name> <value>true</value> </property> <property> <name>hadoop.security.authentication</name> <value>kerberos</value> </property> <property> <name>hbase.rpc.engine</name> <value>org.apache.hadoop.hbase.ipc.SecureRpcEngine</value> </property> <property> <name>hbase.coprocessor.region.classes</name> <value>org.apache.hadoop.hbase.security.token.TokenProvider</value> </property> In addition, the master and regionserver processes must be configured for kerberos authentication using the properties: * hbase.(master|regionserver).keytab.file * hbase.(master|regionserver).kerberos.principal * hbase.(master|regionserver).kerberos.https.principal This addresses bug HBASE-2742 . https://issues.apache.org/jira/browse/HBASE-2742 Diffs ----- conf/hbase-policy.xml PRE-CREATION pom.xml 9d42e2b security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/SecureConnectionHeader.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/Status.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationKey.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationProtocol.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenIdentifier.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSelector.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/TokenProvider.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/ZKLeaderManager.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/ZKSecretWatcher.java PRE-CREATION security/src/test/java/org/apache/hadoop/hbase/security/token/TestTokenAuthentication.java PRE-CREATION security/src/test/java/org/apache/hadoop/hbase/security/token/TestZKSecretWatcher.java PRE-CREATION security/src/test/resources/hbase-site.xml PRE-CREATION src/main/java/org/apache/hadoop/hbase/HServerAddress.java f28240c src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 06a2312 src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java 904c107 src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 1365411 src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java 18310d6 src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 337da78 src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 4a8918a src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java 000f99c src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java 965102c src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 1f8b629 src/main/java/org/apache/hadoop/hbase/ipc/RequestContext.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java 1b5629a src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java 60a9248 src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java db07ed1 src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java ad88b76 src/main/java/org/apache/hadoop/hbase/master/HMaster.java 3f8f929 src/main/java/org/apache/hadoop/hbase/security/KerberosInfo.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/security/TokenInfo.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/security/User.java 00bd06d src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java bb67e53 src/main/resources/hbase-default.xml 3785533 src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java 7194c02 src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java 3982eff Diff: https://reviews.apache.org/r/1991/diff Testing ------- A full test suite run showed failures in TestMasterFailover (timeout) and TestAvroServer. I'll look into those. Cluster testing (1 master + 3 slaves in ec2) with Kerberos authentication and map reduce token authentication, using YCSB, RowCounter, PerformanceEvaluation. Thanks, Gary
          Hide
          jiraposter@reviews.apache.org added a comment -

          On 2011-11-07 06:59:29, Michael Stack wrote:

          > security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java, line 128

          > <https://reviews.apache.org/r/1991/diff/4/?file=53577#file53577line128>

          >

          > This if/else seems a little fuzzy. Server-side, this is how it chooses an auth method?

          Yes, this matches up with server side. If not using kerberos auth, then does SIMPLE auth, otherwise if a token is present does DIGEST, otherwise requires kerberos ticket.

          On 2011-11-07 06:59:29, Michael Stack wrote:

          > security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java, line 233

          > <https://reviews.apache.org/r/1991/diff/4/?file=53577#file53577line233>

          >

          > Formatting. This copied from hadoop?

          Yes, from Hadoop. Cleaning up.

          On 2011-11-07 06:59:29, Michael Stack wrote:

          > security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java, line 195

          > <https://reviews.apache.org/r/1991/diff/4/?file=53577#file53577line195>

          >

          > Should this be configurable?

          Some config here is probably good, for the case of those who are very sensitive to latency.

          On 2011-11-07 06:59:29, Michael Stack wrote:

          > security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java, line 366

          > <https://reviews.apache.org/r/1991/diff/4/?file=53579#file53579line366>

          >

          > Why we need this? This in all our rpc'ing?

          Yes, WritableRpcEngine$Server.call() does the same. Don't know why we'd be implementing RPC protocols with non-public methods, but predates me...

          On 2011-11-07 06:59:29, Michael Stack wrote:

          > security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java, line 85

          > <https://reviews.apache.org/r/1991/diff/4/?file=53580#file53580line85>

          >

          > An insecure client cannot talk to a secure server (which makes 'sense')

          Yes, HEADER bytes are distinct and will prevent this as well.

          On 2011-11-07 06:59:29, Michael Stack wrote:

          > security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java, line 597

          > <https://reviews.apache.org/r/1991/diff/4/?file=53580#file53580line597>

          >

          > This copy/pasted from hadoop? Else its style violation (minor nit)

          Yes, fixed anyway. (HBaseServer$Connection.processData has the same problem).

          On 2011-11-07 06:59:29, Michael Stack wrote:

          > security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java, line 87

          > <https://reviews.apache.org/r/1991/diff/4/?file=53589#file53589line87>

          >

          > Should a 'secret manager' have a public constructor?

          It's currently instantiated from SecureServer, so we need to access it somehow. With secured ZK access, znode ACLs will prevent creating a secret manager instance to obtain the secret keys. There is an opening here with coprocessors though, since they run in process on the RS. We'll ultimately need to lock those down separately.

          On 2011-11-07 06:59:29, Michael Stack wrote:

          > security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenIdentifier.java, line 43

          > <https://reviews.apache.org/r/1991/diff/4/?file=53588#file53588line43>

          >

          > These are copied from hadoop? Don't look like Gary style.

          Yes, largely copied and modified.

          On 2011-11-07 06:59:29, Michael Stack wrote:

          > security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java, line 147

          > <https://reviews.apache.org/r/1991/diff/4/?file=53577#file53577line147>

          >

          > Should we set saslRpcClient to null after dispose? Is it good to just swallow the exception? Should log at least?

          Good points, updating.

          On 2011-11-07 06:59:29, Michael Stack wrote:

          > security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java, line 98

          > <https://reviews.apache.org/r/1991/diff/4/?file=53589#file53589line98>

          >

          > ditoo

          This also gets called from SecureServer, so it needs to be able to access.

          On 2011-11-07 06:59:29, Michael Stack wrote:

          > security/src/main/java/org/apache/hadoop/hbase/security/token/ZKLeaderManager.java, line 47

          > <https://reviews.apache.org/r/1991/diff/4/?file=53593#file53593line47>

          >

          > Is this 'generic' zk facility? Put in zk package?

          Sure, will move to o.a.h.h.zookeeper. I very briefly considered trying to combine with ActiveMasterManager when implementing this but that seemed too far out of scope.

          On 2011-11-07 06:59:29, Michael Stack wrote:

          > security/src/main/java/org/apache/hadoop/hbase/ipc/Status.java, line 1

          > <https://reviews.apache.org/r/1991/diff/4/?file=53581#file53581line1>

          >

          > This has been added already.

          Yes, will update the patch against that change and remove this.

          On 2011-11-07 06:59:29, Michael Stack wrote:

          > security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java, line 172

          > <https://reviews.apache.org/r/1991/diff/4/?file=53579#file53579line172>

          >

          > In secure hbase we don't do that delayed response stuff that was recently added to insecure rpc?

          Delayed stuff should work since it's using the rpc engine interfaces, but I'll check if there are upstream discrepancies since this patch.

          On 2011-11-07 06:59:29, Michael Stack wrote:

          > security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java, line 345

          > <https://reviews.apache.org/r/1991/diff/4/?file=53577#file53577line345>

          >

          > What about the 'length' thing that we added to hbase? I don't see you processing it here. I suppose its ok because these are different client and servers and they just don't do it (and things like asynchbase are not going to do secure hase any time soon).

          Yeah, doesn't do "length" currently. I could add it, but asynchbase would still have to deal with the sasl negotiation on connection setup.

          On 2011-11-07 06:59:29, Michael Stack wrote:

          > security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java, line 381

          > <https://reviews.apache.org/r/1991/diff/4/?file=53579#file53579line381>

          >

          > Where does TRACELOG come from? Why not LOG.trace? Is it from parent? Should logging be from this class?

          TRACELOG comes from HBaseServer I think. We added it with the simple rpc call introspection I did. We wanted it to be a separate logger so it could be easily enabled to get those stats without bringing in all the rest of the debug logging.

          On 2011-11-07 06:59:29, Michael Stack wrote:

          > security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java, line 123

          > <https://reviews.apache.org/r/1991/diff/4/?file=53580#file53580line123>

          >

          > If result is null, this will work?

          Hmm, this is just doing same as HBaseServer, isn't it?

          • Gary

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/1991/#review3072
          -----------------------------------------------------------

          On 2011-10-26 20:23:19, Gary Helmling wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/1991/

          -----------------------------------------------------------

          (Updated 2011-10-26 20:23:19)

          Review request for hbase.

          Summary

          -------

          This patch creates a new secure RPC engine for HBase, which provides Kerberos based authentication of clients, and a token-based authentication mechanism for mapreduce jobs. Primary components of the patch are:

          - a new maven profile for secure Hadoop/HBase: hadoop-0.20S

          - Secure Hadoop dependent classes are separated under a pseudo-module in the security/ directory. These source and test directories are only including if building the secure Hadoop profile

          - Currently the security classes get packaged with the regular HBase build artifacts. We need a way to at least override project.version, so we can append something like a "-security" suffix indicating the additional security components.

          - The pseudo-module here is really a half-step forward. It enables the security code to be optionally included in the build for now, and sets up the structure for a security module. But we still will want to pursue full modularization (see HBASE-4336), which will allow packing the security code in a separate build artifact.

          - a new RPC engine providing kerberos and token-based authentication: org.apache.hadoop.hbase.ipc.SecureRpcEngine

          - implementation under security/src/main/java/org/apache/hadoop/hbase/ipc/

          - The implementation classes extend the existing HBaseClient and HBaseServer to share as much of the RPC code as possible. The main override is of the connection classes to allow control over the SASL negotiation of secure connections

          - existing RPC changes

          - The existing HBaseClient and HBaseServer have been modified to make subclassing possible

          - All references to Hadoop UserGroupInformation have been replaced with org.apache.hadoop.hbase.security.User to insulate from future dependencies on specific Hadoop versions

          - a coprocessor endpoint for obtaining new authentication tokens: TokenProvider, and supporting classes for token generation and synchronization (incorporating HBASE-3615)

          - implementation is under security/src/main/java/org/apache/hadoop/hbase/security/token/

          - Secret keys for token generation and verification are synchronized throughout the cluster in zookeeper, under /hbase/tokenauth/keys

          To enable secure RPC, add the following configuration to hbase-site.xml:

          <property>

          <name>hadoop.security.authorization</name>

          <value>true</value>

          </property>

          <property>

          <name>hadoop.security.authentication</name>

          <value>kerberos</value>

          </property>

          <property>

          <name>hbase.rpc.engine</name>

          <value>org.apache.hadoop.hbase.ipc.SecureRpcEngine</value>

          </property>

          <property>

          <name>hbase.coprocessor.region.classes</name>

          <value>org.apache.hadoop.hbase.security.token.TokenProvider</value>

          </property>

          In addition, the master and regionserver processes must be configured for kerberos authentication using the properties:

          * hbase.(master|regionserver).keytab.file

          * hbase.(master|regionserver).kerberos.principal

          * hbase.(master|regionserver).kerberos.https.principal

          This addresses bug HBASE-2742.

          https://issues.apache.org/jira/browse/HBASE-2742

          Diffs

          -----

          conf/hbase-policy.xml PRE-CREATION

          pom.xml 9d42e2b

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureConnectionHeader.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/ipc/Status.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationKey.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationProtocol.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenIdentifier.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSelector.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/TokenProvider.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/ZKLeaderManager.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/ZKSecretWatcher.java PRE-CREATION

          security/src/test/java/org/apache/hadoop/hbase/security/token/TestTokenAuthentication.java PRE-CREATION

          security/src/test/java/org/apache/hadoop/hbase/security/token/TestZKSecretWatcher.java PRE-CREATION

          security/src/test/resources/hbase-site.xml PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/HServerAddress.java f28240c

          src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 06a2312

          src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java 904c107

          src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 1365411

          src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java 18310d6

          src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 337da78

          src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 4a8918a

          src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java 000f99c

          src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java 965102c

          src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 1f8b629

          src/main/java/org/apache/hadoop/hbase/ipc/RequestContext.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java 1b5629a

          src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java 60a9248

          src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java db07ed1

          src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java ad88b76

          src/main/java/org/apache/hadoop/hbase/master/HMaster.java 3f8f929

          src/main/java/org/apache/hadoop/hbase/security/KerberosInfo.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/security/TokenInfo.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/security/User.java 00bd06d

          src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java bb67e53

          src/main/resources/hbase-default.xml 3785533

          src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java 7194c02

          src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java 3982eff

          Diff: https://reviews.apache.org/r/1991/diff

          Testing

          -------

          A full test suite run showed failures in TestMasterFailover (timeout) and TestAvroServer. I'll look into those.

          Cluster testing (1 master + 3 slaves in ec2) with Kerberos authentication and map reduce token authentication, using YCSB, RowCounter, PerformanceEvaluation.

          Thanks,

          Gary

          Show
          jiraposter@reviews.apache.org added a comment - On 2011-11-07 06:59:29, Michael Stack wrote: > security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java, line 128 > < https://reviews.apache.org/r/1991/diff/4/?file=53577#file53577line128 > > > This if/else seems a little fuzzy. Server-side, this is how it chooses an auth method? Yes, this matches up with server side. If not using kerberos auth, then does SIMPLE auth, otherwise if a token is present does DIGEST, otherwise requires kerberos ticket. On 2011-11-07 06:59:29, Michael Stack wrote: > security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java, line 233 > < https://reviews.apache.org/r/1991/diff/4/?file=53577#file53577line233 > > > Formatting. This copied from hadoop? Yes, from Hadoop. Cleaning up. On 2011-11-07 06:59:29, Michael Stack wrote: > security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java, line 195 > < https://reviews.apache.org/r/1991/diff/4/?file=53577#file53577line195 > > > Should this be configurable? Some config here is probably good, for the case of those who are very sensitive to latency. On 2011-11-07 06:59:29, Michael Stack wrote: > security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java, line 366 > < https://reviews.apache.org/r/1991/diff/4/?file=53579#file53579line366 > > > Why we need this? This in all our rpc'ing? Yes, WritableRpcEngine$Server.call() does the same. Don't know why we'd be implementing RPC protocols with non-public methods, but predates me... On 2011-11-07 06:59:29, Michael Stack wrote: > security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java, line 85 > < https://reviews.apache.org/r/1991/diff/4/?file=53580#file53580line85 > > > An insecure client cannot talk to a secure server (which makes 'sense') Yes, HEADER bytes are distinct and will prevent this as well. On 2011-11-07 06:59:29, Michael Stack wrote: > security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java, line 597 > < https://reviews.apache.org/r/1991/diff/4/?file=53580#file53580line597 > > > This copy/pasted from hadoop? Else its style violation (minor nit) Yes, fixed anyway. (HBaseServer$Connection.processData has the same problem). On 2011-11-07 06:59:29, Michael Stack wrote: > security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java, line 87 > < https://reviews.apache.org/r/1991/diff/4/?file=53589#file53589line87 > > > Should a 'secret manager' have a public constructor? It's currently instantiated from SecureServer, so we need to access it somehow. With secured ZK access, znode ACLs will prevent creating a secret manager instance to obtain the secret keys. There is an opening here with coprocessors though, since they run in process on the RS. We'll ultimately need to lock those down separately. On 2011-11-07 06:59:29, Michael Stack wrote: > security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenIdentifier.java, line 43 > < https://reviews.apache.org/r/1991/diff/4/?file=53588#file53588line43 > > > These are copied from hadoop? Don't look like Gary style. Yes, largely copied and modified. On 2011-11-07 06:59:29, Michael Stack wrote: > security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java, line 147 > < https://reviews.apache.org/r/1991/diff/4/?file=53577#file53577line147 > > > Should we set saslRpcClient to null after dispose? Is it good to just swallow the exception? Should log at least? Good points, updating. On 2011-11-07 06:59:29, Michael Stack wrote: > security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java, line 98 > < https://reviews.apache.org/r/1991/diff/4/?file=53589#file53589line98 > > > ditoo This also gets called from SecureServer, so it needs to be able to access. On 2011-11-07 06:59:29, Michael Stack wrote: > security/src/main/java/org/apache/hadoop/hbase/security/token/ZKLeaderManager.java, line 47 > < https://reviews.apache.org/r/1991/diff/4/?file=53593#file53593line47 > > > Is this 'generic' zk facility? Put in zk package? Sure, will move to o.a.h.h.zookeeper. I very briefly considered trying to combine with ActiveMasterManager when implementing this but that seemed too far out of scope. On 2011-11-07 06:59:29, Michael Stack wrote: > security/src/main/java/org/apache/hadoop/hbase/ipc/Status.java, line 1 > < https://reviews.apache.org/r/1991/diff/4/?file=53581#file53581line1 > > > This has been added already. Yes, will update the patch against that change and remove this. On 2011-11-07 06:59:29, Michael Stack wrote: > security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java, line 172 > < https://reviews.apache.org/r/1991/diff/4/?file=53579#file53579line172 > > > In secure hbase we don't do that delayed response stuff that was recently added to insecure rpc? Delayed stuff should work since it's using the rpc engine interfaces, but I'll check if there are upstream discrepancies since this patch. On 2011-11-07 06:59:29, Michael Stack wrote: > security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java, line 345 > < https://reviews.apache.org/r/1991/diff/4/?file=53577#file53577line345 > > > What about the 'length' thing that we added to hbase? I don't see you processing it here. I suppose its ok because these are different client and servers and they just don't do it (and things like asynchbase are not going to do secure hase any time soon). Yeah, doesn't do "length" currently. I could add it, but asynchbase would still have to deal with the sasl negotiation on connection setup. On 2011-11-07 06:59:29, Michael Stack wrote: > security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java, line 381 > < https://reviews.apache.org/r/1991/diff/4/?file=53579#file53579line381 > > > Where does TRACELOG come from? Why not LOG.trace? Is it from parent? Should logging be from this class? TRACELOG comes from HBaseServer I think. We added it with the simple rpc call introspection I did. We wanted it to be a separate logger so it could be easily enabled to get those stats without bringing in all the rest of the debug logging. On 2011-11-07 06:59:29, Michael Stack wrote: > security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java, line 123 > < https://reviews.apache.org/r/1991/diff/4/?file=53580#file53580line123 > > > If result is null, this will work? Hmm, this is just doing same as HBaseServer, isn't it? Gary ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1991/#review3072 ----------------------------------------------------------- On 2011-10-26 20:23:19, Gary Helmling wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1991/ ----------------------------------------------------------- (Updated 2011-10-26 20:23:19) Review request for hbase. Summary ------- This patch creates a new secure RPC engine for HBase, which provides Kerberos based authentication of clients, and a token-based authentication mechanism for mapreduce jobs. Primary components of the patch are: - a new maven profile for secure Hadoop/HBase: hadoop-0.20S - Secure Hadoop dependent classes are separated under a pseudo-module in the security/ directory. These source and test directories are only including if building the secure Hadoop profile - Currently the security classes get packaged with the regular HBase build artifacts. We need a way to at least override project.version, so we can append something like a "-security" suffix indicating the additional security components. - The pseudo-module here is really a half-step forward. It enables the security code to be optionally included in the build for now, and sets up the structure for a security module. But we still will want to pursue full modularization (see HBASE-4336 ), which will allow packing the security code in a separate build artifact. - a new RPC engine providing kerberos and token-based authentication: org.apache.hadoop.hbase.ipc.SecureRpcEngine - implementation under security/src/main/java/org/apache/hadoop/hbase/ipc/ - The implementation classes extend the existing HBaseClient and HBaseServer to share as much of the RPC code as possible. The main override is of the connection classes to allow control over the SASL negotiation of secure connections - existing RPC changes - The existing HBaseClient and HBaseServer have been modified to make subclassing possible - All references to Hadoop UserGroupInformation have been replaced with org.apache.hadoop.hbase.security.User to insulate from future dependencies on specific Hadoop versions - a coprocessor endpoint for obtaining new authentication tokens: TokenProvider, and supporting classes for token generation and synchronization (incorporating HBASE-3615 ) - implementation is under security/src/main/java/org/apache/hadoop/hbase/security/token/ - Secret keys for token generation and verification are synchronized throughout the cluster in zookeeper, under /hbase/tokenauth/keys To enable secure RPC, add the following configuration to hbase-site.xml: <property> <name>hadoop.security.authorization</name> <value>true</value> </property> <property> <name>hadoop.security.authentication</name> <value>kerberos</value> </property> <property> <name>hbase.rpc.engine</name> <value>org.apache.hadoop.hbase.ipc.SecureRpcEngine</value> </property> <property> <name>hbase.coprocessor.region.classes</name> <value>org.apache.hadoop.hbase.security.token.TokenProvider</value> </property> In addition, the master and regionserver processes must be configured for kerberos authentication using the properties: * hbase.(master|regionserver).keytab.file * hbase.(master|regionserver).kerberos.principal * hbase.(master|regionserver).kerberos.https.principal This addresses bug HBASE-2742 . https://issues.apache.org/jira/browse/HBASE-2742 Diffs ----- conf/hbase-policy.xml PRE-CREATION pom.xml 9d42e2b security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/SecureConnectionHeader.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/Status.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationKey.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationProtocol.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenIdentifier.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSelector.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/TokenProvider.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/ZKLeaderManager.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/ZKSecretWatcher.java PRE-CREATION security/src/test/java/org/apache/hadoop/hbase/security/token/TestTokenAuthentication.java PRE-CREATION security/src/test/java/org/apache/hadoop/hbase/security/token/TestZKSecretWatcher.java PRE-CREATION security/src/test/resources/hbase-site.xml PRE-CREATION src/main/java/org/apache/hadoop/hbase/HServerAddress.java f28240c src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 06a2312 src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java 904c107 src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 1365411 src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java 18310d6 src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 337da78 src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 4a8918a src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java 000f99c src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java 965102c src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 1f8b629 src/main/java/org/apache/hadoop/hbase/ipc/RequestContext.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java 1b5629a src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java 60a9248 src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java db07ed1 src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java ad88b76 src/main/java/org/apache/hadoop/hbase/master/HMaster.java 3f8f929 src/main/java/org/apache/hadoop/hbase/security/KerberosInfo.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/security/TokenInfo.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/security/User.java 00bd06d src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java bb67e53 src/main/resources/hbase-default.xml 3785533 src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java 7194c02 src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java 3982eff Diff: https://reviews.apache.org/r/1991/diff Testing ------- A full test suite run showed failures in TestMasterFailover (timeout) and TestAvroServer. I'll look into those. Cluster testing (1 master + 3 slaves in ec2) with Kerberos authentication and map reduce token authentication, using YCSB, RowCounter, PerformanceEvaluation. Thanks, Gary
          Hide
          jiraposter@reviews.apache.org added a comment -

          On 2011-11-07 06:59:29, Michael Stack wrote:

          > security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java, line 128

          > <https://reviews.apache.org/r/1991/diff/4/?file=53577#file53577line128>

          >

          > This if/else seems a little fuzzy. Server-side, this is how it chooses an auth method?

          Gary Helmling wrote:

          Yes, this matches up with server side. If not using kerberos auth, then does SIMPLE auth, otherwise if a token is present does DIGEST, otherwise requires kerberos ticket.

          Grand

          On 2011-11-07 06:59:29, Michael Stack wrote:

          > security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java, line 233

          > <https://reviews.apache.org/r/1991/diff/4/?file=53577#file53577line233>

          >

          > Formatting. This copied from hadoop?

          Gary Helmling wrote:

          Yes, from Hadoop. Cleaning up.

          I'd say if copied from hadoop, don't clean it up. Just leave it.

          On 2011-11-07 06:59:29, Michael Stack wrote:

          > security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java, line 345

          > <https://reviews.apache.org/r/1991/diff/4/?file=53577#file53577line345>

          >

          > What about the 'length' thing that we added to hbase? I don't see you processing it here. I suppose its ok because these are different client and servers and they just don't do it (and things like asynchbase are not going to do secure hase any time soon).

          Gary Helmling wrote:

          Yeah, doesn't do "length" currently. I could add it, but asynchbase would still have to deal with the sasl negotiation on connection setup.

          asynchbase is not going to be doing secure rpc anytime soon if ever so don't worry about it.

          On 2011-11-07 06:59:29, Michael Stack wrote:

          > security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java, line 366

          > <https://reviews.apache.org/r/1991/diff/4/?file=53579#file53579line366>

          >

          > Why we need this? This in all our rpc'ing?

          Gary Helmling wrote:

          Yes, WritableRpcEngine$Server.call() does the same. Don't know why we'd be implementing RPC protocols with non-public methods, but predates me...

          grand

          On 2011-11-07 06:59:29, Michael Stack wrote:

          > security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java, line 381

          > <https://reviews.apache.org/r/1991/diff/4/?file=53579#file53579line381>

          >

          > Where does TRACELOG come from? Why not LOG.trace? Is it from parent? Should logging be from this class?

          Gary Helmling wrote:

          TRACELOG comes from HBaseServer I think. We added it with the simple rpc call introspection I did. We wanted it to be a separate logger so it could be easily enabled to get those stats without bringing in all the rest of the debug logging.

          ok

          On 2011-11-07 06:59:29, Michael Stack wrote:

          > security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java, line 123

          > <https://reviews.apache.org/r/1991/diff/4/?file=53580#file53580line123>

          >

          > If result is null, this will work?

          Gary Helmling wrote:

          Hmm, this is just doing same as HBaseServer, isn't it?

          Fair enough

          On 2011-11-07 06:59:29, Michael Stack wrote:

          > security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java, line 597

          > <https://reviews.apache.org/r/1991/diff/4/?file=53580#file53580line597>

          >

          > This copy/pasted from hadoop? Else its style violation (minor nit)

          Gary Helmling wrote:

          Yes, fixed anyway. (HBaseServer$Connection.processData has the same problem).

          Don't bother fixing the hadoop copies (you'll be working on it till cows come home and then next update will mess it all up again – not to mention the pain mapping hadoop to the cleaned up hbase version)

          On 2011-11-07 06:59:29, Michael Stack wrote:

          > security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenIdentifier.java, line 43

          > <https://reviews.apache.org/r/1991/diff/4/?file=53588#file53588line43>

          >

          > These are copied from hadoop? Don't look like Gary style.

          Gary Helmling wrote:

          Yes, largely copied and modified.

          np

          On 2011-11-07 06:59:29, Michael Stack wrote:

          > security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java, line 87

          > <https://reviews.apache.org/r/1991/diff/4/?file=53589#file53589line87>

          >

          > Should a 'secret manager' have a public constructor?

          Gary Helmling wrote:

          It's currently instantiated from SecureServer, so we need to access it somehow. With secured ZK access, znode ACLs will prevent creating a secret manager instance to obtain the secret keys. There is an opening here with coprocessors though, since they run in process on the RS. We'll ultimately need to lock those down separately.

          Make a comment for now?

          • Michael

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/1991/#review3072
          -----------------------------------------------------------

          On 2011-10-26 20:23:19, Gary Helmling wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/1991/

          -----------------------------------------------------------

          (Updated 2011-10-26 20:23:19)

          Review request for hbase.

          Summary

          -------

          This patch creates a new secure RPC engine for HBase, which provides Kerberos based authentication of clients, and a token-based authentication mechanism for mapreduce jobs. Primary components of the patch are:

          - a new maven profile for secure Hadoop/HBase: hadoop-0.20S

          - Secure Hadoop dependent classes are separated under a pseudo-module in the security/ directory. These source and test directories are only including if building the secure Hadoop profile

          - Currently the security classes get packaged with the regular HBase build artifacts. We need a way to at least override project.version, so we can append something like a "-security" suffix indicating the additional security components.

          - The pseudo-module here is really a half-step forward. It enables the security code to be optionally included in the build for now, and sets up the structure for a security module. But we still will want to pursue full modularization (see HBASE-4336), which will allow packing the security code in a separate build artifact.

          - a new RPC engine providing kerberos and token-based authentication: org.apache.hadoop.hbase.ipc.SecureRpcEngine

          - implementation under security/src/main/java/org/apache/hadoop/hbase/ipc/

          - The implementation classes extend the existing HBaseClient and HBaseServer to share as much of the RPC code as possible. The main override is of the connection classes to allow control over the SASL negotiation of secure connections

          - existing RPC changes

          - The existing HBaseClient and HBaseServer have been modified to make subclassing possible

          - All references to Hadoop UserGroupInformation have been replaced with org.apache.hadoop.hbase.security.User to insulate from future dependencies on specific Hadoop versions

          - a coprocessor endpoint for obtaining new authentication tokens: TokenProvider, and supporting classes for token generation and synchronization (incorporating HBASE-3615)

          - implementation is under security/src/main/java/org/apache/hadoop/hbase/security/token/

          - Secret keys for token generation and verification are synchronized throughout the cluster in zookeeper, under /hbase/tokenauth/keys

          To enable secure RPC, add the following configuration to hbase-site.xml:

          <property>

          <name>hadoop.security.authorization</name>

          <value>true</value>

          </property>

          <property>

          <name>hadoop.security.authentication</name>

          <value>kerberos</value>

          </property>

          <property>

          <name>hbase.rpc.engine</name>

          <value>org.apache.hadoop.hbase.ipc.SecureRpcEngine</value>

          </property>

          <property>

          <name>hbase.coprocessor.region.classes</name>

          <value>org.apache.hadoop.hbase.security.token.TokenProvider</value>

          </property>

          In addition, the master and regionserver processes must be configured for kerberos authentication using the properties:

          * hbase.(master|regionserver).keytab.file

          * hbase.(master|regionserver).kerberos.principal

          * hbase.(master|regionserver).kerberos.https.principal

          This addresses bug HBASE-2742.

          https://issues.apache.org/jira/browse/HBASE-2742

          Diffs

          -----

          conf/hbase-policy.xml PRE-CREATION

          pom.xml 9d42e2b

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureConnectionHeader.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/ipc/Status.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationKey.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationProtocol.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenIdentifier.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSelector.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/TokenProvider.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/ZKLeaderManager.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/ZKSecretWatcher.java PRE-CREATION

          security/src/test/java/org/apache/hadoop/hbase/security/token/TestTokenAuthentication.java PRE-CREATION

          security/src/test/java/org/apache/hadoop/hbase/security/token/TestZKSecretWatcher.java PRE-CREATION

          security/src/test/resources/hbase-site.xml PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/HServerAddress.java f28240c

          src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 06a2312

          src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java 904c107

          src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 1365411

          src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java 18310d6

          src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 337da78

          src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 4a8918a

          src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java 000f99c

          src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java 965102c

          src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 1f8b629

          src/main/java/org/apache/hadoop/hbase/ipc/RequestContext.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java 1b5629a

          src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java 60a9248

          src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java db07ed1

          src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java ad88b76

          src/main/java/org/apache/hadoop/hbase/master/HMaster.java 3f8f929

          src/main/java/org/apache/hadoop/hbase/security/KerberosInfo.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/security/TokenInfo.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/security/User.java 00bd06d

          src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java bb67e53

          src/main/resources/hbase-default.xml 3785533

          src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java 7194c02

          src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java 3982eff

          Diff: https://reviews.apache.org/r/1991/diff

          Testing

          -------

          A full test suite run showed failures in TestMasterFailover (timeout) and TestAvroServer. I'll look into those.

          Cluster testing (1 master + 3 slaves in ec2) with Kerberos authentication and map reduce token authentication, using YCSB, RowCounter, PerformanceEvaluation.

          Thanks,

          Gary

          Show
          jiraposter@reviews.apache.org added a comment - On 2011-11-07 06:59:29, Michael Stack wrote: > security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java, line 128 > < https://reviews.apache.org/r/1991/diff/4/?file=53577#file53577line128 > > > This if/else seems a little fuzzy. Server-side, this is how it chooses an auth method? Gary Helmling wrote: Yes, this matches up with server side. If not using kerberos auth, then does SIMPLE auth, otherwise if a token is present does DIGEST, otherwise requires kerberos ticket. Grand On 2011-11-07 06:59:29, Michael Stack wrote: > security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java, line 233 > < https://reviews.apache.org/r/1991/diff/4/?file=53577#file53577line233 > > > Formatting. This copied from hadoop? Gary Helmling wrote: Yes, from Hadoop. Cleaning up. I'd say if copied from hadoop, don't clean it up. Just leave it. On 2011-11-07 06:59:29, Michael Stack wrote: > security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java, line 345 > < https://reviews.apache.org/r/1991/diff/4/?file=53577#file53577line345 > > > What about the 'length' thing that we added to hbase? I don't see you processing it here. I suppose its ok because these are different client and servers and they just don't do it (and things like asynchbase are not going to do secure hase any time soon). Gary Helmling wrote: Yeah, doesn't do "length" currently. I could add it, but asynchbase would still have to deal with the sasl negotiation on connection setup. asynchbase is not going to be doing secure rpc anytime soon if ever so don't worry about it. On 2011-11-07 06:59:29, Michael Stack wrote: > security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java, line 366 > < https://reviews.apache.org/r/1991/diff/4/?file=53579#file53579line366 > > > Why we need this? This in all our rpc'ing? Gary Helmling wrote: Yes, WritableRpcEngine$Server.call() does the same. Don't know why we'd be implementing RPC protocols with non-public methods, but predates me... grand On 2011-11-07 06:59:29, Michael Stack wrote: > security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java, line 381 > < https://reviews.apache.org/r/1991/diff/4/?file=53579#file53579line381 > > > Where does TRACELOG come from? Why not LOG.trace? Is it from parent? Should logging be from this class? Gary Helmling wrote: TRACELOG comes from HBaseServer I think. We added it with the simple rpc call introspection I did. We wanted it to be a separate logger so it could be easily enabled to get those stats without bringing in all the rest of the debug logging. ok On 2011-11-07 06:59:29, Michael Stack wrote: > security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java, line 123 > < https://reviews.apache.org/r/1991/diff/4/?file=53580#file53580line123 > > > If result is null, this will work? Gary Helmling wrote: Hmm, this is just doing same as HBaseServer, isn't it? Fair enough On 2011-11-07 06:59:29, Michael Stack wrote: > security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java, line 597 > < https://reviews.apache.org/r/1991/diff/4/?file=53580#file53580line597 > > > This copy/pasted from hadoop? Else its style violation (minor nit) Gary Helmling wrote: Yes, fixed anyway. (HBaseServer$Connection.processData has the same problem). Don't bother fixing the hadoop copies (you'll be working on it till cows come home and then next update will mess it all up again – not to mention the pain mapping hadoop to the cleaned up hbase version) On 2011-11-07 06:59:29, Michael Stack wrote: > security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenIdentifier.java, line 43 > < https://reviews.apache.org/r/1991/diff/4/?file=53588#file53588line43 > > > These are copied from hadoop? Don't look like Gary style. Gary Helmling wrote: Yes, largely copied and modified. np On 2011-11-07 06:59:29, Michael Stack wrote: > security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java, line 87 > < https://reviews.apache.org/r/1991/diff/4/?file=53589#file53589line87 > > > Should a 'secret manager' have a public constructor? Gary Helmling wrote: It's currently instantiated from SecureServer, so we need to access it somehow. With secured ZK access, znode ACLs will prevent creating a secret manager instance to obtain the secret keys. There is an opening here with coprocessors though, since they run in process on the RS. We'll ultimately need to lock those down separately. Make a comment for now? Michael ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1991/#review3072 ----------------------------------------------------------- On 2011-10-26 20:23:19, Gary Helmling wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1991/ ----------------------------------------------------------- (Updated 2011-10-26 20:23:19) Review request for hbase. Summary ------- This patch creates a new secure RPC engine for HBase, which provides Kerberos based authentication of clients, and a token-based authentication mechanism for mapreduce jobs. Primary components of the patch are: - a new maven profile for secure Hadoop/HBase: hadoop-0.20S - Secure Hadoop dependent classes are separated under a pseudo-module in the security/ directory. These source and test directories are only including if building the secure Hadoop profile - Currently the security classes get packaged with the regular HBase build artifacts. We need a way to at least override project.version, so we can append something like a "-security" suffix indicating the additional security components. - The pseudo-module here is really a half-step forward. It enables the security code to be optionally included in the build for now, and sets up the structure for a security module. But we still will want to pursue full modularization (see HBASE-4336 ), which will allow packing the security code in a separate build artifact. - a new RPC engine providing kerberos and token-based authentication: org.apache.hadoop.hbase.ipc.SecureRpcEngine - implementation under security/src/main/java/org/apache/hadoop/hbase/ipc/ - The implementation classes extend the existing HBaseClient and HBaseServer to share as much of the RPC code as possible. The main override is of the connection classes to allow control over the SASL negotiation of secure connections - existing RPC changes - The existing HBaseClient and HBaseServer have been modified to make subclassing possible - All references to Hadoop UserGroupInformation have been replaced with org.apache.hadoop.hbase.security.User to insulate from future dependencies on specific Hadoop versions - a coprocessor endpoint for obtaining new authentication tokens: TokenProvider, and supporting classes for token generation and synchronization (incorporating HBASE-3615 ) - implementation is under security/src/main/java/org/apache/hadoop/hbase/security/token/ - Secret keys for token generation and verification are synchronized throughout the cluster in zookeeper, under /hbase/tokenauth/keys To enable secure RPC, add the following configuration to hbase-site.xml: <property> <name>hadoop.security.authorization</name> <value>true</value> </property> <property> <name>hadoop.security.authentication</name> <value>kerberos</value> </property> <property> <name>hbase.rpc.engine</name> <value>org.apache.hadoop.hbase.ipc.SecureRpcEngine</value> </property> <property> <name>hbase.coprocessor.region.classes</name> <value>org.apache.hadoop.hbase.security.token.TokenProvider</value> </property> In addition, the master and regionserver processes must be configured for kerberos authentication using the properties: * hbase.(master|regionserver).keytab.file * hbase.(master|regionserver).kerberos.principal * hbase.(master|regionserver).kerberos.https.principal This addresses bug HBASE-2742 . https://issues.apache.org/jira/browse/HBASE-2742 Diffs ----- conf/hbase-policy.xml PRE-CREATION pom.xml 9d42e2b security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/SecureConnectionHeader.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/Status.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationKey.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationProtocol.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenIdentifier.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSelector.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/TokenProvider.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/ZKLeaderManager.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/ZKSecretWatcher.java PRE-CREATION security/src/test/java/org/apache/hadoop/hbase/security/token/TestTokenAuthentication.java PRE-CREATION security/src/test/java/org/apache/hadoop/hbase/security/token/TestZKSecretWatcher.java PRE-CREATION security/src/test/resources/hbase-site.xml PRE-CREATION src/main/java/org/apache/hadoop/hbase/HServerAddress.java f28240c src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 06a2312 src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java 904c107 src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 1365411 src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java 18310d6 src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 337da78 src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 4a8918a src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java 000f99c src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java 965102c src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 1f8b629 src/main/java/org/apache/hadoop/hbase/ipc/RequestContext.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java 1b5629a src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java 60a9248 src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java db07ed1 src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java ad88b76 src/main/java/org/apache/hadoop/hbase/master/HMaster.java 3f8f929 src/main/java/org/apache/hadoop/hbase/security/KerberosInfo.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/security/TokenInfo.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/security/User.java 00bd06d src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java bb67e53 src/main/resources/hbase-default.xml 3785533 src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java 7194c02 src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java 3982eff Diff: https://reviews.apache.org/r/1991/diff Testing ------- A full test suite run showed failures in TestMasterFailover (timeout) and TestAvroServer. I'll look into those. Cluster testing (1 master + 3 slaves in ec2) with Kerberos authentication and map reduce token authentication, using YCSB, RowCounter, PerformanceEvaluation. Thanks, Gary
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/1991/#review3198
          -----------------------------------------------------------

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java
          <https://reviews.apache.org/r/1991/#comment7094>

          I don't see this member used.
          Shall we remove this.authorize ?

          security/src/main/java/org/apache/hadoop/hbase/security/token/ZKSecretWatcher.java
          <https://reviews.apache.org/r/1991/#comment7095>

          I think we should respond to the IOE beyond logging it.

          src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
          <https://reviews.apache.org/r/1991/#comment7096>

          username should also be compared in equals().

          • Ted

          On 2011-10-26 20:23:19, Gary Helmling wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/1991/

          -----------------------------------------------------------

          (Updated 2011-10-26 20:23:19)

          Review request for hbase.

          Summary

          -------

          This patch creates a new secure RPC engine for HBase, which provides Kerberos based authentication of clients, and a token-based authentication mechanism for mapreduce jobs. Primary components of the patch are:

          - a new maven profile for secure Hadoop/HBase: hadoop-0.20S

          - Secure Hadoop dependent classes are separated under a pseudo-module in the security/ directory. These source and test directories are only including if building the secure Hadoop profile

          - Currently the security classes get packaged with the regular HBase build artifacts. We need a way to at least override project.version, so we can append something like a "-security" suffix indicating the additional security components.

          - The pseudo-module here is really a half-step forward. It enables the security code to be optionally included in the build for now, and sets up the structure for a security module. But we still will want to pursue full modularization (see HBASE-4336), which will allow packing the security code in a separate build artifact.

          - a new RPC engine providing kerberos and token-based authentication: org.apache.hadoop.hbase.ipc.SecureRpcEngine

          - implementation under security/src/main/java/org/apache/hadoop/hbase/ipc/

          - The implementation classes extend the existing HBaseClient and HBaseServer to share as much of the RPC code as possible. The main override is of the connection classes to allow control over the SASL negotiation of secure connections

          - existing RPC changes

          - The existing HBaseClient and HBaseServer have been modified to make subclassing possible

          - All references to Hadoop UserGroupInformation have been replaced with org.apache.hadoop.hbase.security.User to insulate from future dependencies on specific Hadoop versions

          - a coprocessor endpoint for obtaining new authentication tokens: TokenProvider, and supporting classes for token generation and synchronization (incorporating HBASE-3615)

          - implementation is under security/src/main/java/org/apache/hadoop/hbase/security/token/

          - Secret keys for token generation and verification are synchronized throughout the cluster in zookeeper, under /hbase/tokenauth/keys

          To enable secure RPC, add the following configuration to hbase-site.xml:

          <property>

          <name>hadoop.security.authorization</name>

          <value>true</value>

          </property>

          <property>

          <name>hadoop.security.authentication</name>

          <value>kerberos</value>

          </property>

          <property>

          <name>hbase.rpc.engine</name>

          <value>org.apache.hadoop.hbase.ipc.SecureRpcEngine</value>

          </property>

          <property>

          <name>hbase.coprocessor.region.classes</name>

          <value>org.apache.hadoop.hbase.security.token.TokenProvider</value>

          </property>

          In addition, the master and regionserver processes must be configured for kerberos authentication using the properties:

          * hbase.(master|regionserver).keytab.file

          * hbase.(master|regionserver).kerberos.principal

          * hbase.(master|regionserver).kerberos.https.principal

          This addresses bug HBASE-2742.

          https://issues.apache.org/jira/browse/HBASE-2742

          Diffs

          -----

          conf/hbase-policy.xml PRE-CREATION

          pom.xml 9d42e2b

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureConnectionHeader.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/ipc/Status.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationKey.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationProtocol.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenIdentifier.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSelector.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/TokenProvider.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/ZKLeaderManager.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/ZKSecretWatcher.java PRE-CREATION

          security/src/test/java/org/apache/hadoop/hbase/security/token/TestTokenAuthentication.java PRE-CREATION

          security/src/test/java/org/apache/hadoop/hbase/security/token/TestZKSecretWatcher.java PRE-CREATION

          security/src/test/resources/hbase-site.xml PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/HServerAddress.java f28240c

          src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 06a2312

          src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java 904c107

          src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 1365411

          src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java 18310d6

          src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 337da78

          src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 4a8918a

          src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java 000f99c

          src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java 965102c

          src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 1f8b629

          src/main/java/org/apache/hadoop/hbase/ipc/RequestContext.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java 1b5629a

          src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java 60a9248

          src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java db07ed1

          src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java ad88b76

          src/main/java/org/apache/hadoop/hbase/master/HMaster.java 3f8f929

          src/main/java/org/apache/hadoop/hbase/security/KerberosInfo.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/security/TokenInfo.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/security/User.java 00bd06d

          src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java bb67e53

          src/main/resources/hbase-default.xml 3785533

          src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java 7194c02

          src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java 3982eff

          Diff: https://reviews.apache.org/r/1991/diff

          Testing

          -------

          A full test suite run showed failures in TestMasterFailover (timeout) and TestAvroServer. I'll look into those.

          Cluster testing (1 master + 3 slaves in ec2) with Kerberos authentication and map reduce token authentication, using YCSB, RowCounter, PerformanceEvaluation.

          Thanks,

          Gary

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1991/#review3198 ----------------------------------------------------------- security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java < https://reviews.apache.org/r/1991/#comment7094 > I don't see this member used. Shall we remove this.authorize ? security/src/main/java/org/apache/hadoop/hbase/security/token/ZKSecretWatcher.java < https://reviews.apache.org/r/1991/#comment7095 > I think we should respond to the IOE beyond logging it. src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java < https://reviews.apache.org/r/1991/#comment7096 > username should also be compared in equals(). Ted On 2011-10-26 20:23:19, Gary Helmling wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1991/ ----------------------------------------------------------- (Updated 2011-10-26 20:23:19) Review request for hbase. Summary ------- This patch creates a new secure RPC engine for HBase, which provides Kerberos based authentication of clients, and a token-based authentication mechanism for mapreduce jobs. Primary components of the patch are: - a new maven profile for secure Hadoop/HBase: hadoop-0.20S - Secure Hadoop dependent classes are separated under a pseudo-module in the security/ directory. These source and test directories are only including if building the secure Hadoop profile - Currently the security classes get packaged with the regular HBase build artifacts. We need a way to at least override project.version, so we can append something like a "-security" suffix indicating the additional security components. - The pseudo-module here is really a half-step forward. It enables the security code to be optionally included in the build for now, and sets up the structure for a security module. But we still will want to pursue full modularization (see HBASE-4336 ), which will allow packing the security code in a separate build artifact. - a new RPC engine providing kerberos and token-based authentication: org.apache.hadoop.hbase.ipc.SecureRpcEngine - implementation under security/src/main/java/org/apache/hadoop/hbase/ipc/ - The implementation classes extend the existing HBaseClient and HBaseServer to share as much of the RPC code as possible. The main override is of the connection classes to allow control over the SASL negotiation of secure connections - existing RPC changes - The existing HBaseClient and HBaseServer have been modified to make subclassing possible - All references to Hadoop UserGroupInformation have been replaced with org.apache.hadoop.hbase.security.User to insulate from future dependencies on specific Hadoop versions - a coprocessor endpoint for obtaining new authentication tokens: TokenProvider, and supporting classes for token generation and synchronization (incorporating HBASE-3615 ) - implementation is under security/src/main/java/org/apache/hadoop/hbase/security/token/ - Secret keys for token generation and verification are synchronized throughout the cluster in zookeeper, under /hbase/tokenauth/keys To enable secure RPC, add the following configuration to hbase-site.xml: <property> <name>hadoop.security.authorization</name> <value>true</value> </property> <property> <name>hadoop.security.authentication</name> <value>kerberos</value> </property> <property> <name>hbase.rpc.engine</name> <value>org.apache.hadoop.hbase.ipc.SecureRpcEngine</value> </property> <property> <name>hbase.coprocessor.region.classes</name> <value>org.apache.hadoop.hbase.security.token.TokenProvider</value> </property> In addition, the master and regionserver processes must be configured for kerberos authentication using the properties: * hbase.(master|regionserver).keytab.file * hbase.(master|regionserver).kerberos.principal * hbase.(master|regionserver).kerberos.https.principal This addresses bug HBASE-2742 . https://issues.apache.org/jira/browse/HBASE-2742 Diffs ----- conf/hbase-policy.xml PRE-CREATION pom.xml 9d42e2b security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/SecureConnectionHeader.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/Status.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationKey.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationProtocol.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenIdentifier.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSelector.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/TokenProvider.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/ZKLeaderManager.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/ZKSecretWatcher.java PRE-CREATION security/src/test/java/org/apache/hadoop/hbase/security/token/TestTokenAuthentication.java PRE-CREATION security/src/test/java/org/apache/hadoop/hbase/security/token/TestZKSecretWatcher.java PRE-CREATION security/src/test/resources/hbase-site.xml PRE-CREATION src/main/java/org/apache/hadoop/hbase/HServerAddress.java f28240c src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 06a2312 src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java 904c107 src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 1365411 src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java 18310d6 src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 337da78 src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 4a8918a src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java 000f99c src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java 965102c src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 1f8b629 src/main/java/org/apache/hadoop/hbase/ipc/RequestContext.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java 1b5629a src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java 60a9248 src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java db07ed1 src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java ad88b76 src/main/java/org/apache/hadoop/hbase/master/HMaster.java 3f8f929 src/main/java/org/apache/hadoop/hbase/security/KerberosInfo.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/security/TokenInfo.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/security/User.java 00bd06d src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java bb67e53 src/main/resources/hbase-default.xml 3785533 src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java 7194c02 src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java 3982eff Diff: https://reviews.apache.org/r/1991/diff Testing ------- A full test suite run showed failures in TestMasterFailover (timeout) and TestAvroServer. I'll look into those. Cluster testing (1 master + 3 slaves in ec2) with Kerberos authentication and map reduce token authentication, using YCSB, RowCounter, PerformanceEvaluation. Thanks, Gary
          Hide
          jiraposter@reviews.apache.org added a comment -

          On 2011-11-13 23:22:58, Ted Yu wrote:

          > security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java, line 317

          > <https://reviews.apache.org/r/1991/diff/4/?file=53579#file53579line317>

          >

          > I don't see this member used.

          > Shall we remove this.authorize ?

          Yes, removed.

          On 2011-11-13 23:22:58, Ted Yu wrote:

          > src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java, line 405

          > <https://reviews.apache.org/r/1991/diff/4/?file=53599#file53599line405>

          >

          > username should also be compared in equals().

          Yes, good catch. Added.

          On 2011-11-13 23:22:58, Ted Yu wrote:

          > security/src/main/java/org/apache/hadoop/hbase/security/token/ZKSecretWatcher.java, line 141

          > <https://reviews.apache.org/r/1991/diff/4/?file=53594#file53594line141>

          >

          > I think we should respond to the IOE beyond logging it.

          What would you suggest? An IOE here means there was a failure in deserializing the key as a Writable, which is bad, but need not be fatal. Some authentication attempts based on the corrupted key may fail, but authentication using other existing keys may still be succeeding. The only other option I can think of is aborting, but that could mean all RS's in the cluster going down, which is pretty bad in itself.

          Any other options you can think of for how we handle this?

          • Gary

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/1991/#review3198
          -----------------------------------------------------------

          On 2011-10-26 20:23:19, Gary Helmling wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/1991/

          -----------------------------------------------------------

          (Updated 2011-10-26 20:23:19)

          Review request for hbase.

          Summary

          -------

          This patch creates a new secure RPC engine for HBase, which provides Kerberos based authentication of clients, and a token-based authentication mechanism for mapreduce jobs. Primary components of the patch are:

          - a new maven profile for secure Hadoop/HBase: hadoop-0.20S

          - Secure Hadoop dependent classes are separated under a pseudo-module in the security/ directory. These source and test directories are only including if building the secure Hadoop profile

          - Currently the security classes get packaged with the regular HBase build artifacts. We need a way to at least override project.version, so we can append something like a "-security" suffix indicating the additional security components.

          - The pseudo-module here is really a half-step forward. It enables the security code to be optionally included in the build for now, and sets up the structure for a security module. But we still will want to pursue full modularization (see HBASE-4336), which will allow packing the security code in a separate build artifact.

          - a new RPC engine providing kerberos and token-based authentication: org.apache.hadoop.hbase.ipc.SecureRpcEngine

          - implementation under security/src/main/java/org/apache/hadoop/hbase/ipc/

          - The implementation classes extend the existing HBaseClient and HBaseServer to share as much of the RPC code as possible. The main override is of the connection classes to allow control over the SASL negotiation of secure connections

          - existing RPC changes

          - The existing HBaseClient and HBaseServer have been modified to make subclassing possible

          - All references to Hadoop UserGroupInformation have been replaced with org.apache.hadoop.hbase.security.User to insulate from future dependencies on specific Hadoop versions

          - a coprocessor endpoint for obtaining new authentication tokens: TokenProvider, and supporting classes for token generation and synchronization (incorporating HBASE-3615)

          - implementation is under security/src/main/java/org/apache/hadoop/hbase/security/token/

          - Secret keys for token generation and verification are synchronized throughout the cluster in zookeeper, under /hbase/tokenauth/keys

          To enable secure RPC, add the following configuration to hbase-site.xml:

          <property>

          <name>hadoop.security.authorization</name>

          <value>true</value>

          </property>

          <property>

          <name>hadoop.security.authentication</name>

          <value>kerberos</value>

          </property>

          <property>

          <name>hbase.rpc.engine</name>

          <value>org.apache.hadoop.hbase.ipc.SecureRpcEngine</value>

          </property>

          <property>

          <name>hbase.coprocessor.region.classes</name>

          <value>org.apache.hadoop.hbase.security.token.TokenProvider</value>

          </property>

          In addition, the master and regionserver processes must be configured for kerberos authentication using the properties:

          * hbase.(master|regionserver).keytab.file

          * hbase.(master|regionserver).kerberos.principal

          * hbase.(master|regionserver).kerberos.https.principal

          This addresses bug HBASE-2742.

          https://issues.apache.org/jira/browse/HBASE-2742

          Diffs

          -----

          conf/hbase-policy.xml PRE-CREATION

          pom.xml 9d42e2b

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureConnectionHeader.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/ipc/Status.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationKey.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationProtocol.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenIdentifier.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSelector.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/TokenProvider.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/ZKLeaderManager.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/ZKSecretWatcher.java PRE-CREATION

          security/src/test/java/org/apache/hadoop/hbase/security/token/TestTokenAuthentication.java PRE-CREATION

          security/src/test/java/org/apache/hadoop/hbase/security/token/TestZKSecretWatcher.java PRE-CREATION

          security/src/test/resources/hbase-site.xml PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/HServerAddress.java f28240c

          src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 06a2312

          src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java 904c107

          src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 1365411

          src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java 18310d6

          src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 337da78

          src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 4a8918a

          src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java 000f99c

          src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java 965102c

          src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 1f8b629

          src/main/java/org/apache/hadoop/hbase/ipc/RequestContext.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java 1b5629a

          src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java 60a9248

          src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java db07ed1

          src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java ad88b76

          src/main/java/org/apache/hadoop/hbase/master/HMaster.java 3f8f929

          src/main/java/org/apache/hadoop/hbase/security/KerberosInfo.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/security/TokenInfo.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/security/User.java 00bd06d

          src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java bb67e53

          src/main/resources/hbase-default.xml 3785533

          src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java 7194c02

          src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java 3982eff

          Diff: https://reviews.apache.org/r/1991/diff

          Testing

          -------

          A full test suite run showed failures in TestMasterFailover (timeout) and TestAvroServer. I'll look into those.

          Cluster testing (1 master + 3 slaves in ec2) with Kerberos authentication and map reduce token authentication, using YCSB, RowCounter, PerformanceEvaluation.

          Thanks,

          Gary

          Show
          jiraposter@reviews.apache.org added a comment - On 2011-11-13 23:22:58, Ted Yu wrote: > security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java, line 317 > < https://reviews.apache.org/r/1991/diff/4/?file=53579#file53579line317 > > > I don't see this member used. > Shall we remove this.authorize ? Yes, removed. On 2011-11-13 23:22:58, Ted Yu wrote: > src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java, line 405 > < https://reviews.apache.org/r/1991/diff/4/?file=53599#file53599line405 > > > username should also be compared in equals(). Yes, good catch. Added. On 2011-11-13 23:22:58, Ted Yu wrote: > security/src/main/java/org/apache/hadoop/hbase/security/token/ZKSecretWatcher.java, line 141 > < https://reviews.apache.org/r/1991/diff/4/?file=53594#file53594line141 > > > I think we should respond to the IOE beyond logging it. What would you suggest? An IOE here means there was a failure in deserializing the key as a Writable, which is bad, but need not be fatal. Some authentication attempts based on the corrupted key may fail, but authentication using other existing keys may still be succeeding. The only other option I can think of is aborting, but that could mean all RS's in the cluster going down, which is pretty bad in itself. Any other options you can think of for how we handle this? Gary ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1991/#review3198 ----------------------------------------------------------- On 2011-10-26 20:23:19, Gary Helmling wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1991/ ----------------------------------------------------------- (Updated 2011-10-26 20:23:19) Review request for hbase. Summary ------- This patch creates a new secure RPC engine for HBase, which provides Kerberos based authentication of clients, and a token-based authentication mechanism for mapreduce jobs. Primary components of the patch are: - a new maven profile for secure Hadoop/HBase: hadoop-0.20S - Secure Hadoop dependent classes are separated under a pseudo-module in the security/ directory. These source and test directories are only including if building the secure Hadoop profile - Currently the security classes get packaged with the regular HBase build artifacts. We need a way to at least override project.version, so we can append something like a "-security" suffix indicating the additional security components. - The pseudo-module here is really a half-step forward. It enables the security code to be optionally included in the build for now, and sets up the structure for a security module. But we still will want to pursue full modularization (see HBASE-4336 ), which will allow packing the security code in a separate build artifact. - a new RPC engine providing kerberos and token-based authentication: org.apache.hadoop.hbase.ipc.SecureRpcEngine - implementation under security/src/main/java/org/apache/hadoop/hbase/ipc/ - The implementation classes extend the existing HBaseClient and HBaseServer to share as much of the RPC code as possible. The main override is of the connection classes to allow control over the SASL negotiation of secure connections - existing RPC changes - The existing HBaseClient and HBaseServer have been modified to make subclassing possible - All references to Hadoop UserGroupInformation have been replaced with org.apache.hadoop.hbase.security.User to insulate from future dependencies on specific Hadoop versions - a coprocessor endpoint for obtaining new authentication tokens: TokenProvider, and supporting classes for token generation and synchronization (incorporating HBASE-3615 ) - implementation is under security/src/main/java/org/apache/hadoop/hbase/security/token/ - Secret keys for token generation and verification are synchronized throughout the cluster in zookeeper, under /hbase/tokenauth/keys To enable secure RPC, add the following configuration to hbase-site.xml: <property> <name>hadoop.security.authorization</name> <value>true</value> </property> <property> <name>hadoop.security.authentication</name> <value>kerberos</value> </property> <property> <name>hbase.rpc.engine</name> <value>org.apache.hadoop.hbase.ipc.SecureRpcEngine</value> </property> <property> <name>hbase.coprocessor.region.classes</name> <value>org.apache.hadoop.hbase.security.token.TokenProvider</value> </property> In addition, the master and regionserver processes must be configured for kerberos authentication using the properties: * hbase.(master|regionserver).keytab.file * hbase.(master|regionserver).kerberos.principal * hbase.(master|regionserver).kerberos.https.principal This addresses bug HBASE-2742 . https://issues.apache.org/jira/browse/HBASE-2742 Diffs ----- conf/hbase-policy.xml PRE-CREATION pom.xml 9d42e2b security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/SecureConnectionHeader.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/Status.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationKey.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationProtocol.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenIdentifier.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSelector.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/TokenProvider.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/ZKLeaderManager.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/ZKSecretWatcher.java PRE-CREATION security/src/test/java/org/apache/hadoop/hbase/security/token/TestTokenAuthentication.java PRE-CREATION security/src/test/java/org/apache/hadoop/hbase/security/token/TestZKSecretWatcher.java PRE-CREATION security/src/test/resources/hbase-site.xml PRE-CREATION src/main/java/org/apache/hadoop/hbase/HServerAddress.java f28240c src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 06a2312 src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java 904c107 src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 1365411 src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java 18310d6 src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 337da78 src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 4a8918a src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java 000f99c src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java 965102c src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 1f8b629 src/main/java/org/apache/hadoop/hbase/ipc/RequestContext.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java 1b5629a src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java 60a9248 src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java db07ed1 src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java ad88b76 src/main/java/org/apache/hadoop/hbase/master/HMaster.java 3f8f929 src/main/java/org/apache/hadoop/hbase/security/KerberosInfo.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/security/TokenInfo.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/security/User.java 00bd06d src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java bb67e53 src/main/resources/hbase-default.xml 3785533 src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java 7194c02 src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java 3982eff Diff: https://reviews.apache.org/r/1991/diff Testing ------- A full test suite run showed failures in TestMasterFailover (timeout) and TestAvroServer. I'll look into those. Cluster testing (1 master + 3 slaves in ec2) with Kerberos authentication and map reduce token authentication, using YCSB, RowCounter, PerformanceEvaluation. Thanks, Gary
          Hide
          Ted Yu added a comment - - edited

          For ZKSecretWatcher.refreshNodes(), can we remove the zk node corresponding to path whose deserialization resulted in IOE ?

          Show
          Ted Yu added a comment - - edited For ZKSecretWatcher.refreshNodes(), can we remove the zk node corresponding to path whose deserialization resulted in IOE ?
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/1991/
          -----------------------------------------------------------

          (Updated 2011-11-14 20:03:17.125142)

          Review request for hbase.

          Changes
          -------

          This update addresses the current review comments.

          The only comment not addressed is Ted's suggestion in ZKSecretWatcher to attempt to remove the znode for any secret key that could not be deserialized. I need to look at the implications of this for the "leader" secret manager, as well as for potential upgrade scenarios.

          Summary
          -------

          This patch creates a new secure RPC engine for HBase, which provides Kerberos based authentication of clients, and a token-based authentication mechanism for mapreduce jobs. Primary components of the patch are:

          • a new maven profile for secure Hadoop/HBase: hadoop-0.20S
          • Secure Hadoop dependent classes are separated under a pseudo-module in the security/ directory. These source and test directories are only including if building the secure Hadoop profile
          • Currently the security classes get packaged with the regular HBase build artifacts. We need a way to at least override project.version, so we can append something like a "-security" suffix indicating the additional security components.
          • The pseudo-module here is really a half-step forward. It enables the security code to be optionally included in the build for now, and sets up the structure for a security module. But we still will want to pursue full modularization (see HBASE-4336), which will allow packing the security code in a separate build artifact.
          • a new RPC engine providing kerberos and token-based authentication: org.apache.hadoop.hbase.ipc.SecureRpcEngine
          • implementation under security/src/main/java/org/apache/hadoop/hbase/ipc/
          • The implementation classes extend the existing HBaseClient and HBaseServer to share as much of the RPC code as possible. The main override is of the connection classes to allow control over the SASL negotiation of secure connections
          • existing RPC changes
          • The existing HBaseClient and HBaseServer have been modified to make subclassing possible
          • All references to Hadoop UserGroupInformation have been replaced with org.apache.hadoop.hbase.security.User to insulate from future dependencies on specific Hadoop versions
          • a coprocessor endpoint for obtaining new authentication tokens: TokenProvider, and supporting classes for token generation and synchronization (incorporating HBASE-3615)
          • implementation is under security/src/main/java/org/apache/hadoop/hbase/security/token/
          • Secret keys for token generation and verification are synchronized throughout the cluster in zookeeper, under /hbase/tokenauth/keys

          To enable secure RPC, add the following configuration to hbase-site.xml:

          <property>
          <name>hadoop.security.authorization</name>
          <value>true</value>
          </property>
          <property>
          <name>hadoop.security.authentication</name>
          <value>kerberos</value>
          </property>
          <property>
          <name>hbase.rpc.engine</name>
          <value>org.apache.hadoop.hbase.ipc.SecureRpcEngine</value>
          </property>
          <property>
          <name>hbase.coprocessor.region.classes</name>
          <value>org.apache.hadoop.hbase.security.token.TokenProvider</value>
          </property>

          In addition, the master and regionserver processes must be configured for kerberos authentication using the properties:

          • hbase.(master|regionserver).keytab.file
          • hbase.(master|regionserver).kerberos.principal
          • hbase.(master|regionserver).kerberos.https.principal

          This addresses bug HBASE-2742.
          https://issues.apache.org/jira/browse/HBASE-2742

          Diffs (updated)


          conf/hbase-policy.xml PRE-CREATION
          pom.xml 2847416
          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureConnectionHeader.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationKey.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationProtocol.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenIdentifier.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSelector.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/token/TokenProvider.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/token/ZKSecretWatcher.java PRE-CREATION
          security/src/test/java/org/apache/hadoop/hbase/security/token/TestTokenAuthentication.java PRE-CREATION
          security/src/test/java/org/apache/hadoop/hbase/security/token/TestZKSecretWatcher.java PRE-CREATION
          security/src/test/resources/hbase-site.xml PRE-CREATION
          src/assembly/all.xml 3ad8ab3
          src/main/java/org/apache/hadoop/hbase/HServerAddress.java f28240c
          src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java cba7bd1
          src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java 904c107
          src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 4086829
          src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java 054b92b
          src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 337da78
          src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 9117f12
          src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java 9e0535e
          src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java 0dc5bea
          src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java e8b2c9e
          src/main/java/org/apache/hadoop/hbase/ipc/RequestContext.java PRE-CREATION
          src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java 1b5629a
          src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java c8e030c
          src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java db07ed1
          src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java ad88b76
          src/main/java/org/apache/hadoop/hbase/master/HMaster.java 19ed5b8
          src/main/java/org/apache/hadoop/hbase/security/KerberosInfo.java PRE-CREATION
          src/main/java/org/apache/hadoop/hbase/security/TokenInfo.java PRE-CREATION
          src/main/java/org/apache/hadoop/hbase/security/User.java 00bd06d
          src/main/java/org/apache/hadoop/hbase/zookeeper/ZKLeaderManager.java PRE-CREATION
          src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java 4c5e844
          src/main/resources/hbase-default.xml 6f98f5d
          src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java c0634f4
          src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java bb9ea45

          Diff: https://reviews.apache.org/r/1991/diff

          Testing
          -------

          A full test suite run showed failures in TestMasterFailover (timeout) and TestAvroServer. I'll look into those.

          Cluster testing (1 master + 3 slaves in ec2) with Kerberos authentication and map reduce token authentication, using YCSB, RowCounter, PerformanceEvaluation.

          Thanks,

          Gary

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1991/ ----------------------------------------------------------- (Updated 2011-11-14 20:03:17.125142) Review request for hbase. Changes ------- This update addresses the current review comments. The only comment not addressed is Ted's suggestion in ZKSecretWatcher to attempt to remove the znode for any secret key that could not be deserialized. I need to look at the implications of this for the "leader" secret manager, as well as for potential upgrade scenarios. Summary ------- This patch creates a new secure RPC engine for HBase, which provides Kerberos based authentication of clients, and a token-based authentication mechanism for mapreduce jobs. Primary components of the patch are: a new maven profile for secure Hadoop/HBase: hadoop-0.20S Secure Hadoop dependent classes are separated under a pseudo-module in the security/ directory. These source and test directories are only including if building the secure Hadoop profile Currently the security classes get packaged with the regular HBase build artifacts. We need a way to at least override project.version, so we can append something like a "-security" suffix indicating the additional security components. The pseudo-module here is really a half-step forward. It enables the security code to be optionally included in the build for now, and sets up the structure for a security module. But we still will want to pursue full modularization (see HBASE-4336 ), which will allow packing the security code in a separate build artifact. a new RPC engine providing kerberos and token-based authentication: org.apache.hadoop.hbase.ipc.SecureRpcEngine implementation under security/src/main/java/org/apache/hadoop/hbase/ipc/ The implementation classes extend the existing HBaseClient and HBaseServer to share as much of the RPC code as possible. The main override is of the connection classes to allow control over the SASL negotiation of secure connections existing RPC changes The existing HBaseClient and HBaseServer have been modified to make subclassing possible All references to Hadoop UserGroupInformation have been replaced with org.apache.hadoop.hbase.security.User to insulate from future dependencies on specific Hadoop versions a coprocessor endpoint for obtaining new authentication tokens: TokenProvider, and supporting classes for token generation and synchronization (incorporating HBASE-3615 ) implementation is under security/src/main/java/org/apache/hadoop/hbase/security/token/ Secret keys for token generation and verification are synchronized throughout the cluster in zookeeper, under /hbase/tokenauth/keys To enable secure RPC, add the following configuration to hbase-site.xml: <property> <name>hadoop.security.authorization</name> <value>true</value> </property> <property> <name>hadoop.security.authentication</name> <value>kerberos</value> </property> <property> <name>hbase.rpc.engine</name> <value>org.apache.hadoop.hbase.ipc.SecureRpcEngine</value> </property> <property> <name>hbase.coprocessor.region.classes</name> <value>org.apache.hadoop.hbase.security.token.TokenProvider</value> </property> In addition, the master and regionserver processes must be configured for kerberos authentication using the properties: hbase.(master|regionserver).keytab.file hbase.(master|regionserver).kerberos.principal hbase.(master|regionserver).kerberos.https.principal This addresses bug HBASE-2742 . https://issues.apache.org/jira/browse/HBASE-2742 Diffs (updated) conf/hbase-policy.xml PRE-CREATION pom.xml 2847416 security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/SecureConnectionHeader.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationKey.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationProtocol.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenIdentifier.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSelector.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/TokenProvider.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/ZKSecretWatcher.java PRE-CREATION security/src/test/java/org/apache/hadoop/hbase/security/token/TestTokenAuthentication.java PRE-CREATION security/src/test/java/org/apache/hadoop/hbase/security/token/TestZKSecretWatcher.java PRE-CREATION security/src/test/resources/hbase-site.xml PRE-CREATION src/assembly/all.xml 3ad8ab3 src/main/java/org/apache/hadoop/hbase/HServerAddress.java f28240c src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java cba7bd1 src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java 904c107 src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 4086829 src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java 054b92b src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 337da78 src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 9117f12 src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java 9e0535e src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java 0dc5bea src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java e8b2c9e src/main/java/org/apache/hadoop/hbase/ipc/RequestContext.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java 1b5629a src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java c8e030c src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java db07ed1 src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java ad88b76 src/main/java/org/apache/hadoop/hbase/master/HMaster.java 19ed5b8 src/main/java/org/apache/hadoop/hbase/security/KerberosInfo.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/security/TokenInfo.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/security/User.java 00bd06d src/main/java/org/apache/hadoop/hbase/zookeeper/ZKLeaderManager.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java 4c5e844 src/main/resources/hbase-default.xml 6f98f5d src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java c0634f4 src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java bb9ea45 Diff: https://reviews.apache.org/r/1991/diff Testing ------- A full test suite run showed failures in TestMasterFailover (timeout) and TestAvroServer. I'll look into those. Cluster testing (1 master + 3 slaves in ec2) with Kerberos authentication and map reduce token authentication, using YCSB, RowCounter, PerformanceEvaluation. Thanks, Gary
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/1991/#review3235
          -----------------------------------------------------------

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java
          <https://reviews.apache.org/r/1991/#comment7204>

          Should we use org.apache.hadoop.hbase.ipc.SecureClient ?

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java
          <https://reviews.apache.org/r/1991/#comment7203>

          Should read 'when running as master'

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java
          <https://reviews.apache.org/r/1991/#comment7202>

          This log should be guarded by LOG.isDebugEnabled()

          src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
          <https://reviews.apache.org/r/1991/#comment7231>

          Please add curly braces around the log statement.

          src/main/java/org/apache/hadoop/hbase/zookeeper/ZKLeaderManager.java
          <https://reviews.apache.org/r/1991/#comment7227>

          Can we rewrite using a loop instead of recursion ?

          • Ted

          On 2011-11-14 20:03:17, Gary Helmling wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/1991/

          -----------------------------------------------------------

          (Updated 2011-11-14 20:03:17)

          Review request for hbase.

          Summary

          -------

          This patch creates a new secure RPC engine for HBase, which provides Kerberos based authentication of clients, and a token-based authentication mechanism for mapreduce jobs. Primary components of the patch are:

          - a new maven profile for secure Hadoop/HBase: hadoop-0.20S

          - Secure Hadoop dependent classes are separated under a pseudo-module in the security/ directory. These source and test directories are only including if building the secure Hadoop profile

          - Currently the security classes get packaged with the regular HBase build artifacts. We need a way to at least override project.version, so we can append something like a "-security" suffix indicating the additional security components.

          - The pseudo-module here is really a half-step forward. It enables the security code to be optionally included in the build for now, and sets up the structure for a security module. But we still will want to pursue full modularization (see HBASE-4336), which will allow packing the security code in a separate build artifact.

          - a new RPC engine providing kerberos and token-based authentication: org.apache.hadoop.hbase.ipc.SecureRpcEngine

          - implementation under security/src/main/java/org/apache/hadoop/hbase/ipc/

          - The implementation classes extend the existing HBaseClient and HBaseServer to share as much of the RPC code as possible. The main override is of the connection classes to allow control over the SASL negotiation of secure connections

          - existing RPC changes

          - The existing HBaseClient and HBaseServer have been modified to make subclassing possible

          - All references to Hadoop UserGroupInformation have been replaced with org.apache.hadoop.hbase.security.User to insulate from future dependencies on specific Hadoop versions

          - a coprocessor endpoint for obtaining new authentication tokens: TokenProvider, and supporting classes for token generation and synchronization (incorporating HBASE-3615)

          - implementation is under security/src/main/java/org/apache/hadoop/hbase/security/token/

          - Secret keys for token generation and verification are synchronized throughout the cluster in zookeeper, under /hbase/tokenauth/keys

          To enable secure RPC, add the following configuration to hbase-site.xml:

          <property>

          <name>hadoop.security.authorization</name>

          <value>true</value>

          </property>

          <property>

          <name>hadoop.security.authentication</name>

          <value>kerberos</value>

          </property>

          <property>

          <name>hbase.rpc.engine</name>

          <value>org.apache.hadoop.hbase.ipc.SecureRpcEngine</value>

          </property>

          <property>

          <name>hbase.coprocessor.region.classes</name>

          <value>org.apache.hadoop.hbase.security.token.TokenProvider</value>

          </property>

          In addition, the master and regionserver processes must be configured for kerberos authentication using the properties:

          * hbase.(master|regionserver).keytab.file

          * hbase.(master|regionserver).kerberos.principal

          * hbase.(master|regionserver).kerberos.https.principal

          This addresses bug HBASE-2742.

          https://issues.apache.org/jira/browse/HBASE-2742

          Diffs

          -----

          conf/hbase-policy.xml PRE-CREATION

          pom.xml 2847416

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureConnectionHeader.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationKey.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationProtocol.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenIdentifier.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSelector.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/TokenProvider.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/ZKSecretWatcher.java PRE-CREATION

          security/src/test/java/org/apache/hadoop/hbase/security/token/TestTokenAuthentication.java PRE-CREATION

          security/src/test/java/org/apache/hadoop/hbase/security/token/TestZKSecretWatcher.java PRE-CREATION

          security/src/test/resources/hbase-site.xml PRE-CREATION

          src/assembly/all.xml 3ad8ab3

          src/main/java/org/apache/hadoop/hbase/HServerAddress.java f28240c

          src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java cba7bd1

          src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java 904c107

          src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 4086829

          src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java 054b92b

          src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 337da78

          src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 9117f12

          src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java 9e0535e

          src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java 0dc5bea

          src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java e8b2c9e

          src/main/java/org/apache/hadoop/hbase/ipc/RequestContext.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java 1b5629a

          src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java c8e030c

          src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java db07ed1

          src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java ad88b76

          src/main/java/org/apache/hadoop/hbase/master/HMaster.java 19ed5b8

          src/main/java/org/apache/hadoop/hbase/security/KerberosInfo.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/security/TokenInfo.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/security/User.java 00bd06d

          src/main/java/org/apache/hadoop/hbase/zookeeper/ZKLeaderManager.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java 4c5e844

          src/main/resources/hbase-default.xml 6f98f5d

          src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java c0634f4

          src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java bb9ea45

          Diff: https://reviews.apache.org/r/1991/diff

          Testing

          -------

          A full test suite run showed failures in TestMasterFailover (timeout) and TestAvroServer. I'll look into those.

          Cluster testing (1 master + 3 slaves in ec2) with Kerberos authentication and map reduce token authentication, using YCSB, RowCounter, PerformanceEvaluation.

          Thanks,

          Gary

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1991/#review3235 ----------------------------------------------------------- security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java < https://reviews.apache.org/r/1991/#comment7204 > Should we use org.apache.hadoop.hbase.ipc.SecureClient ? security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java < https://reviews.apache.org/r/1991/#comment7203 > Should read 'when running as master' security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java < https://reviews.apache.org/r/1991/#comment7202 > This log should be guarded by LOG.isDebugEnabled() src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java < https://reviews.apache.org/r/1991/#comment7231 > Please add curly braces around the log statement. src/main/java/org/apache/hadoop/hbase/zookeeper/ZKLeaderManager.java < https://reviews.apache.org/r/1991/#comment7227 > Can we rewrite using a loop instead of recursion ? Ted On 2011-11-14 20:03:17, Gary Helmling wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1991/ ----------------------------------------------------------- (Updated 2011-11-14 20:03:17) Review request for hbase. Summary ------- This patch creates a new secure RPC engine for HBase, which provides Kerberos based authentication of clients, and a token-based authentication mechanism for mapreduce jobs. Primary components of the patch are: - a new maven profile for secure Hadoop/HBase: hadoop-0.20S - Secure Hadoop dependent classes are separated under a pseudo-module in the security/ directory. These source and test directories are only including if building the secure Hadoop profile - Currently the security classes get packaged with the regular HBase build artifacts. We need a way to at least override project.version, so we can append something like a "-security" suffix indicating the additional security components. - The pseudo-module here is really a half-step forward. It enables the security code to be optionally included in the build for now, and sets up the structure for a security module. But we still will want to pursue full modularization (see HBASE-4336 ), which will allow packing the security code in a separate build artifact. - a new RPC engine providing kerberos and token-based authentication: org.apache.hadoop.hbase.ipc.SecureRpcEngine - implementation under security/src/main/java/org/apache/hadoop/hbase/ipc/ - The implementation classes extend the existing HBaseClient and HBaseServer to share as much of the RPC code as possible. The main override is of the connection classes to allow control over the SASL negotiation of secure connections - existing RPC changes - The existing HBaseClient and HBaseServer have been modified to make subclassing possible - All references to Hadoop UserGroupInformation have been replaced with org.apache.hadoop.hbase.security.User to insulate from future dependencies on specific Hadoop versions - a coprocessor endpoint for obtaining new authentication tokens: TokenProvider, and supporting classes for token generation and synchronization (incorporating HBASE-3615 ) - implementation is under security/src/main/java/org/apache/hadoop/hbase/security/token/ - Secret keys for token generation and verification are synchronized throughout the cluster in zookeeper, under /hbase/tokenauth/keys To enable secure RPC, add the following configuration to hbase-site.xml: <property> <name>hadoop.security.authorization</name> <value>true</value> </property> <property> <name>hadoop.security.authentication</name> <value>kerberos</value> </property> <property> <name>hbase.rpc.engine</name> <value>org.apache.hadoop.hbase.ipc.SecureRpcEngine</value> </property> <property> <name>hbase.coprocessor.region.classes</name> <value>org.apache.hadoop.hbase.security.token.TokenProvider</value> </property> In addition, the master and regionserver processes must be configured for kerberos authentication using the properties: * hbase.(master|regionserver).keytab.file * hbase.(master|regionserver).kerberos.principal * hbase.(master|regionserver).kerberos.https.principal This addresses bug HBASE-2742 . https://issues.apache.org/jira/browse/HBASE-2742 Diffs ----- conf/hbase-policy.xml PRE-CREATION pom.xml 2847416 security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/SecureConnectionHeader.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationKey.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationProtocol.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenIdentifier.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSelector.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/TokenProvider.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/ZKSecretWatcher.java PRE-CREATION security/src/test/java/org/apache/hadoop/hbase/security/token/TestTokenAuthentication.java PRE-CREATION security/src/test/java/org/apache/hadoop/hbase/security/token/TestZKSecretWatcher.java PRE-CREATION security/src/test/resources/hbase-site.xml PRE-CREATION src/assembly/all.xml 3ad8ab3 src/main/java/org/apache/hadoop/hbase/HServerAddress.java f28240c src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java cba7bd1 src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java 904c107 src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 4086829 src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java 054b92b src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 337da78 src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 9117f12 src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java 9e0535e src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java 0dc5bea src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java e8b2c9e src/main/java/org/apache/hadoop/hbase/ipc/RequestContext.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java 1b5629a src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java c8e030c src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java db07ed1 src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java ad88b76 src/main/java/org/apache/hadoop/hbase/master/HMaster.java 19ed5b8 src/main/java/org/apache/hadoop/hbase/security/KerberosInfo.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/security/TokenInfo.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/security/User.java 00bd06d src/main/java/org/apache/hadoop/hbase/zookeeper/ZKLeaderManager.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java 4c5e844 src/main/resources/hbase-default.xml 6f98f5d src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java c0634f4 src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java bb9ea45 Diff: https://reviews.apache.org/r/1991/diff Testing ------- A full test suite run showed failures in TestMasterFailover (timeout) and TestAvroServer. I'll look into those. Cluster testing (1 master + 3 slaves in ec2) with Kerberos authentication and map reduce token authentication, using YCSB, RowCounter, PerformanceEvaluation. Thanks, Gary
          Hide
          jiraposter@reviews.apache.org added a comment -

          On 2011-11-14 22:36:05, Ted Yu wrote:

          > security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java, line 72

          > <https://reviews.apache.org/r/1991/diff/5/?file=57988#file57988line72>

          >

          > Should we use org.apache.hadoop.hbase.ipc.SecureClient ?

          This follows the same logging convention as HBaseClient/HBaseServer – it uses a mangled package name so that we don't get noisy IPC debug logging when setting org.apache.hadoop.hbase=DEBUG in log4j config.

          It's a bit weird, but it works. Since log4j does actually support "trace" level now, it might be better to correct the package names and use trace level for all the noisy (per request) logging. This would keep debug logging relatively clean without the package mangling. But I think we should tackle that as a separate issue.

          On 2011-11-14 22:36:05, Ted Yu wrote:

          > src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java, line 1229

          > <https://reviews.apache.org/r/1991/diff/5/?file=58014#file58014line1229>

          >

          > Please add curly braces around the log statement.

          Since this is pre-existing code copied from Hadoop, style changes here will just make it more difficult to merge Hadoop RPC updates in the future, but maybe we're already beyond that...

          On 2011-11-14 22:36:05, Ted Yu wrote:

          > src/main/java/org/apache/hadoop/hbase/zookeeper/ZKLeaderManager.java, line 147

          > <https://reviews.apache.org/r/1991/diff/5/?file=58027#file58027line147>

          >

          > Can we rewrite using a loop instead of recursion ?

          Done, with additional test.

          • Gary

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/1991/#review3235
          -----------------------------------------------------------

          On 2011-11-14 20:03:17, Gary Helmling wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/1991/

          -----------------------------------------------------------

          (Updated 2011-11-14 20:03:17)

          Review request for hbase.

          Summary

          -------

          This patch creates a new secure RPC engine for HBase, which provides Kerberos based authentication of clients, and a token-based authentication mechanism for mapreduce jobs. Primary components of the patch are:

          - a new maven profile for secure Hadoop/HBase: hadoop-0.20S

          - Secure Hadoop dependent classes are separated under a pseudo-module in the security/ directory. These source and test directories are only including if building the secure Hadoop profile

          - Currently the security classes get packaged with the regular HBase build artifacts. We need a way to at least override project.version, so we can append something like a "-security" suffix indicating the additional security components.

          - The pseudo-module here is really a half-step forward. It enables the security code to be optionally included in the build for now, and sets up the structure for a security module. But we still will want to pursue full modularization (see HBASE-4336), which will allow packing the security code in a separate build artifact.

          - a new RPC engine providing kerberos and token-based authentication: org.apache.hadoop.hbase.ipc.SecureRpcEngine

          - implementation under security/src/main/java/org/apache/hadoop/hbase/ipc/

          - The implementation classes extend the existing HBaseClient and HBaseServer to share as much of the RPC code as possible. The main override is of the connection classes to allow control over the SASL negotiation of secure connections

          - existing RPC changes

          - The existing HBaseClient and HBaseServer have been modified to make subclassing possible

          - All references to Hadoop UserGroupInformation have been replaced with org.apache.hadoop.hbase.security.User to insulate from future dependencies on specific Hadoop versions

          - a coprocessor endpoint for obtaining new authentication tokens: TokenProvider, and supporting classes for token generation and synchronization (incorporating HBASE-3615)

          - implementation is under security/src/main/java/org/apache/hadoop/hbase/security/token/

          - Secret keys for token generation and verification are synchronized throughout the cluster in zookeeper, under /hbase/tokenauth/keys

          To enable secure RPC, add the following configuration to hbase-site.xml:

          <property>

          <name>hadoop.security.authorization</name>

          <value>true</value>

          </property>

          <property>

          <name>hadoop.security.authentication</name>

          <value>kerberos</value>

          </property>

          <property>

          <name>hbase.rpc.engine</name>

          <value>org.apache.hadoop.hbase.ipc.SecureRpcEngine</value>

          </property>

          <property>

          <name>hbase.coprocessor.region.classes</name>

          <value>org.apache.hadoop.hbase.security.token.TokenProvider</value>

          </property>

          In addition, the master and regionserver processes must be configured for kerberos authentication using the properties:

          * hbase.(master|regionserver).keytab.file

          * hbase.(master|regionserver).kerberos.principal

          * hbase.(master|regionserver).kerberos.https.principal

          This addresses bug HBASE-2742.

          https://issues.apache.org/jira/browse/HBASE-2742

          Diffs

          -----

          conf/hbase-policy.xml PRE-CREATION

          pom.xml 2847416

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureConnectionHeader.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationKey.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationProtocol.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenIdentifier.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSelector.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/TokenProvider.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/ZKSecretWatcher.java PRE-CREATION

          security/src/test/java/org/apache/hadoop/hbase/security/token/TestTokenAuthentication.java PRE-CREATION

          security/src/test/java/org/apache/hadoop/hbase/security/token/TestZKSecretWatcher.java PRE-CREATION

          security/src/test/resources/hbase-site.xml PRE-CREATION

          src/assembly/all.xml 3ad8ab3

          src/main/java/org/apache/hadoop/hbase/HServerAddress.java f28240c

          src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java cba7bd1

          src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java 904c107

          src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 4086829

          src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java 054b92b

          src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 337da78

          src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 9117f12

          src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java 9e0535e

          src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java 0dc5bea

          src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java e8b2c9e

          src/main/java/org/apache/hadoop/hbase/ipc/RequestContext.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java 1b5629a

          src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java c8e030c

          src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java db07ed1

          src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java ad88b76

          src/main/java/org/apache/hadoop/hbase/master/HMaster.java 19ed5b8

          src/main/java/org/apache/hadoop/hbase/security/KerberosInfo.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/security/TokenInfo.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/security/User.java 00bd06d

          src/main/java/org/apache/hadoop/hbase/zookeeper/ZKLeaderManager.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java 4c5e844

          src/main/resources/hbase-default.xml 6f98f5d

          src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java c0634f4

          src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java bb9ea45

          Diff: https://reviews.apache.org/r/1991/diff

          Testing

          -------

          A full test suite run showed failures in TestMasterFailover (timeout) and TestAvroServer. I'll look into those.

          Cluster testing (1 master + 3 slaves in ec2) with Kerberos authentication and map reduce token authentication, using YCSB, RowCounter, PerformanceEvaluation.

          Thanks,

          Gary

          Show
          jiraposter@reviews.apache.org added a comment - On 2011-11-14 22:36:05, Ted Yu wrote: > security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java, line 72 > < https://reviews.apache.org/r/1991/diff/5/?file=57988#file57988line72 > > > Should we use org.apache.hadoop.hbase.ipc.SecureClient ? This follows the same logging convention as HBaseClient/HBaseServer – it uses a mangled package name so that we don't get noisy IPC debug logging when setting org.apache.hadoop.hbase=DEBUG in log4j config. It's a bit weird, but it works. Since log4j does actually support "trace" level now, it might be better to correct the package names and use trace level for all the noisy (per request) logging. This would keep debug logging relatively clean without the package mangling. But I think we should tackle that as a separate issue. On 2011-11-14 22:36:05, Ted Yu wrote: > src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java, line 1229 > < https://reviews.apache.org/r/1991/diff/5/?file=58014#file58014line1229 > > > Please add curly braces around the log statement. Since this is pre-existing code copied from Hadoop, style changes here will just make it more difficult to merge Hadoop RPC updates in the future, but maybe we're already beyond that... On 2011-11-14 22:36:05, Ted Yu wrote: > src/main/java/org/apache/hadoop/hbase/zookeeper/ZKLeaderManager.java, line 147 > < https://reviews.apache.org/r/1991/diff/5/?file=58027#file58027line147 > > > Can we rewrite using a loop instead of recursion ? Done, with additional test. Gary ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1991/#review3235 ----------------------------------------------------------- On 2011-11-14 20:03:17, Gary Helmling wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1991/ ----------------------------------------------------------- (Updated 2011-11-14 20:03:17) Review request for hbase. Summary ------- This patch creates a new secure RPC engine for HBase, which provides Kerberos based authentication of clients, and a token-based authentication mechanism for mapreduce jobs. Primary components of the patch are: - a new maven profile for secure Hadoop/HBase: hadoop-0.20S - Secure Hadoop dependent classes are separated under a pseudo-module in the security/ directory. These source and test directories are only including if building the secure Hadoop profile - Currently the security classes get packaged with the regular HBase build artifacts. We need a way to at least override project.version, so we can append something like a "-security" suffix indicating the additional security components. - The pseudo-module here is really a half-step forward. It enables the security code to be optionally included in the build for now, and sets up the structure for a security module. But we still will want to pursue full modularization (see HBASE-4336 ), which will allow packing the security code in a separate build artifact. - a new RPC engine providing kerberos and token-based authentication: org.apache.hadoop.hbase.ipc.SecureRpcEngine - implementation under security/src/main/java/org/apache/hadoop/hbase/ipc/ - The implementation classes extend the existing HBaseClient and HBaseServer to share as much of the RPC code as possible. The main override is of the connection classes to allow control over the SASL negotiation of secure connections - existing RPC changes - The existing HBaseClient and HBaseServer have been modified to make subclassing possible - All references to Hadoop UserGroupInformation have been replaced with org.apache.hadoop.hbase.security.User to insulate from future dependencies on specific Hadoop versions - a coprocessor endpoint for obtaining new authentication tokens: TokenProvider, and supporting classes for token generation and synchronization (incorporating HBASE-3615 ) - implementation is under security/src/main/java/org/apache/hadoop/hbase/security/token/ - Secret keys for token generation and verification are synchronized throughout the cluster in zookeeper, under /hbase/tokenauth/keys To enable secure RPC, add the following configuration to hbase-site.xml: <property> <name>hadoop.security.authorization</name> <value>true</value> </property> <property> <name>hadoop.security.authentication</name> <value>kerberos</value> </property> <property> <name>hbase.rpc.engine</name> <value>org.apache.hadoop.hbase.ipc.SecureRpcEngine</value> </property> <property> <name>hbase.coprocessor.region.classes</name> <value>org.apache.hadoop.hbase.security.token.TokenProvider</value> </property> In addition, the master and regionserver processes must be configured for kerberos authentication using the properties: * hbase.(master|regionserver).keytab.file * hbase.(master|regionserver).kerberos.principal * hbase.(master|regionserver).kerberos.https.principal This addresses bug HBASE-2742 . https://issues.apache.org/jira/browse/HBASE-2742 Diffs ----- conf/hbase-policy.xml PRE-CREATION pom.xml 2847416 security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/SecureConnectionHeader.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationKey.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationProtocol.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenIdentifier.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSelector.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/TokenProvider.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/ZKSecretWatcher.java PRE-CREATION security/src/test/java/org/apache/hadoop/hbase/security/token/TestTokenAuthentication.java PRE-CREATION security/src/test/java/org/apache/hadoop/hbase/security/token/TestZKSecretWatcher.java PRE-CREATION security/src/test/resources/hbase-site.xml PRE-CREATION src/assembly/all.xml 3ad8ab3 src/main/java/org/apache/hadoop/hbase/HServerAddress.java f28240c src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java cba7bd1 src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java 904c107 src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 4086829 src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java 054b92b src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 337da78 src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 9117f12 src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java 9e0535e src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java 0dc5bea src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java e8b2c9e src/main/java/org/apache/hadoop/hbase/ipc/RequestContext.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java 1b5629a src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java c8e030c src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java db07ed1 src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java ad88b76 src/main/java/org/apache/hadoop/hbase/master/HMaster.java 19ed5b8 src/main/java/org/apache/hadoop/hbase/security/KerberosInfo.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/security/TokenInfo.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/security/User.java 00bd06d src/main/java/org/apache/hadoop/hbase/zookeeper/ZKLeaderManager.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java 4c5e844 src/main/resources/hbase-default.xml 6f98f5d src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java c0634f4 src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java bb9ea45 Diff: https://reviews.apache.org/r/1991/diff Testing ------- A full test suite run showed failures in TestMasterFailover (timeout) and TestAvroServer. I'll look into those. Cluster testing (1 master + 3 slaves in ec2) with Kerberos authentication and map reduce token authentication, using YCSB, RowCounter, PerformanceEvaluation. Thanks, Gary
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/1991/
          -----------------------------------------------------------

          (Updated 2011-11-17 07:27:20.090955)

          Review request for hbase.

          Changes
          -------

          Updated patch addressing Ted's comments.

          Summary
          -------

          This patch creates a new secure RPC engine for HBase, which provides Kerberos based authentication of clients, and a token-based authentication mechanism for mapreduce jobs. Primary components of the patch are:

          • a new maven profile for secure Hadoop/HBase: hadoop-0.20S
          • Secure Hadoop dependent classes are separated under a pseudo-module in the security/ directory. These source and test directories are only including if building the secure Hadoop profile
          • Currently the security classes get packaged with the regular HBase build artifacts. We need a way to at least override project.version, so we can append something like a "-security" suffix indicating the additional security components.
          • The pseudo-module here is really a half-step forward. It enables the security code to be optionally included in the build for now, and sets up the structure for a security module. But we still will want to pursue full modularization (see HBASE-4336), which will allow packing the security code in a separate build artifact.
          • a new RPC engine providing kerberos and token-based authentication: org.apache.hadoop.hbase.ipc.SecureRpcEngine
          • implementation under security/src/main/java/org/apache/hadoop/hbase/ipc/
          • The implementation classes extend the existing HBaseClient and HBaseServer to share as much of the RPC code as possible. The main override is of the connection classes to allow control over the SASL negotiation of secure connections
          • existing RPC changes
          • The existing HBaseClient and HBaseServer have been modified to make subclassing possible
          • All references to Hadoop UserGroupInformation have been replaced with org.apache.hadoop.hbase.security.User to insulate from future dependencies on specific Hadoop versions
          • a coprocessor endpoint for obtaining new authentication tokens: TokenProvider, and supporting classes for token generation and synchronization (incorporating HBASE-3615)
          • implementation is under security/src/main/java/org/apache/hadoop/hbase/security/token/
          • Secret keys for token generation and verification are synchronized throughout the cluster in zookeeper, under /hbase/tokenauth/keys

          To enable secure RPC, add the following configuration to hbase-site.xml:

          <property>
          <name>hadoop.security.authorization</name>
          <value>true</value>
          </property>
          <property>
          <name>hadoop.security.authentication</name>
          <value>kerberos</value>
          </property>
          <property>
          <name>hbase.rpc.engine</name>
          <value>org.apache.hadoop.hbase.ipc.SecureRpcEngine</value>
          </property>
          <property>
          <name>hbase.coprocessor.region.classes</name>
          <value>org.apache.hadoop.hbase.security.token.TokenProvider</value>
          </property>

          In addition, the master and regionserver processes must be configured for kerberos authentication using the properties:

          • hbase.(master|regionserver).keytab.file
          • hbase.(master|regionserver).kerberos.principal
          • hbase.(master|regionserver).kerberos.https.principal

          This addresses bug HBASE-2742.
          https://issues.apache.org/jira/browse/HBASE-2742

          Diffs (updated)


          conf/hbase-policy.xml PRE-CREATION
          pom.xml 2847416
          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureConnectionHeader.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationKey.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationProtocol.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenIdentifier.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSelector.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/token/TokenProvider.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/token/ZKSecretWatcher.java PRE-CREATION
          security/src/test/java/org/apache/hadoop/hbase/security/token/TestTokenAuthentication.java PRE-CREATION
          security/src/test/java/org/apache/hadoop/hbase/security/token/TestZKSecretWatcher.java PRE-CREATION
          security/src/test/resources/hbase-site.xml PRE-CREATION
          src/assembly/all.xml 3ad8ab3
          src/main/java/org/apache/hadoop/hbase/HServerAddress.java f28240c
          src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java cba7bd1
          src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java 904c107
          src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 4086829
          src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java 054b92b
          src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 337da78
          src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java d9ba0ea
          src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java 9e0535e
          src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java 0dc5bea
          src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java e8b2c9e
          src/main/java/org/apache/hadoop/hbase/ipc/RequestContext.java PRE-CREATION
          src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java 1b5629a
          src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java c8e030c
          src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java db07ed1
          src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java ad88b76
          src/main/java/org/apache/hadoop/hbase/master/HMaster.java 19ed5b8
          src/main/java/org/apache/hadoop/hbase/security/KerberosInfo.java PRE-CREATION
          src/main/java/org/apache/hadoop/hbase/security/TokenInfo.java PRE-CREATION
          src/main/java/org/apache/hadoop/hbase/security/User.java 00bd06d
          src/main/java/org/apache/hadoop/hbase/zookeeper/ZKLeaderManager.java PRE-CREATION
          src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java 4c5e844
          src/main/resources/hbase-default.xml 6f98f5d
          src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java c0634f4
          src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java bb9ea45
          src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKLeaderManager.java PRE-CREATION

          Diff: https://reviews.apache.org/r/1991/diff

          Testing
          -------

          A full test suite run showed failures in TestMasterFailover (timeout) and TestAvroServer. I'll look into those.

          Cluster testing (1 master + 3 slaves in ec2) with Kerberos authentication and map reduce token authentication, using YCSB, RowCounter, PerformanceEvaluation.

          Thanks,

          Gary

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1991/ ----------------------------------------------------------- (Updated 2011-11-17 07:27:20.090955) Review request for hbase. Changes ------- Updated patch addressing Ted's comments. Summary ------- This patch creates a new secure RPC engine for HBase, which provides Kerberos based authentication of clients, and a token-based authentication mechanism for mapreduce jobs. Primary components of the patch are: a new maven profile for secure Hadoop/HBase: hadoop-0.20S Secure Hadoop dependent classes are separated under a pseudo-module in the security/ directory. These source and test directories are only including if building the secure Hadoop profile Currently the security classes get packaged with the regular HBase build artifacts. We need a way to at least override project.version, so we can append something like a "-security" suffix indicating the additional security components. The pseudo-module here is really a half-step forward. It enables the security code to be optionally included in the build for now, and sets up the structure for a security module. But we still will want to pursue full modularization (see HBASE-4336 ), which will allow packing the security code in a separate build artifact. a new RPC engine providing kerberos and token-based authentication: org.apache.hadoop.hbase.ipc.SecureRpcEngine implementation under security/src/main/java/org/apache/hadoop/hbase/ipc/ The implementation classes extend the existing HBaseClient and HBaseServer to share as much of the RPC code as possible. The main override is of the connection classes to allow control over the SASL negotiation of secure connections existing RPC changes The existing HBaseClient and HBaseServer have been modified to make subclassing possible All references to Hadoop UserGroupInformation have been replaced with org.apache.hadoop.hbase.security.User to insulate from future dependencies on specific Hadoop versions a coprocessor endpoint for obtaining new authentication tokens: TokenProvider, and supporting classes for token generation and synchronization (incorporating HBASE-3615 ) implementation is under security/src/main/java/org/apache/hadoop/hbase/security/token/ Secret keys for token generation and verification are synchronized throughout the cluster in zookeeper, under /hbase/tokenauth/keys To enable secure RPC, add the following configuration to hbase-site.xml: <property> <name>hadoop.security.authorization</name> <value>true</value> </property> <property> <name>hadoop.security.authentication</name> <value>kerberos</value> </property> <property> <name>hbase.rpc.engine</name> <value>org.apache.hadoop.hbase.ipc.SecureRpcEngine</value> </property> <property> <name>hbase.coprocessor.region.classes</name> <value>org.apache.hadoop.hbase.security.token.TokenProvider</value> </property> In addition, the master and regionserver processes must be configured for kerberos authentication using the properties: hbase.(master|regionserver).keytab.file hbase.(master|regionserver).kerberos.principal hbase.(master|regionserver).kerberos.https.principal This addresses bug HBASE-2742 . https://issues.apache.org/jira/browse/HBASE-2742 Diffs (updated) conf/hbase-policy.xml PRE-CREATION pom.xml 2847416 security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/SecureConnectionHeader.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationKey.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationProtocol.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenIdentifier.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSelector.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/TokenProvider.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/ZKSecretWatcher.java PRE-CREATION security/src/test/java/org/apache/hadoop/hbase/security/token/TestTokenAuthentication.java PRE-CREATION security/src/test/java/org/apache/hadoop/hbase/security/token/TestZKSecretWatcher.java PRE-CREATION security/src/test/resources/hbase-site.xml PRE-CREATION src/assembly/all.xml 3ad8ab3 src/main/java/org/apache/hadoop/hbase/HServerAddress.java f28240c src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java cba7bd1 src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java 904c107 src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 4086829 src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java 054b92b src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 337da78 src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java d9ba0ea src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java 9e0535e src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java 0dc5bea src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java e8b2c9e src/main/java/org/apache/hadoop/hbase/ipc/RequestContext.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java 1b5629a src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java c8e030c src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java db07ed1 src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java ad88b76 src/main/java/org/apache/hadoop/hbase/master/HMaster.java 19ed5b8 src/main/java/org/apache/hadoop/hbase/security/KerberosInfo.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/security/TokenInfo.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/security/User.java 00bd06d src/main/java/org/apache/hadoop/hbase/zookeeper/ZKLeaderManager.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java 4c5e844 src/main/resources/hbase-default.xml 6f98f5d src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java c0634f4 src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java bb9ea45 src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKLeaderManager.java PRE-CREATION Diff: https://reviews.apache.org/r/1991/diff Testing ------- A full test suite run showed failures in TestMasterFailover (timeout) and TestAvroServer. I'll look into those. Cluster testing (1 master + 3 slaves in ec2) with Kerberos authentication and map reduce token authentication, using YCSB, RowCounter, PerformanceEvaluation. Thanks, Gary
          Hide
          jiraposter@reviews.apache.org added a comment -

          On 2011-11-14 22:36:05, Ted Yu wrote:

          > security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java, line 72

          > <https://reviews.apache.org/r/1991/diff/5/?file=57988#file57988line72>

          >

          > Should we use org.apache.hadoop.hbase.ipc.SecureClient ?

          Gary Helmling wrote:

          This follows the same logging convention as HBaseClient/HBaseServer – it uses a mangled package name so that we don't get noisy IPC debug logging when setting org.apache.hadoop.hbase=DEBUG in log4j config.

          It's a bit weird, but it works. Since log4j does actually support "trace" level now, it might be better to correct the package names and use trace level for all the noisy (per request) logging. This would keep debug logging relatively clean without the package mangling. But I think we should tackle that as a separate issue.

          Opened HBASE-4810 for broader logging cleanup task.

          On 2011-11-14 22:36:05, Ted Yu wrote:

          > security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java, line 171

          > <https://reviews.apache.org/r/1991/diff/5/?file=57999#file57999line171>

          >

          > Should read 'when running as master'

          Done.

          On 2011-11-14 22:36:05, Ted Yu wrote:

          > security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java, line 173

          > <https://reviews.apache.org/r/1991/diff/5/?file=57999#file57999line173>

          >

          > This log should be guarded by LOG.isDebugEnabled()

          Done.

          • Gary

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/1991/#review3235
          -----------------------------------------------------------

          On 2011-11-17 07:27:20, Gary Helmling wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/1991/

          -----------------------------------------------------------

          (Updated 2011-11-17 07:27:20)

          Review request for hbase.

          Summary

          -------

          This patch creates a new secure RPC engine for HBase, which provides Kerberos based authentication of clients, and a token-based authentication mechanism for mapreduce jobs. Primary components of the patch are:

          - a new maven profile for secure Hadoop/HBase: hadoop-0.20S

          - Secure Hadoop dependent classes are separated under a pseudo-module in the security/ directory. These source and test directories are only including if building the secure Hadoop profile

          - Currently the security classes get packaged with the regular HBase build artifacts. We need a way to at least override project.version, so we can append something like a "-security" suffix indicating the additional security components.

          - The pseudo-module here is really a half-step forward. It enables the security code to be optionally included in the build for now, and sets up the structure for a security module. But we still will want to pursue full modularization (see HBASE-4336), which will allow packing the security code in a separate build artifact.

          - a new RPC engine providing kerberos and token-based authentication: org.apache.hadoop.hbase.ipc.SecureRpcEngine

          - implementation under security/src/main/java/org/apache/hadoop/hbase/ipc/

          - The implementation classes extend the existing HBaseClient and HBaseServer to share as much of the RPC code as possible. The main override is of the connection classes to allow control over the SASL negotiation of secure connections

          - existing RPC changes

          - The existing HBaseClient and HBaseServer have been modified to make subclassing possible

          - All references to Hadoop UserGroupInformation have been replaced with org.apache.hadoop.hbase.security.User to insulate from future dependencies on specific Hadoop versions

          - a coprocessor endpoint for obtaining new authentication tokens: TokenProvider, and supporting classes for token generation and synchronization (incorporating HBASE-3615)

          - implementation is under security/src/main/java/org/apache/hadoop/hbase/security/token/

          - Secret keys for token generation and verification are synchronized throughout the cluster in zookeeper, under /hbase/tokenauth/keys

          To enable secure RPC, add the following configuration to hbase-site.xml:

          <property>

          <name>hadoop.security.authorization</name>

          <value>true</value>

          </property>

          <property>

          <name>hadoop.security.authentication</name>

          <value>kerberos</value>

          </property>

          <property>

          <name>hbase.rpc.engine</name>

          <value>org.apache.hadoop.hbase.ipc.SecureRpcEngine</value>

          </property>

          <property>

          <name>hbase.coprocessor.region.classes</name>

          <value>org.apache.hadoop.hbase.security.token.TokenProvider</value>

          </property>

          In addition, the master and regionserver processes must be configured for kerberos authentication using the properties:

          * hbase.(master|regionserver).keytab.file

          * hbase.(master|regionserver).kerberos.principal

          * hbase.(master|regionserver).kerberos.https.principal

          This addresses bug HBASE-2742.

          https://issues.apache.org/jira/browse/HBASE-2742

          Diffs

          -----

          conf/hbase-policy.xml PRE-CREATION

          pom.xml 2847416

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureConnectionHeader.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationKey.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationProtocol.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenIdentifier.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSelector.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/TokenProvider.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/ZKSecretWatcher.java PRE-CREATION

          security/src/test/java/org/apache/hadoop/hbase/security/token/TestTokenAuthentication.java PRE-CREATION

          security/src/test/java/org/apache/hadoop/hbase/security/token/TestZKSecretWatcher.java PRE-CREATION

          security/src/test/resources/hbase-site.xml PRE-CREATION

          src/assembly/all.xml 3ad8ab3

          src/main/java/org/apache/hadoop/hbase/HServerAddress.java f28240c

          src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java cba7bd1

          src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java 904c107

          src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 4086829

          src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java 054b92b

          src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 337da78

          src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java d9ba0ea

          src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java 9e0535e

          src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java 0dc5bea

          src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java e8b2c9e

          src/main/java/org/apache/hadoop/hbase/ipc/RequestContext.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java 1b5629a

          src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java c8e030c

          src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java db07ed1

          src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java ad88b76

          src/main/java/org/apache/hadoop/hbase/master/HMaster.java 19ed5b8

          src/main/java/org/apache/hadoop/hbase/security/KerberosInfo.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/security/TokenInfo.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/security/User.java 00bd06d

          src/main/java/org/apache/hadoop/hbase/zookeeper/ZKLeaderManager.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java 4c5e844

          src/main/resources/hbase-default.xml 6f98f5d

          src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java c0634f4

          src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java bb9ea45

          src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKLeaderManager.java PRE-CREATION

          Diff: https://reviews.apache.org/r/1991/diff

          Testing

          -------

          A full test suite run showed failures in TestMasterFailover (timeout) and TestAvroServer. I'll look into those.

          Cluster testing (1 master + 3 slaves in ec2) with Kerberos authentication and map reduce token authentication, using YCSB, RowCounter, PerformanceEvaluation.

          Thanks,

          Gary

          Show
          jiraposter@reviews.apache.org added a comment - On 2011-11-14 22:36:05, Ted Yu wrote: > security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java, line 72 > < https://reviews.apache.org/r/1991/diff/5/?file=57988#file57988line72 > > > Should we use org.apache.hadoop.hbase.ipc.SecureClient ? Gary Helmling wrote: This follows the same logging convention as HBaseClient/HBaseServer – it uses a mangled package name so that we don't get noisy IPC debug logging when setting org.apache.hadoop.hbase=DEBUG in log4j config. It's a bit weird, but it works. Since log4j does actually support "trace" level now, it might be better to correct the package names and use trace level for all the noisy (per request) logging. This would keep debug logging relatively clean without the package mangling. But I think we should tackle that as a separate issue. Opened HBASE-4810 for broader logging cleanup task. On 2011-11-14 22:36:05, Ted Yu wrote: > security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java, line 171 > < https://reviews.apache.org/r/1991/diff/5/?file=57999#file57999line171 > > > Should read 'when running as master' Done. On 2011-11-14 22:36:05, Ted Yu wrote: > security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java, line 173 > < https://reviews.apache.org/r/1991/diff/5/?file=57999#file57999line173 > > > This log should be guarded by LOG.isDebugEnabled() Done. Gary ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1991/#review3235 ----------------------------------------------------------- On 2011-11-17 07:27:20, Gary Helmling wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1991/ ----------------------------------------------------------- (Updated 2011-11-17 07:27:20) Review request for hbase. Summary ------- This patch creates a new secure RPC engine for HBase, which provides Kerberos based authentication of clients, and a token-based authentication mechanism for mapreduce jobs. Primary components of the patch are: - a new maven profile for secure Hadoop/HBase: hadoop-0.20S - Secure Hadoop dependent classes are separated under a pseudo-module in the security/ directory. These source and test directories are only including if building the secure Hadoop profile - Currently the security classes get packaged with the regular HBase build artifacts. We need a way to at least override project.version, so we can append something like a "-security" suffix indicating the additional security components. - The pseudo-module here is really a half-step forward. It enables the security code to be optionally included in the build for now, and sets up the structure for a security module. But we still will want to pursue full modularization (see HBASE-4336 ), which will allow packing the security code in a separate build artifact. - a new RPC engine providing kerberos and token-based authentication: org.apache.hadoop.hbase.ipc.SecureRpcEngine - implementation under security/src/main/java/org/apache/hadoop/hbase/ipc/ - The implementation classes extend the existing HBaseClient and HBaseServer to share as much of the RPC code as possible. The main override is of the connection classes to allow control over the SASL negotiation of secure connections - existing RPC changes - The existing HBaseClient and HBaseServer have been modified to make subclassing possible - All references to Hadoop UserGroupInformation have been replaced with org.apache.hadoop.hbase.security.User to insulate from future dependencies on specific Hadoop versions - a coprocessor endpoint for obtaining new authentication tokens: TokenProvider, and supporting classes for token generation and synchronization (incorporating HBASE-3615 ) - implementation is under security/src/main/java/org/apache/hadoop/hbase/security/token/ - Secret keys for token generation and verification are synchronized throughout the cluster in zookeeper, under /hbase/tokenauth/keys To enable secure RPC, add the following configuration to hbase-site.xml: <property> <name>hadoop.security.authorization</name> <value>true</value> </property> <property> <name>hadoop.security.authentication</name> <value>kerberos</value> </property> <property> <name>hbase.rpc.engine</name> <value>org.apache.hadoop.hbase.ipc.SecureRpcEngine</value> </property> <property> <name>hbase.coprocessor.region.classes</name> <value>org.apache.hadoop.hbase.security.token.TokenProvider</value> </property> In addition, the master and regionserver processes must be configured for kerberos authentication using the properties: * hbase.(master|regionserver).keytab.file * hbase.(master|regionserver).kerberos.principal * hbase.(master|regionserver).kerberos.https.principal This addresses bug HBASE-2742 . https://issues.apache.org/jira/browse/HBASE-2742 Diffs ----- conf/hbase-policy.xml PRE-CREATION pom.xml 2847416 security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/SecureConnectionHeader.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationKey.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationProtocol.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenIdentifier.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSelector.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/TokenProvider.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/ZKSecretWatcher.java PRE-CREATION security/src/test/java/org/apache/hadoop/hbase/security/token/TestTokenAuthentication.java PRE-CREATION security/src/test/java/org/apache/hadoop/hbase/security/token/TestZKSecretWatcher.java PRE-CREATION security/src/test/resources/hbase-site.xml PRE-CREATION src/assembly/all.xml 3ad8ab3 src/main/java/org/apache/hadoop/hbase/HServerAddress.java f28240c src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java cba7bd1 src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java 904c107 src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 4086829 src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java 054b92b src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 337da78 src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java d9ba0ea src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java 9e0535e src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java 0dc5bea src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java e8b2c9e src/main/java/org/apache/hadoop/hbase/ipc/RequestContext.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java 1b5629a src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java c8e030c src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java db07ed1 src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java ad88b76 src/main/java/org/apache/hadoop/hbase/master/HMaster.java 19ed5b8 src/main/java/org/apache/hadoop/hbase/security/KerberosInfo.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/security/TokenInfo.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/security/User.java 00bd06d src/main/java/org/apache/hadoop/hbase/zookeeper/ZKLeaderManager.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java 4c5e844 src/main/resources/hbase-default.xml 6f98f5d src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java c0634f4 src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java bb9ea45 src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKLeaderManager.java PRE-CREATION Diff: https://reviews.apache.org/r/1991/diff Testing ------- A full test suite run showed failures in TestMasterFailover (timeout) and TestAvroServer. I'll look into those. Cluster testing (1 master + 3 slaves in ec2) with Kerberos authentication and map reduce token authentication, using YCSB, RowCounter, PerformanceEvaluation. Thanks, Gary
          Hide
          Ted Yu added a comment -

          Changes to waitToBecomeLeader() look good.

          Looking at AuthenticationTokenSecretManager, it becomes leader upon entering run().
          Periodically it calls rollCurrentKey() which checks whether it is still leader. I think rollCurrentKey() should return a boolean value indicating leadership status so that the while loop can re-obtain leadership if leadership is lost.

          I think exception handling below may have some room for improvement:

                  } catch (IOException ioe) {
                    LOG.error("Error updating keys", ioe);
                  }
          

          In my opinion, failure to remove expired key or to add new key should be treated with more caution.
          addKeyToZK() and updateKeyInZK() throws RuntimeException, alleviating this concern a little. But zkWatcher.removeKeyFromZK() swallows KeeperException, making things complicated. We may need to call ZKUtil.deleteNode() directly.

          In rollCurrentKey():

                  Long.MAX_VALUE, // don't allow to expire until it's replace by a new key
          

          The above should read 'replaced by'.

          Show
          Ted Yu added a comment - Changes to waitToBecomeLeader() look good. Looking at AuthenticationTokenSecretManager, it becomes leader upon entering run(). Periodically it calls rollCurrentKey() which checks whether it is still leader. I think rollCurrentKey() should return a boolean value indicating leadership status so that the while loop can re-obtain leadership if leadership is lost. I think exception handling below may have some room for improvement: } catch (IOException ioe) { LOG.error( "Error updating keys" , ioe); } In my opinion, failure to remove expired key or to add new key should be treated with more caution. addKeyToZK() and updateKeyInZK() throws RuntimeException, alleviating this concern a little. But zkWatcher.removeKeyFromZK() swallows KeeperException, making things complicated. We may need to call ZKUtil.deleteNode() directly. In rollCurrentKey(): Long .MAX_VALUE, // don't allow to expire until it's replace by a new key The above should read 'replaced by'.
          Hide
          Ted Yu added a comment -

          In src/main/resources/hbase-default.xml, I think we should use dot instead of dash for parameter key names:
          hbase.auth.key.update-interval should be hbase.auth.key.update.interval
          hbase.auth.token.max-lifetime should be hbase.auth.token.max.lifetime

          Show
          Ted Yu added a comment - In src/main/resources/hbase-default.xml, I think we should use dot instead of dash for parameter key names: hbase.auth.key.update-interval should be hbase.auth.key.update.interval hbase.auth.token.max-lifetime should be hbase.auth.token.max.lifetime
          Hide
          Ted Yu added a comment -

          In ZKLeaderManager.stepDownAsLeader():

                    ZKUtil.deleteNodeFailSilent(watcher, leaderZNode);
          

          I think we should call ZKUtil.deleteNode() instead and handle KeeperException gracefully because waitToBecomeLeader() would only delete zk node which has the other candidate's Id.

          Show
          Ted Yu added a comment - In ZKLeaderManager.stepDownAsLeader(): ZKUtil.deleteNodeFailSilent(watcher, leaderZNode); I think we should call ZKUtil.deleteNode() instead and handle KeeperException gracefully because waitToBecomeLeader() would only delete zk node which has the other candidate's Id.
          Hide
          Ted Yu added a comment -

          Minor:
          For User.obtainAuthTokenForJob() methods, we rethrow RuntimeException so that we can reach the final Exception clause.
          I wonder if this is necessary. NPE, e.g., is a RuntimeException.

          Show
          Ted Yu added a comment - Minor: For User.obtainAuthTokenForJob() methods, we rethrow RuntimeException so that we can reach the final Exception clause. I wonder if this is necessary. NPE, e.g., is a RuntimeException.
          Hide
          Ted Yu added a comment -

          In HBaseServer.closeCurrentConnection(), the following is added:

                    c = null;
          

          Since c is the attachment to SelectionKey, should we call key.attach(null) ?

          Show
          Ted Yu added a comment - In HBaseServer.closeCurrentConnection(), the following is added: c = null ; Since c is the attachment to SelectionKey, should we call key.attach(null) ?
          Hide
          Ted Yu added a comment -

          HBaseRPC.call() is marked deprecated. Probably we should mention SecureRpcEngine so that developers know what to look for.

          Show
          Ted Yu added a comment - HBaseRPC.call() is marked deprecated. Probably we should mention SecureRpcEngine so that developers know what to look for.
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/1991/#review3322
          -----------------------------------------------------------

          Ship it!

          Pretty crazy patch Gary. I tried reviewing the security code but my eyes started to glaze over and besides it looks like a bunch of copy/paste; in other words, it don't look like you wrote it. I skipped ahead after a while to core changes. There are a few questions below but as long as all tests pass, I'm fine w/ commit (even fine w/ the bit of pollution of core w/ security mentions).

          pom.xml
          <https://reviews.apache.org/r/1991/#comment7409>

          Whats going on here? Why would we exclude the test hbase-site.xml? I thought it had all configs. for test running? Or is this in the security profile? I see you then add it back in later under the plugin section. Did we have it in wrong place?

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java
          <https://reviews.apache.org/r/1991/#comment7424>

          This is a number?

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java
          <https://reviews.apache.org/r/1991/#comment7413>

          Could we get an enum we don't know about?

          If copied from hadoop code, don't worry about this comment.

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java
          <https://reviews.apache.org/r/1991/#comment7414>

          Man, this is kinda ugly... its a data member in a super class? Cryptic!

          (This is hadoop code?)

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java
          <https://reviews.apache.org/r/1991/#comment7422>

          This stuff is hard to review since seems to be mostly hadoop copy paste code.

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java
          <https://reviews.apache.org/r/1991/#comment7423>

          Funny. Above we compare on State.SUCCESS.state. What if there is a FATAL here? Anything special we should do?

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java
          <https://reviews.apache.org/r/1991/#comment7430>

          This ain't you but crazy stuff here... You'd think hadoop would at least log it? I suppose notthing you can do about these on close anyways?

          src/assembly/all.xml
          <https://reviews.apache.org/r/1991/#comment7431>

          This will add the 'security' on the end if we are building w/ the security profile?

          src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
          <https://reviews.apache.org/r/1991/#comment7433>

          So, in insecure hbase, is user provided? If not, this message will be all over our logs?

          src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java
          <https://reviews.apache.org/r/1991/#comment7434>

          Good.

          src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java
          <https://reviews.apache.org/r/1991/#comment7436>

          Say what to use instead?

          src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
          <https://reviews.apache.org/r/1991/#comment7438>

          This seems like an odd one to expose to subclasses.

          src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
          <https://reviews.apache.org/r/1991/#comment7439>

          Is this 'fix' copied from hadoop? Seems sensible.

          src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java
          <https://reviews.apache.org/r/1991/#comment7440>

          Whats happening here? Do we have to do this – security pollution of base hbase interface?

          src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java
          <https://reviews.apache.org/r/1991/#comment7441>

          ditto

          Not a problem if this is the way it has to be...

          src/main/java/org/apache/hadoop/hbase/master/HMaster.java
          <https://reviews.apache.org/r/1991/#comment7442>

          this a new call we have to make?

          src/main/resources/hbase-default.xml
          <https://reviews.apache.org/r/1991/#comment7443>

          This stuff should be in here and not in an hbase-security-default.xml that comes in only when running security?

          Its ok this stuff is here... just wondering if could be shown only when a security context.

          • Michael

          On 2011-11-17 07:27:20, Gary Helmling wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/1991/

          -----------------------------------------------------------

          (Updated 2011-11-17 07:27:20)

          Review request for hbase.

          Summary

          -------

          This patch creates a new secure RPC engine for HBase, which provides Kerberos based authentication of clients, and a token-based authentication mechanism for mapreduce jobs. Primary components of the patch are:

          - a new maven profile for secure Hadoop/HBase: hadoop-0.20S

          - Secure Hadoop dependent classes are separated under a pseudo-module in the security/ directory. These source and test directories are only including if building the secure Hadoop profile

          - Currently the security classes get packaged with the regular HBase build artifacts. We need a way to at least override project.version, so we can append something like a "-security" suffix indicating the additional security components.

          - The pseudo-module here is really a half-step forward. It enables the security code to be optionally included in the build for now, and sets up the structure for a security module. But we still will want to pursue full modularization (see HBASE-4336), which will allow packing the security code in a separate build artifact.

          - a new RPC engine providing kerberos and token-based authentication: org.apache.hadoop.hbase.ipc.SecureRpcEngine

          - implementation under security/src/main/java/org/apache/hadoop/hbase/ipc/

          - The implementation classes extend the existing HBaseClient and HBaseServer to share as much of the RPC code as possible. The main override is of the connection classes to allow control over the SASL negotiation of secure connections

          - existing RPC changes

          - The existing HBaseClient and HBaseServer have been modified to make subclassing possible

          - All references to Hadoop UserGroupInformation have been replaced with org.apache.hadoop.hbase.security.User to insulate from future dependencies on specific Hadoop versions

          - a coprocessor endpoint for obtaining new authentication tokens: TokenProvider, and supporting classes for token generation and synchronization (incorporating HBASE-3615)

          - implementation is under security/src/main/java/org/apache/hadoop/hbase/security/token/

          - Secret keys for token generation and verification are synchronized throughout the cluster in zookeeper, under /hbase/tokenauth/keys

          To enable secure RPC, add the following configuration to hbase-site.xml:

          <property>

          <name>hadoop.security.authorization</name>

          <value>true</value>

          </property>

          <property>

          <name>hadoop.security.authentication</name>

          <value>kerberos</value>

          </property>

          <property>

          <name>hbase.rpc.engine</name>

          <value>org.apache.hadoop.hbase.ipc.SecureRpcEngine</value>

          </property>

          <property>

          <name>hbase.coprocessor.region.classes</name>

          <value>org.apache.hadoop.hbase.security.token.TokenProvider</value>

          </property>

          In addition, the master and regionserver processes must be configured for kerberos authentication using the properties:

          * hbase.(master|regionserver).keytab.file

          * hbase.(master|regionserver).kerberos.principal

          * hbase.(master|regionserver).kerberos.https.principal

          This addresses bug HBASE-2742.

          https://issues.apache.org/jira/browse/HBASE-2742

          Diffs

          -----

          conf/hbase-policy.xml PRE-CREATION

          pom.xml 2847416

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureConnectionHeader.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationKey.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationProtocol.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenIdentifier.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSelector.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/TokenProvider.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/ZKSecretWatcher.java PRE-CREATION

          security/src/test/java/org/apache/hadoop/hbase/security/token/TestTokenAuthentication.java PRE-CREATION

          security/src/test/java/org/apache/hadoop/hbase/security/token/TestZKSecretWatcher.java PRE-CREATION

          security/src/test/resources/hbase-site.xml PRE-CREATION

          src/assembly/all.xml 3ad8ab3

          src/main/java/org/apache/hadoop/hbase/HServerAddress.java f28240c

          src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java cba7bd1

          src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java 904c107

          src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 4086829

          src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java 054b92b

          src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 337da78

          src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java d9ba0ea

          src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java 9e0535e

          src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java 0dc5bea

          src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java e8b2c9e

          src/main/java/org/apache/hadoop/hbase/ipc/RequestContext.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java 1b5629a

          src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java c8e030c

          src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java db07ed1

          src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java ad88b76

          src/main/java/org/apache/hadoop/hbase/master/HMaster.java 19ed5b8

          src/main/java/org/apache/hadoop/hbase/security/KerberosInfo.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/security/TokenInfo.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/security/User.java 00bd06d

          src/main/java/org/apache/hadoop/hbase/zookeeper/ZKLeaderManager.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java 4c5e844

          src/main/resources/hbase-default.xml 6f98f5d

          src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java c0634f4

          src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java bb9ea45

          src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKLeaderManager.java PRE-CREATION

          Diff: https://reviews.apache.org/r/1991/diff

          Testing

          -------

          A full test suite run showed failures in TestMasterFailover (timeout) and TestAvroServer. I'll look into those.

          Cluster testing (1 master + 3 slaves in ec2) with Kerberos authentication and map reduce token authentication, using YCSB, RowCounter, PerformanceEvaluation.

          Thanks,

          Gary

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1991/#review3322 ----------------------------------------------------------- Ship it! Pretty crazy patch Gary. I tried reviewing the security code but my eyes started to glaze over and besides it looks like a bunch of copy/paste; in other words, it don't look like you wrote it. I skipped ahead after a while to core changes. There are a few questions below but as long as all tests pass, I'm fine w/ commit (even fine w/ the bit of pollution of core w/ security mentions). pom.xml < https://reviews.apache.org/r/1991/#comment7409 > Whats going on here? Why would we exclude the test hbase-site.xml? I thought it had all configs. for test running? Or is this in the security profile? I see you then add it back in later under the plugin section. Did we have it in wrong place? security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java < https://reviews.apache.org/r/1991/#comment7424 > This is a number? security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java < https://reviews.apache.org/r/1991/#comment7413 > Could we get an enum we don't know about? If copied from hadoop code, don't worry about this comment. security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java < https://reviews.apache.org/r/1991/#comment7414 > Man, this is kinda ugly... its a data member in a super class? Cryptic! (This is hadoop code?) security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java < https://reviews.apache.org/r/1991/#comment7422 > This stuff is hard to review since seems to be mostly hadoop copy paste code. security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java < https://reviews.apache.org/r/1991/#comment7423 > Funny. Above we compare on State.SUCCESS.state. What if there is a FATAL here? Anything special we should do? security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java < https://reviews.apache.org/r/1991/#comment7430 > This ain't you but crazy stuff here... You'd think hadoop would at least log it? I suppose notthing you can do about these on close anyways? src/assembly/all.xml < https://reviews.apache.org/r/1991/#comment7431 > This will add the 'security' on the end if we are building w/ the security profile? src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java < https://reviews.apache.org/r/1991/#comment7433 > So, in insecure hbase, is user provided? If not, this message will be all over our logs? src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java < https://reviews.apache.org/r/1991/#comment7434 > Good. src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java < https://reviews.apache.org/r/1991/#comment7436 > Say what to use instead? src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java < https://reviews.apache.org/r/1991/#comment7438 > This seems like an odd one to expose to subclasses. src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java < https://reviews.apache.org/r/1991/#comment7439 > Is this 'fix' copied from hadoop? Seems sensible. src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java < https://reviews.apache.org/r/1991/#comment7440 > Whats happening here? Do we have to do this – security pollution of base hbase interface? src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java < https://reviews.apache.org/r/1991/#comment7441 > ditto Not a problem if this is the way it has to be... src/main/java/org/apache/hadoop/hbase/master/HMaster.java < https://reviews.apache.org/r/1991/#comment7442 > this a new call we have to make? src/main/resources/hbase-default.xml < https://reviews.apache.org/r/1991/#comment7443 > This stuff should be in here and not in an hbase-security-default.xml that comes in only when running security? Its ok this stuff is here... just wondering if could be shown only when a security context. Michael On 2011-11-17 07:27:20, Gary Helmling wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1991/ ----------------------------------------------------------- (Updated 2011-11-17 07:27:20) Review request for hbase. Summary ------- This patch creates a new secure RPC engine for HBase, which provides Kerberos based authentication of clients, and a token-based authentication mechanism for mapreduce jobs. Primary components of the patch are: - a new maven profile for secure Hadoop/HBase: hadoop-0.20S - Secure Hadoop dependent classes are separated under a pseudo-module in the security/ directory. These source and test directories are only including if building the secure Hadoop profile - Currently the security classes get packaged with the regular HBase build artifacts. We need a way to at least override project.version, so we can append something like a "-security" suffix indicating the additional security components. - The pseudo-module here is really a half-step forward. It enables the security code to be optionally included in the build for now, and sets up the structure for a security module. But we still will want to pursue full modularization (see HBASE-4336 ), which will allow packing the security code in a separate build artifact. - a new RPC engine providing kerberos and token-based authentication: org.apache.hadoop.hbase.ipc.SecureRpcEngine - implementation under security/src/main/java/org/apache/hadoop/hbase/ipc/ - The implementation classes extend the existing HBaseClient and HBaseServer to share as much of the RPC code as possible. The main override is of the connection classes to allow control over the SASL negotiation of secure connections - existing RPC changes - The existing HBaseClient and HBaseServer have been modified to make subclassing possible - All references to Hadoop UserGroupInformation have been replaced with org.apache.hadoop.hbase.security.User to insulate from future dependencies on specific Hadoop versions - a coprocessor endpoint for obtaining new authentication tokens: TokenProvider, and supporting classes for token generation and synchronization (incorporating HBASE-3615 ) - implementation is under security/src/main/java/org/apache/hadoop/hbase/security/token/ - Secret keys for token generation and verification are synchronized throughout the cluster in zookeeper, under /hbase/tokenauth/keys To enable secure RPC, add the following configuration to hbase-site.xml: <property> <name>hadoop.security.authorization</name> <value>true</value> </property> <property> <name>hadoop.security.authentication</name> <value>kerberos</value> </property> <property> <name>hbase.rpc.engine</name> <value>org.apache.hadoop.hbase.ipc.SecureRpcEngine</value> </property> <property> <name>hbase.coprocessor.region.classes</name> <value>org.apache.hadoop.hbase.security.token.TokenProvider</value> </property> In addition, the master and regionserver processes must be configured for kerberos authentication using the properties: * hbase.(master|regionserver).keytab.file * hbase.(master|regionserver).kerberos.principal * hbase.(master|regionserver).kerberos.https.principal This addresses bug HBASE-2742 . https://issues.apache.org/jira/browse/HBASE-2742 Diffs ----- conf/hbase-policy.xml PRE-CREATION pom.xml 2847416 security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/SecureConnectionHeader.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationKey.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationProtocol.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenIdentifier.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSelector.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/TokenProvider.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/ZKSecretWatcher.java PRE-CREATION security/src/test/java/org/apache/hadoop/hbase/security/token/TestTokenAuthentication.java PRE-CREATION security/src/test/java/org/apache/hadoop/hbase/security/token/TestZKSecretWatcher.java PRE-CREATION security/src/test/resources/hbase-site.xml PRE-CREATION src/assembly/all.xml 3ad8ab3 src/main/java/org/apache/hadoop/hbase/HServerAddress.java f28240c src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java cba7bd1 src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java 904c107 src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 4086829 src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java 054b92b src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 337da78 src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java d9ba0ea src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java 9e0535e src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java 0dc5bea src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java e8b2c9e src/main/java/org/apache/hadoop/hbase/ipc/RequestContext.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java 1b5629a src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java c8e030c src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java db07ed1 src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java ad88b76 src/main/java/org/apache/hadoop/hbase/master/HMaster.java 19ed5b8 src/main/java/org/apache/hadoop/hbase/security/KerberosInfo.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/security/TokenInfo.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/security/User.java 00bd06d src/main/java/org/apache/hadoop/hbase/zookeeper/ZKLeaderManager.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java 4c5e844 src/main/resources/hbase-default.xml 6f98f5d src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java c0634f4 src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java bb9ea45 src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKLeaderManager.java PRE-CREATION Diff: https://reviews.apache.org/r/1991/diff Testing ------- A full test suite run showed failures in TestMasterFailover (timeout) and TestAvroServer. I'll look into those. Cluster testing (1 master + 3 slaves in ec2) with Kerberos authentication and map reduce token authentication, using YCSB, RowCounter, PerformanceEvaluation. Thanks, Gary
          Hide
          jiraposter@reviews.apache.org added a comment -

          On 2011-11-17 23:03:06, Michael Stack wrote:

          > pom.xml, line 408

          > <https://reviews.apache.org/r/1991/diff/6/?file=58999#file58999line408>

          >

          > Whats going on here? Why would we exclude the test hbase-site.xml? I thought it had all configs. for test running? Or is this in the security profile? I see you then add it back in later under the plugin section. Did we have it in wrong place?

          This is so that, when building with security profile, we can override the hbase-site.xml used for tests – specifically set it to use SecureRpcEngine. It's an ugly change to the POM and kind of ugly to duplicate the test hbase-site.xml, but I didn't see an easier way of doing it. I'm open to other suggestions to improve it.

          On 2011-11-17 23:03:06, Michael Stack wrote:

          > security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java, line 355

          > <https://reviews.apache.org/r/1991/diff/6/?file=59000#file59000line355>

          >

          > This is a number?

          Yes:

          SUCCESS (0),
          ERROR (1),
          FATAL (-1);

          On 2011-11-17 23:03:06, Michael Stack wrote:

          > security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java, line 369

          > <https://reviews.apache.org/r/1991/diff/6/?file=59000#file59000line369>

          >

          > Could we get an enum we don't know about?

          >

          > If copied from hadoop code, don't worry about this comment.

          Yeah this is direct from Hadoop. It is a bit odd not to have a default block, which should probably behave same as fatal. Depends if we want to diverge more by adding it.

          On 2011-11-17 23:03:06, Michael Stack wrote:

          > security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java, line 375

          > <https://reviews.apache.org/r/1991/diff/6/?file=59000#file59000line375>

          >

          > Man, this is kinda ugly... its a data member in a super class? Cryptic!

          >

          > (This is hadoop code?)

          This is actually from HBaseClient. Hadoop just does:

          } catch (IOException e)

          { markClosed(e); }

          On 2011-11-17 23:03:06, Michael Stack wrote:

          > security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java, line 143

          > <https://reviews.apache.org/r/1991/diff/6/?file=59003#file59003line143>

          >

          > Funny. Above we compare on State.SUCCESS.state. What if there is a FATAL here? Anything special we should do?

          In SecureClient, I think the check on State.SUCCESS.state is to avoid mapping the deserialized value back to the enum value.

          In the case of FATAL, the client will still see fatal status written separately in the response. For errorClass and error, FATAL is equivalent to ERROR.

          On 2011-11-17 23:03:06, Michael Stack wrote:

          > src/assembly/all.xml, line 60

          > <https://reviews.apache.org/r/1991/diff/6/?file=59019#file59019line60>

          >

          > This will add the 'security' on the end if we are building w/ the security profile?

          Yes, so for say 0.92, when building with -P security, you would wind up with hbase-0.92.0-security.jar. Without the security profile, you just get hbase-0.92.0.jar.

          This at least saves having to fiddle with the project version for secure vs. insecure builds, and gives artifacts named as expected. But not sure if it complicates deploying to a maven repo? It might still need more tweaking to improve how it works.

          On 2011-11-17 23:03:06, Michael Stack wrote:

          > src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java, line 399

          > <https://reviews.apache.org/r/1991/diff/6/?file=59021#file59021line399>

          >

          > So, in insecure hbase, is user provided? If not, this message will be all over our logs?

          In insecure HBase, User.getCurrent() should return the operating system user. So I don't think this should show up in normal circumstances with either insecure or secure versions.

          On 2011-11-17 23:03:06, Michael Stack wrote:

          > src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java, line 359

          > <https://reviews.apache.org/r/1991/diff/6/?file=59024#file59024line359>

          >

          > Say what to use instead?

          Ted pointed this out too. Fixing to reference RpcEngine.call().

          On 2011-11-17 23:03:06, Michael Stack wrote:

          > src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java, line 119

          > <https://reviews.apache.org/r/1991/diff/6/?file=59026#file59026line119>

          >

          > This seems like an odd one to expose to subclasses.

          Yeah, must have been necessary at one point, but I no longer see any usage that requires it. Changing back to private.

          On 2011-11-17 23:03:06, Michael Stack wrote:

          > src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java, line 826

          > <https://reviews.apache.org/r/1991/diff/6/?file=59026#file59026line826>

          >

          > Is this 'fix' copied from hadoop? Seems sensible.

          Yes, copied from Hadoop. Made sense to me as well.

          On 2011-11-17 23:03:06, Michael Stack wrote:

          > src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java, line 44

          > <https://reviews.apache.org/r/1991/diff/6/?file=59027#file59027line44>

          >

          > Whats happening here? Do we have to do this – security pollution of base hbase interface?

          >

          Yes, these annotations are required for the connection authorization process and authentication. I copied over the annotation definitions to org.apache.hadoop.hbase.security as part of core. But actually removing them and using another means might take some work.

          On 2011-11-17 23:03:06, Michael Stack wrote:

          > src/main/java/org/apache/hadoop/hbase/master/HMaster.java, line 256

          > <https://reviews.apache.org/r/1991/diff/6/?file=59035#file59035line256>

          >

          > this a new call we have to make?

          This was moved down to after ZooKeeperWatcher setup so that AuthenticationTokenSecretManager (which needs zk) can be setup correctly in SecureServer.

          On 2011-11-17 23:03:06, Michael Stack wrote:

          > src/main/resources/hbase-default.xml, line 468

          > <https://reviews.apache.org/r/1991/diff/6/?file=59041#file59041line468>

          >

          > This stuff should be in here and not in an hbase-security-default.xml that comes in only when running security?

          >

          > Its ok this stuff is here... just wondering if could be shown only when a security context.

          Hmm, looks like the keytab stuff got duplicated. It's already there from previous patches for running on secure Hadoop.

          The hbase.auth.*, hbase.policy.file, and hbase.superuser properties are new. I guess we could split them out into a separate default file and add it to the resources in HBaseConfiguration. How important do you think it is?

          • Gary

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/1991/#review3322
          -----------------------------------------------------------

          On 2011-11-17 07:27:20, Gary Helmling wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/1991/

          -----------------------------------------------------------

          (Updated 2011-11-17 07:27:20)

          Review request for hbase.

          Summary

          -------

          This patch creates a new secure RPC engine for HBase, which provides Kerberos based authentication of clients, and a token-based authentication mechanism for mapreduce jobs. Primary components of the patch are:

          - a new maven profile for secure Hadoop/HBase: hadoop-0.20S

          - Secure Hadoop dependent classes are separated under a pseudo-module in the security/ directory. These source and test directories are only including if building the secure Hadoop profile

          - Currently the security classes get packaged with the regular HBase build artifacts. We need a way to at least override project.version, so we can append something like a "-security" suffix indicating the additional security components.

          - The pseudo-module here is really a half-step forward. It enables the security code to be optionally included in the build for now, and sets up the structure for a security module. But we still will want to pursue full modularization (see HBASE-4336), which will allow packing the security code in a separate build artifact.

          - a new RPC engine providing kerberos and token-based authentication: org.apache.hadoop.hbase.ipc.SecureRpcEngine

          - implementation under security/src/main/java/org/apache/hadoop/hbase/ipc/

          - The implementation classes extend the existing HBaseClient and HBaseServer to share as much of the RPC code as possible. The main override is of the connection classes to allow control over the SASL negotiation of secure connections

          - existing RPC changes

          - The existing HBaseClient and HBaseServer have been modified to make subclassing possible

          - All references to Hadoop UserGroupInformation have been replaced with org.apache.hadoop.hbase.security.User to insulate from future dependencies on specific Hadoop versions

          - a coprocessor endpoint for obtaining new authentication tokens: TokenProvider, and supporting classes for token generation and synchronization (incorporating HBASE-3615)

          - implementation is under security/src/main/java/org/apache/hadoop/hbase/security/token/

          - Secret keys for token generation and verification are synchronized throughout the cluster in zookeeper, under /hbase/tokenauth/keys

          To enable secure RPC, add the following configuration to hbase-site.xml:

          <property>

          <name>hadoop.security.authorization</name>

          <value>true</value>

          </property>

          <property>

          <name>hadoop.security.authentication</name>

          <value>kerberos</value>

          </property>

          <property>

          <name>hbase.rpc.engine</name>

          <value>org.apache.hadoop.hbase.ipc.SecureRpcEngine</value>

          </property>

          <property>

          <name>hbase.coprocessor.region.classes</name>

          <value>org.apache.hadoop.hbase.security.token.TokenProvider</value>

          </property>

          In addition, the master and regionserver processes must be configured for kerberos authentication using the properties:

          * hbase.(master|regionserver).keytab.file

          * hbase.(master|regionserver).kerberos.principal

          * hbase.(master|regionserver).kerberos.https.principal

          This addresses bug HBASE-2742.

          https://issues.apache.org/jira/browse/HBASE-2742

          Diffs

          -----

          conf/hbase-policy.xml PRE-CREATION

          pom.xml 2847416

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureConnectionHeader.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationKey.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationProtocol.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenIdentifier.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSelector.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/TokenProvider.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/ZKSecretWatcher.java PRE-CREATION

          security/src/test/java/org/apache/hadoop/hbase/security/token/TestTokenAuthentication.java PRE-CREATION

          security/src/test/java/org/apache/hadoop/hbase/security/token/TestZKSecretWatcher.java PRE-CREATION

          security/src/test/resources/hbase-site.xml PRE-CREATION

          src/assembly/all.xml 3ad8ab3

          src/main/java/org/apache/hadoop/hbase/HServerAddress.java f28240c

          src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java cba7bd1

          src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java 904c107

          src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 4086829

          src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java 054b92b

          src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 337da78

          src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java d9ba0ea

          src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java 9e0535e

          src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java 0dc5bea

          src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java e8b2c9e

          src/main/java/org/apache/hadoop/hbase/ipc/RequestContext.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java 1b5629a

          src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java c8e030c

          src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java db07ed1

          src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java ad88b76

          src/main/java/org/apache/hadoop/hbase/master/HMaster.java 19ed5b8

          src/main/java/org/apache/hadoop/hbase/security/KerberosInfo.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/security/TokenInfo.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/security/User.java 00bd06d

          src/main/java/org/apache/hadoop/hbase/zookeeper/ZKLeaderManager.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java 4c5e844

          src/main/resources/hbase-default.xml 6f98f5d

          src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java c0634f4

          src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java bb9ea45

          src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKLeaderManager.java PRE-CREATION

          Diff: https://reviews.apache.org/r/1991/diff

          Testing

          -------

          A full test suite run showed failures in TestMasterFailover (timeout) and TestAvroServer. I'll look into those.

          Cluster testing (1 master + 3 slaves in ec2) with Kerberos authentication and map reduce token authentication, using YCSB, RowCounter, PerformanceEvaluation.

          Thanks,

          Gary

          Show
          jiraposter@reviews.apache.org added a comment - On 2011-11-17 23:03:06, Michael Stack wrote: > pom.xml, line 408 > < https://reviews.apache.org/r/1991/diff/6/?file=58999#file58999line408 > > > Whats going on here? Why would we exclude the test hbase-site.xml? I thought it had all configs. for test running? Or is this in the security profile? I see you then add it back in later under the plugin section. Did we have it in wrong place? This is so that, when building with security profile, we can override the hbase-site.xml used for tests – specifically set it to use SecureRpcEngine. It's an ugly change to the POM and kind of ugly to duplicate the test hbase-site.xml, but I didn't see an easier way of doing it. I'm open to other suggestions to improve it. On 2011-11-17 23:03:06, Michael Stack wrote: > security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java, line 355 > < https://reviews.apache.org/r/1991/diff/6/?file=59000#file59000line355 > > > This is a number? Yes: SUCCESS (0), ERROR (1), FATAL (-1); On 2011-11-17 23:03:06, Michael Stack wrote: > security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java, line 369 > < https://reviews.apache.org/r/1991/diff/6/?file=59000#file59000line369 > > > Could we get an enum we don't know about? > > If copied from hadoop code, don't worry about this comment. Yeah this is direct from Hadoop. It is a bit odd not to have a default block, which should probably behave same as fatal. Depends if we want to diverge more by adding it. On 2011-11-17 23:03:06, Michael Stack wrote: > security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java, line 375 > < https://reviews.apache.org/r/1991/diff/6/?file=59000#file59000line375 > > > Man, this is kinda ugly... its a data member in a super class? Cryptic! > > (This is hadoop code?) This is actually from HBaseClient. Hadoop just does: } catch (IOException e) { markClosed(e); } On 2011-11-17 23:03:06, Michael Stack wrote: > security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java, line 143 > < https://reviews.apache.org/r/1991/diff/6/?file=59003#file59003line143 > > > Funny. Above we compare on State.SUCCESS.state. What if there is a FATAL here? Anything special we should do? In SecureClient, I think the check on State.SUCCESS.state is to avoid mapping the deserialized value back to the enum value. In the case of FATAL, the client will still see fatal status written separately in the response. For errorClass and error, FATAL is equivalent to ERROR. On 2011-11-17 23:03:06, Michael Stack wrote: > src/assembly/all.xml, line 60 > < https://reviews.apache.org/r/1991/diff/6/?file=59019#file59019line60 > > > This will add the 'security' on the end if we are building w/ the security profile? Yes, so for say 0.92, when building with -P security, you would wind up with hbase-0.92.0-security.jar. Without the security profile, you just get hbase-0.92.0.jar. This at least saves having to fiddle with the project version for secure vs. insecure builds, and gives artifacts named as expected. But not sure if it complicates deploying to a maven repo? It might still need more tweaking to improve how it works. On 2011-11-17 23:03:06, Michael Stack wrote: > src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java, line 399 > < https://reviews.apache.org/r/1991/diff/6/?file=59021#file59021line399 > > > So, in insecure hbase, is user provided? If not, this message will be all over our logs? In insecure HBase, User.getCurrent() should return the operating system user. So I don't think this should show up in normal circumstances with either insecure or secure versions. On 2011-11-17 23:03:06, Michael Stack wrote: > src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java, line 359 > < https://reviews.apache.org/r/1991/diff/6/?file=59024#file59024line359 > > > Say what to use instead? Ted pointed this out too. Fixing to reference RpcEngine.call(). On 2011-11-17 23:03:06, Michael Stack wrote: > src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java, line 119 > < https://reviews.apache.org/r/1991/diff/6/?file=59026#file59026line119 > > > This seems like an odd one to expose to subclasses. Yeah, must have been necessary at one point, but I no longer see any usage that requires it. Changing back to private. On 2011-11-17 23:03:06, Michael Stack wrote: > src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java, line 826 > < https://reviews.apache.org/r/1991/diff/6/?file=59026#file59026line826 > > > Is this 'fix' copied from hadoop? Seems sensible. Yes, copied from Hadoop. Made sense to me as well. On 2011-11-17 23:03:06, Michael Stack wrote: > src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java, line 44 > < https://reviews.apache.org/r/1991/diff/6/?file=59027#file59027line44 > > > Whats happening here? Do we have to do this – security pollution of base hbase interface? > Yes, these annotations are required for the connection authorization process and authentication. I copied over the annotation definitions to org.apache.hadoop.hbase.security as part of core. But actually removing them and using another means might take some work. On 2011-11-17 23:03:06, Michael Stack wrote: > src/main/java/org/apache/hadoop/hbase/master/HMaster.java, line 256 > < https://reviews.apache.org/r/1991/diff/6/?file=59035#file59035line256 > > > this a new call we have to make? This was moved down to after ZooKeeperWatcher setup so that AuthenticationTokenSecretManager (which needs zk) can be setup correctly in SecureServer. On 2011-11-17 23:03:06, Michael Stack wrote: > src/main/resources/hbase-default.xml, line 468 > < https://reviews.apache.org/r/1991/diff/6/?file=59041#file59041line468 > > > This stuff should be in here and not in an hbase-security-default.xml that comes in only when running security? > > Its ok this stuff is here... just wondering if could be shown only when a security context. Hmm, looks like the keytab stuff got duplicated. It's already there from previous patches for running on secure Hadoop. The hbase.auth.*, hbase.policy.file, and hbase.superuser properties are new. I guess we could split them out into a separate default file and add it to the resources in HBaseConfiguration. How important do you think it is? Gary ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1991/#review3322 ----------------------------------------------------------- On 2011-11-17 07:27:20, Gary Helmling wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1991/ ----------------------------------------------------------- (Updated 2011-11-17 07:27:20) Review request for hbase. Summary ------- This patch creates a new secure RPC engine for HBase, which provides Kerberos based authentication of clients, and a token-based authentication mechanism for mapreduce jobs. Primary components of the patch are: - a new maven profile for secure Hadoop/HBase: hadoop-0.20S - Secure Hadoop dependent classes are separated under a pseudo-module in the security/ directory. These source and test directories are only including if building the secure Hadoop profile - Currently the security classes get packaged with the regular HBase build artifacts. We need a way to at least override project.version, so we can append something like a "-security" suffix indicating the additional security components. - The pseudo-module here is really a half-step forward. It enables the security code to be optionally included in the build for now, and sets up the structure for a security module. But we still will want to pursue full modularization (see HBASE-4336 ), which will allow packing the security code in a separate build artifact. - a new RPC engine providing kerberos and token-based authentication: org.apache.hadoop.hbase.ipc.SecureRpcEngine - implementation under security/src/main/java/org/apache/hadoop/hbase/ipc/ - The implementation classes extend the existing HBaseClient and HBaseServer to share as much of the RPC code as possible. The main override is of the connection classes to allow control over the SASL negotiation of secure connections - existing RPC changes - The existing HBaseClient and HBaseServer have been modified to make subclassing possible - All references to Hadoop UserGroupInformation have been replaced with org.apache.hadoop.hbase.security.User to insulate from future dependencies on specific Hadoop versions - a coprocessor endpoint for obtaining new authentication tokens: TokenProvider, and supporting classes for token generation and synchronization (incorporating HBASE-3615 ) - implementation is under security/src/main/java/org/apache/hadoop/hbase/security/token/ - Secret keys for token generation and verification are synchronized throughout the cluster in zookeeper, under /hbase/tokenauth/keys To enable secure RPC, add the following configuration to hbase-site.xml: <property> <name>hadoop.security.authorization</name> <value>true</value> </property> <property> <name>hadoop.security.authentication</name> <value>kerberos</value> </property> <property> <name>hbase.rpc.engine</name> <value>org.apache.hadoop.hbase.ipc.SecureRpcEngine</value> </property> <property> <name>hbase.coprocessor.region.classes</name> <value>org.apache.hadoop.hbase.security.token.TokenProvider</value> </property> In addition, the master and regionserver processes must be configured for kerberos authentication using the properties: * hbase.(master|regionserver).keytab.file * hbase.(master|regionserver).kerberos.principal * hbase.(master|regionserver).kerberos.https.principal This addresses bug HBASE-2742 . https://issues.apache.org/jira/browse/HBASE-2742 Diffs ----- conf/hbase-policy.xml PRE-CREATION pom.xml 2847416 security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/SecureConnectionHeader.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationKey.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationProtocol.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenIdentifier.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSelector.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/TokenProvider.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/ZKSecretWatcher.java PRE-CREATION security/src/test/java/org/apache/hadoop/hbase/security/token/TestTokenAuthentication.java PRE-CREATION security/src/test/java/org/apache/hadoop/hbase/security/token/TestZKSecretWatcher.java PRE-CREATION security/src/test/resources/hbase-site.xml PRE-CREATION src/assembly/all.xml 3ad8ab3 src/main/java/org/apache/hadoop/hbase/HServerAddress.java f28240c src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java cba7bd1 src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java 904c107 src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 4086829 src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java 054b92b src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 337da78 src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java d9ba0ea src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java 9e0535e src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java 0dc5bea src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java e8b2c9e src/main/java/org/apache/hadoop/hbase/ipc/RequestContext.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java 1b5629a src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java c8e030c src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java db07ed1 src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java ad88b76 src/main/java/org/apache/hadoop/hbase/master/HMaster.java 19ed5b8 src/main/java/org/apache/hadoop/hbase/security/KerberosInfo.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/security/TokenInfo.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/security/User.java 00bd06d src/main/java/org/apache/hadoop/hbase/zookeeper/ZKLeaderManager.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java 4c5e844 src/main/resources/hbase-default.xml 6f98f5d src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java c0634f4 src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java bb9ea45 src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKLeaderManager.java PRE-CREATION Diff: https://reviews.apache.org/r/1991/diff Testing ------- A full test suite run showed failures in TestMasterFailover (timeout) and TestAvroServer. I'll look into those. Cluster testing (1 master + 3 slaves in ec2) with Kerberos authentication and map reduce token authentication, using YCSB, RowCounter, PerformanceEvaluation. Thanks, Gary
          Hide
          jiraposter@reviews.apache.org added a comment -

          On 2011-11-17 23:03:06, Michael Stack wrote:

          > pom.xml, line 408

          > <https://reviews.apache.org/r/1991/diff/6/?file=58999#file58999line408>

          >

          > Whats going on here? Why would we exclude the test hbase-site.xml? I thought it had all configs. for test running? Or is this in the security profile? I see you then add it back in later under the plugin section. Did we have it in wrong place?

          Gary Helmling wrote:

          This is so that, when building with security profile, we can override the hbase-site.xml used for tests – specifically set it to use SecureRpcEngine. It's an ugly change to the POM and kind of ugly to duplicate the test hbase-site.xml, but I didn't see an easier way of doing it. I'm open to other suggestions to improve it.

          Its fine.

          On 2011-11-17 23:03:06, Michael Stack wrote:

          > security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java, line 369

          > <https://reviews.apache.org/r/1991/diff/6/?file=59000#file59000line369>

          >

          > Could we get an enum we don't know about?

          >

          > If copied from hadoop code, don't worry about this comment.

          Gary Helmling wrote:

          Yeah this is direct from Hadoop. It is a bit odd not to have a default block, which should probably behave same as fatal. Depends if we want to diverge more by adding it.

          No to diverging.

          On 2011-11-17 23:03:06, Michael Stack wrote:

          > src/assembly/all.xml, line 60

          > <https://reviews.apache.org/r/1991/diff/6/?file=59019#file59019line60>

          >

          > This will add the 'security' on the end if we are building w/ the security profile?

          Gary Helmling wrote:

          Yes, so for say 0.92, when building with -P security, you would wind up with hbase-0.92.0-security.jar. Without the security profile, you just get hbase-0.92.0.jar.

          This at least saves having to fiddle with the project version for secure vs. insecure builds, and gives artifacts named as expected. But not sure if it complicates deploying to a maven repo? It might still need more tweaking to improve how it works.

          Ok

          On 2011-11-17 23:03:06, Michael Stack wrote:

          > src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java, line 399

          > <https://reviews.apache.org/r/1991/diff/6/?file=59021#file59021line399>

          >

          > So, in insecure hbase, is user provided? If not, this message will be all over our logs?

          Gary Helmling wrote:

          In insecure HBase, User.getCurrent() should return the operating system user. So I don't think this should show up in normal circumstances with either insecure or secure versions.

          ok

          On 2011-11-17 23:03:06, Michael Stack wrote:

          > src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java, line 44

          > <https://reviews.apache.org/r/1991/diff/6/?file=59027#file59027line44>

          >

          > Whats happening here? Do we have to do this – security pollution of base hbase interface?

          >

          Gary Helmling wrote:

          Yes, these annotations are required for the connection authorization process and authentication. I copied over the annotation definitions to org.apache.hadoop.hbase.security as part of core. But actually removing them and using another means might take some work.

          ok. we'll live with it.

          On 2011-11-17 23:03:06, Michael Stack wrote:

          > src/main/resources/hbase-default.xml, line 468

          > <https://reviews.apache.org/r/1991/diff/6/?file=59041#file59041line468>

          >

          > This stuff should be in here and not in an hbase-security-default.xml that comes in only when running security?

          >

          > Its ok this stuff is here... just wondering if could be shown only when a security context.

          Gary Helmling wrote:

          Hmm, looks like the keytab stuff got duplicated. It's already there from previous patches for running on secure Hadoop.

          The hbase.auth.*, hbase.policy.file, and hbase.superuser properties are new. I guess we could split them out into a separate default file and add it to the resources in HBaseConfiguration. How important do you think it is?

          We can do that later. You've done enough gymnastics Gary getting security in w/o too much disruption to core. Maybe add note that these configs are for secure hbase only if its not obvious already?

          • Michael

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/1991/#review3322
          -----------------------------------------------------------

          On 2011-11-17 07:27:20, Gary Helmling wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/1991/

          -----------------------------------------------------------

          (Updated 2011-11-17 07:27:20)

          Review request for hbase.

          Summary

          -------

          This patch creates a new secure RPC engine for HBase, which provides Kerberos based authentication of clients, and a token-based authentication mechanism for mapreduce jobs. Primary components of the patch are:

          - a new maven profile for secure Hadoop/HBase: hadoop-0.20S

          - Secure Hadoop dependent classes are separated under a pseudo-module in the security/ directory. These source and test directories are only including if building the secure Hadoop profile

          - Currently the security classes get packaged with the regular HBase build artifacts. We need a way to at least override project.version, so we can append something like a "-security" suffix indicating the additional security components.

          - The pseudo-module here is really a half-step forward. It enables the security code to be optionally included in the build for now, and sets up the structure for a security module. But we still will want to pursue full modularization (see HBASE-4336), which will allow packing the security code in a separate build artifact.

          - a new RPC engine providing kerberos and token-based authentication: org.apache.hadoop.hbase.ipc.SecureRpcEngine

          - implementation under security/src/main/java/org/apache/hadoop/hbase/ipc/

          - The implementation classes extend the existing HBaseClient and HBaseServer to share as much of the RPC code as possible. The main override is of the connection classes to allow control over the SASL negotiation of secure connections

          - existing RPC changes

          - The existing HBaseClient and HBaseServer have been modified to make subclassing possible

          - All references to Hadoop UserGroupInformation have been replaced with org.apache.hadoop.hbase.security.User to insulate from future dependencies on specific Hadoop versions

          - a coprocessor endpoint for obtaining new authentication tokens: TokenProvider, and supporting classes for token generation and synchronization (incorporating HBASE-3615)

          - implementation is under security/src/main/java/org/apache/hadoop/hbase/security/token/

          - Secret keys for token generation and verification are synchronized throughout the cluster in zookeeper, under /hbase/tokenauth/keys

          To enable secure RPC, add the following configuration to hbase-site.xml:

          <property>

          <name>hadoop.security.authorization</name>

          <value>true</value>

          </property>

          <property>

          <name>hadoop.security.authentication</name>

          <value>kerberos</value>

          </property>

          <property>

          <name>hbase.rpc.engine</name>

          <value>org.apache.hadoop.hbase.ipc.SecureRpcEngine</value>

          </property>

          <property>

          <name>hbase.coprocessor.region.classes</name>

          <value>org.apache.hadoop.hbase.security.token.TokenProvider</value>

          </property>

          In addition, the master and regionserver processes must be configured for kerberos authentication using the properties:

          * hbase.(master|regionserver).keytab.file

          * hbase.(master|regionserver).kerberos.principal

          * hbase.(master|regionserver).kerberos.https.principal

          This addresses bug HBASE-2742.

          https://issues.apache.org/jira/browse/HBASE-2742

          Diffs

          -----

          conf/hbase-policy.xml PRE-CREATION

          pom.xml 2847416

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureConnectionHeader.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationKey.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationProtocol.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenIdentifier.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSelector.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/TokenProvider.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java PRE-CREATION

          security/src/main/java/org/apache/hadoop/hbase/security/token/ZKSecretWatcher.java PRE-CREATION

          security/src/test/java/org/apache/hadoop/hbase/security/token/TestTokenAuthentication.java PRE-CREATION

          security/src/test/java/org/apache/hadoop/hbase/security/token/TestZKSecretWatcher.java PRE-CREATION

          security/src/test/resources/hbase-site.xml PRE-CREATION

          src/assembly/all.xml 3ad8ab3

          src/main/java/org/apache/hadoop/hbase/HServerAddress.java f28240c

          src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java cba7bd1

          src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java 904c107

          src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 4086829

          src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java 054b92b

          src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 337da78

          src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java d9ba0ea

          src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java 9e0535e

          src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java 0dc5bea

          src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java e8b2c9e

          src/main/java/org/apache/hadoop/hbase/ipc/RequestContext.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java 1b5629a

          src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java c8e030c

          src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java db07ed1

          src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java ad88b76

          src/main/java/org/apache/hadoop/hbase/master/HMaster.java 19ed5b8

          src/main/java/org/apache/hadoop/hbase/security/KerberosInfo.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/security/TokenInfo.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/security/User.java 00bd06d

          src/main/java/org/apache/hadoop/hbase/zookeeper/ZKLeaderManager.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java 4c5e844

          src/main/resources/hbase-default.xml 6f98f5d

          src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java c0634f4

          src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java bb9ea45

          src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKLeaderManager.java PRE-CREATION

          Diff: https://reviews.apache.org/r/1991/diff

          Testing

          -------

          A full test suite run showed failures in TestMasterFailover (timeout) and TestAvroServer. I'll look into those.

          Cluster testing (1 master + 3 slaves in ec2) with Kerberos authentication and map reduce token authentication, using YCSB, RowCounter, PerformanceEvaluation.

          Thanks,

          Gary

          Show
          jiraposter@reviews.apache.org added a comment - On 2011-11-17 23:03:06, Michael Stack wrote: > pom.xml, line 408 > < https://reviews.apache.org/r/1991/diff/6/?file=58999#file58999line408 > > > Whats going on here? Why would we exclude the test hbase-site.xml? I thought it had all configs. for test running? Or is this in the security profile? I see you then add it back in later under the plugin section. Did we have it in wrong place? Gary Helmling wrote: This is so that, when building with security profile, we can override the hbase-site.xml used for tests – specifically set it to use SecureRpcEngine. It's an ugly change to the POM and kind of ugly to duplicate the test hbase-site.xml, but I didn't see an easier way of doing it. I'm open to other suggestions to improve it. Its fine. On 2011-11-17 23:03:06, Michael Stack wrote: > security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java, line 369 > < https://reviews.apache.org/r/1991/diff/6/?file=59000#file59000line369 > > > Could we get an enum we don't know about? > > If copied from hadoop code, don't worry about this comment. Gary Helmling wrote: Yeah this is direct from Hadoop. It is a bit odd not to have a default block, which should probably behave same as fatal. Depends if we want to diverge more by adding it. No to diverging. On 2011-11-17 23:03:06, Michael Stack wrote: > src/assembly/all.xml, line 60 > < https://reviews.apache.org/r/1991/diff/6/?file=59019#file59019line60 > > > This will add the 'security' on the end if we are building w/ the security profile? Gary Helmling wrote: Yes, so for say 0.92, when building with -P security, you would wind up with hbase-0.92.0-security.jar. Without the security profile, you just get hbase-0.92.0.jar. This at least saves having to fiddle with the project version for secure vs. insecure builds, and gives artifacts named as expected. But not sure if it complicates deploying to a maven repo? It might still need more tweaking to improve how it works. Ok On 2011-11-17 23:03:06, Michael Stack wrote: > src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java, line 399 > < https://reviews.apache.org/r/1991/diff/6/?file=59021#file59021line399 > > > So, in insecure hbase, is user provided? If not, this message will be all over our logs? Gary Helmling wrote: In insecure HBase, User.getCurrent() should return the operating system user. So I don't think this should show up in normal circumstances with either insecure or secure versions. ok On 2011-11-17 23:03:06, Michael Stack wrote: > src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java, line 44 > < https://reviews.apache.org/r/1991/diff/6/?file=59027#file59027line44 > > > Whats happening here? Do we have to do this – security pollution of base hbase interface? > Gary Helmling wrote: Yes, these annotations are required for the connection authorization process and authentication. I copied over the annotation definitions to org.apache.hadoop.hbase.security as part of core. But actually removing them and using another means might take some work. ok. we'll live with it. On 2011-11-17 23:03:06, Michael Stack wrote: > src/main/resources/hbase-default.xml, line 468 > < https://reviews.apache.org/r/1991/diff/6/?file=59041#file59041line468 > > > This stuff should be in here and not in an hbase-security-default.xml that comes in only when running security? > > Its ok this stuff is here... just wondering if could be shown only when a security context. Gary Helmling wrote: Hmm, looks like the keytab stuff got duplicated. It's already there from previous patches for running on secure Hadoop. The hbase.auth.*, hbase.policy.file, and hbase.superuser properties are new. I guess we could split them out into a separate default file and add it to the resources in HBaseConfiguration. How important do you think it is? We can do that later. You've done enough gymnastics Gary getting security in w/o too much disruption to core. Maybe add note that these configs are for secure hbase only if its not obvious already? Michael ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1991/#review3322 ----------------------------------------------------------- On 2011-11-17 07:27:20, Gary Helmling wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1991/ ----------------------------------------------------------- (Updated 2011-11-17 07:27:20) Review request for hbase. Summary ------- This patch creates a new secure RPC engine for HBase, which provides Kerberos based authentication of clients, and a token-based authentication mechanism for mapreduce jobs. Primary components of the patch are: - a new maven profile for secure Hadoop/HBase: hadoop-0.20S - Secure Hadoop dependent classes are separated under a pseudo-module in the security/ directory. These source and test directories are only including if building the secure Hadoop profile - Currently the security classes get packaged with the regular HBase build artifacts. We need a way to at least override project.version, so we can append something like a "-security" suffix indicating the additional security components. - The pseudo-module here is really a half-step forward. It enables the security code to be optionally included in the build for now, and sets up the structure for a security module. But we still will want to pursue full modularization (see HBASE-4336 ), which will allow packing the security code in a separate build artifact. - a new RPC engine providing kerberos and token-based authentication: org.apache.hadoop.hbase.ipc.SecureRpcEngine - implementation under security/src/main/java/org/apache/hadoop/hbase/ipc/ - The implementation classes extend the existing HBaseClient and HBaseServer to share as much of the RPC code as possible. The main override is of the connection classes to allow control over the SASL negotiation of secure connections - existing RPC changes - The existing HBaseClient and HBaseServer have been modified to make subclassing possible - All references to Hadoop UserGroupInformation have been replaced with org.apache.hadoop.hbase.security.User to insulate from future dependencies on specific Hadoop versions - a coprocessor endpoint for obtaining new authentication tokens: TokenProvider, and supporting classes for token generation and synchronization (incorporating HBASE-3615 ) - implementation is under security/src/main/java/org/apache/hadoop/hbase/security/token/ - Secret keys for token generation and verification are synchronized throughout the cluster in zookeeper, under /hbase/tokenauth/keys To enable secure RPC, add the following configuration to hbase-site.xml: <property> <name>hadoop.security.authorization</name> <value>true</value> </property> <property> <name>hadoop.security.authentication</name> <value>kerberos</value> </property> <property> <name>hbase.rpc.engine</name> <value>org.apache.hadoop.hbase.ipc.SecureRpcEngine</value> </property> <property> <name>hbase.coprocessor.region.classes</name> <value>org.apache.hadoop.hbase.security.token.TokenProvider</value> </property> In addition, the master and regionserver processes must be configured for kerberos authentication using the properties: * hbase.(master|regionserver).keytab.file * hbase.(master|regionserver).kerberos.principal * hbase.(master|regionserver).kerberos.https.principal This addresses bug HBASE-2742 . https://issues.apache.org/jira/browse/HBASE-2742 Diffs ----- conf/hbase-policy.xml PRE-CREATION pom.xml 2847416 security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/SecureConnectionHeader.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationKey.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationProtocol.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenIdentifier.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSelector.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/TokenProvider.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/ZKSecretWatcher.java PRE-CREATION security/src/test/java/org/apache/hadoop/hbase/security/token/TestTokenAuthentication.java PRE-CREATION security/src/test/java/org/apache/hadoop/hbase/security/token/TestZKSecretWatcher.java PRE-CREATION security/src/test/resources/hbase-site.xml PRE-CREATION src/assembly/all.xml 3ad8ab3 src/main/java/org/apache/hadoop/hbase/HServerAddress.java f28240c src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java cba7bd1 src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java 904c107 src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 4086829 src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java 054b92b src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 337da78 src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java d9ba0ea src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java 9e0535e src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java 0dc5bea src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java e8b2c9e src/main/java/org/apache/hadoop/hbase/ipc/RequestContext.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java 1b5629a src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java c8e030c src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java db07ed1 src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java ad88b76 src/main/java/org/apache/hadoop/hbase/master/HMaster.java 19ed5b8 src/main/java/org/apache/hadoop/hbase/security/KerberosInfo.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/security/TokenInfo.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/security/User.java 00bd06d src/main/java/org/apache/hadoop/hbase/zookeeper/ZKLeaderManager.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java 4c5e844 src/main/resources/hbase-default.xml 6f98f5d src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java c0634f4 src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java bb9ea45 src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKLeaderManager.java PRE-CREATION Diff: https://reviews.apache.org/r/1991/diff Testing ------- A full test suite run showed failures in TestMasterFailover (timeout) and TestAvroServer. I'll look into those. Cluster testing (1 master + 3 slaves in ec2) with Kerberos authentication and map reduce token authentication, using YCSB, RowCounter, PerformanceEvaluation. Thanks, Gary
          Hide
          Gary Helmling added a comment -

          Thanks for the review! Please add additional comments to review board. In a patch this big, it's much easier for me to keep track of comments and replies there.


          Looking at AuthenticationTokenSecretManager, it becomes leader upon entering run().
          Periodically it calls rollCurrentKey() which checks whether it is still leader. I think rollCurrentKey() should return a boolean value indicating leadership status so that the while loop can re-obtain leadership if leadership is lost.

          Leader election for AuthenticationTokenSecretManager is equivalent to active master management. In run(), all instances block on zkLeader.waitToBecomeLeader(). Only the leader returns. The leader remains active until stopped or the zk session is lost (which should cause an abort), so I don't see where we would need to recheck in this case?


          In my opinion, failure to remove expired key or to add new key should be treated with more caution.
          addKeyToZK() and updateKeyInZK() throws RuntimeException, alleviating this concern a little. But zkWatcher.removeKeyFromZK() swallows KeeperException, making things complicated. We may need to call ZKUtil.deleteNode() directly.

          Agree, removeKeyFromZK() should be aborting on KeeperException. That was missing in other places as well. Cleaning up to call watch.abort() on KeeperException, as well as replacing the RuntimeExceptions with watcher.abort().

          I am very hesitant, though, to call ZKUtil.deleteNode() in the case of IOExceptions (which only occur from the Writable serialization). In the case of say an upgrade with changed serialization format or a znode that somehow got corrupted data, this seems appropriate. But I'm more worried about what would occur in the case of a massive misconfiguration, where zookeeper.znode.tokenauth.parent is pointing to the wrong path, and what the consequences might be from deleting every znode it can't deserialize. I'd rather abort the server in this case and let the end user deal with it.

          Some ZKWatcher methods shouldn't even have been throwing IOExceptions. Those have been fixed. The remaining cases (nodeDataChanged(), refreshNodes(), addKeyToZK(), updatedKeyInZK()) have been changed to abort on IOException since it would likely mean a programming error.


          In src/main/resources/hbase-default.xml, I think we should use dot instead of dash for parameter key names:
          hbase.auth.key.update-interval should be hbase.auth.key.update.interval
          hbase.auth.token.max-lifetime should be hbase.auth.token.max.lifetime

          These keys were named to match the keys used for HDFS delegation tokens:
          dfs.namenode.delegation.key.update-interval
          dfs.namenode.delegation.token.max-lifetime

          But, since we are consistent about not using '-' in HBase config I will replace.


          In ZKLeaderManager.stepDownAsLeader():

          ZKUtil.deleteNodeFailSilent(watcher, leaderZNode);

          I think we should call ZKUtil.deleteNode() instead and handle KeeperException gracefully because waitToBecomeLeader() would only delete zk node which has the other candidate's Id.

          I don't really see what ZKUtil.deleteNode() would improve in this context? If we instead got the KeeperException.NoNodeException that it exposes, what would we want to do differently? Since the point of stepDownAsLeader() is to step down, my opinion would be to ignore the NoNodeException if the znode has somehow been removed. The logic in waitToBecomeLeader() is completely separate and clones the logic in ActiveMasterManager.blockUntilBecomingActiveMaster().


          In HBaseServer.closeCurrentConnection(), the following is added:

          c = null;

          Since c is the attachment to SelectionKey, should we call key.attach(null) ?

          Good point. The c = null assignment is useless. Changing to key.attach(null).


          HBaseRPC.call() is marked deprecated. Probably we should mention SecureRpcEngine so that developers know what to look for.

          Added @depecated tag pointing to RpcEngine.call().

          Show
          Gary Helmling added a comment - Thanks for the review! Please add additional comments to review board. In a patch this big, it's much easier for me to keep track of comments and replies there. Looking at AuthenticationTokenSecretManager, it becomes leader upon entering run(). Periodically it calls rollCurrentKey() which checks whether it is still leader. I think rollCurrentKey() should return a boolean value indicating leadership status so that the while loop can re-obtain leadership if leadership is lost. Leader election for AuthenticationTokenSecretManager is equivalent to active master management. In run(), all instances block on zkLeader.waitToBecomeLeader(). Only the leader returns. The leader remains active until stopped or the zk session is lost (which should cause an abort), so I don't see where we would need to recheck in this case? In my opinion, failure to remove expired key or to add new key should be treated with more caution. addKeyToZK() and updateKeyInZK() throws RuntimeException, alleviating this concern a little. But zkWatcher.removeKeyFromZK() swallows KeeperException, making things complicated. We may need to call ZKUtil.deleteNode() directly. Agree, removeKeyFromZK() should be aborting on KeeperException. That was missing in other places as well. Cleaning up to call watch.abort() on KeeperException, as well as replacing the RuntimeExceptions with watcher.abort(). I am very hesitant, though, to call ZKUtil.deleteNode() in the case of IOExceptions (which only occur from the Writable serialization). In the case of say an upgrade with changed serialization format or a znode that somehow got corrupted data, this seems appropriate. But I'm more worried about what would occur in the case of a massive misconfiguration, where zookeeper.znode.tokenauth.parent is pointing to the wrong path, and what the consequences might be from deleting every znode it can't deserialize. I'd rather abort the server in this case and let the end user deal with it. Some ZKWatcher methods shouldn't even have been throwing IOExceptions. Those have been fixed. The remaining cases (nodeDataChanged(), refreshNodes(), addKeyToZK(), updatedKeyInZK()) have been changed to abort on IOException since it would likely mean a programming error. In src/main/resources/hbase-default.xml, I think we should use dot instead of dash for parameter key names: hbase.auth.key.update-interval should be hbase.auth.key.update.interval hbase.auth.token.max-lifetime should be hbase.auth.token.max.lifetime These keys were named to match the keys used for HDFS delegation tokens: dfs.namenode.delegation.key.update-interval dfs.namenode.delegation.token.max-lifetime But, since we are consistent about not using '-' in HBase config I will replace. In ZKLeaderManager.stepDownAsLeader(): ZKUtil.deleteNodeFailSilent(watcher, leaderZNode); I think we should call ZKUtil.deleteNode() instead and handle KeeperException gracefully because waitToBecomeLeader() would only delete zk node which has the other candidate's Id. I don't really see what ZKUtil.deleteNode() would improve in this context? If we instead got the KeeperException.NoNodeException that it exposes, what would we want to do differently? Since the point of stepDownAsLeader() is to step down, my opinion would be to ignore the NoNodeException if the znode has somehow been removed. The logic in waitToBecomeLeader() is completely separate and clones the logic in ActiveMasterManager.blockUntilBecomingActiveMaster(). In HBaseServer.closeCurrentConnection(), the following is added: c = null; Since c is the attachment to SelectionKey, should we call key.attach(null) ? Good point. The c = null assignment is useless. Changing to key.attach(null) . HBaseRPC.call() is marked deprecated. Probably we should mention SecureRpcEngine so that developers know what to look for. Added @depecated tag pointing to RpcEngine.call().
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/1991/
          -----------------------------------------------------------

          (Updated 2011-11-18 01:37:36.587135)

          Review request for hbase.

          Changes
          -------

          Updating to address Ted's comments in JIRA.

          Summary
          -------

          This patch creates a new secure RPC engine for HBase, which provides Kerberos based authentication of clients, and a token-based authentication mechanism for mapreduce jobs. Primary components of the patch are:

          • a new maven profile for secure Hadoop/HBase: hadoop-0.20S
          • Secure Hadoop dependent classes are separated under a pseudo-module in the security/ directory. These source and test directories are only including if building the secure Hadoop profile
          • Currently the security classes get packaged with the regular HBase build artifacts. We need a way to at least override project.version, so we can append something like a "-security" suffix indicating the additional security components.
          • The pseudo-module here is really a half-step forward. It enables the security code to be optionally included in the build for now, and sets up the structure for a security module. But we still will want to pursue full modularization (see HBASE-4336), which will allow packing the security code in a separate build artifact.
          • a new RPC engine providing kerberos and token-based authentication: org.apache.hadoop.hbase.ipc.SecureRpcEngine
          • implementation under security/src/main/java/org/apache/hadoop/hbase/ipc/
          • The implementation classes extend the existing HBaseClient and HBaseServer to share as much of the RPC code as possible. The main override is of the connection classes to allow control over the SASL negotiation of secure connections
          • existing RPC changes
          • The existing HBaseClient and HBaseServer have been modified to make subclassing possible
          • All references to Hadoop UserGroupInformation have been replaced with org.apache.hadoop.hbase.security.User to insulate from future dependencies on specific Hadoop versions
          • a coprocessor endpoint for obtaining new authentication tokens: TokenProvider, and supporting classes for token generation and synchronization (incorporating HBASE-3615)
          • implementation is under security/src/main/java/org/apache/hadoop/hbase/security/token/
          • Secret keys for token generation and verification are synchronized throughout the cluster in zookeeper, under /hbase/tokenauth/keys

          To enable secure RPC, add the following configuration to hbase-site.xml:

          <property>
          <name>hadoop.security.authorization</name>
          <value>true</value>
          </property>
          <property>
          <name>hadoop.security.authentication</name>
          <value>kerberos</value>
          </property>
          <property>
          <name>hbase.rpc.engine</name>
          <value>org.apache.hadoop.hbase.ipc.SecureRpcEngine</value>
          </property>
          <property>
          <name>hbase.coprocessor.region.classes</name>
          <value>org.apache.hadoop.hbase.security.token.TokenProvider</value>
          </property>

          In addition, the master and regionserver processes must be configured for kerberos authentication using the properties:

          • hbase.(master|regionserver).keytab.file
          • hbase.(master|regionserver).kerberos.principal
          • hbase.(master|regionserver).kerberos.https.principal

          This addresses bug HBASE-2742.
          https://issues.apache.org/jira/browse/HBASE-2742

          Diffs (updated)


          conf/hbase-policy.xml PRE-CREATION
          pom.xml 2847416
          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureConnectionHeader.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationKey.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationProtocol.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenIdentifier.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSelector.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/token/TokenProvider.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java PRE-CREATION
          security/src/main/java/org/apache/hadoop/hbase/security/token/ZKSecretWatcher.java PRE-CREATION
          security/src/test/java/org/apache/hadoop/hbase/security/token/TestTokenAuthentication.java PRE-CREATION
          security/src/test/java/org/apache/hadoop/hbase/security/token/TestZKSecretWatcher.java PRE-CREATION
          security/src/test/resources/hbase-site.xml PRE-CREATION
          src/assembly/all.xml 3ad8ab3
          src/main/java/org/apache/hadoop/hbase/HServerAddress.java f28240c
          src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java cba7bd1
          src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java 904c107
          src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 4086829
          src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java 054b92b
          src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 337da78
          src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java d9ba0ea
          src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java 9e0535e
          src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java 0dc5bea
          src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java e8b2c9e
          src/main/java/org/apache/hadoop/hbase/ipc/RequestContext.java PRE-CREATION
          src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java 1b5629a
          src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java c8e030c
          src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java db07ed1
          src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java ad88b76
          src/main/java/org/apache/hadoop/hbase/master/HMaster.java 19ed5b8
          src/main/java/org/apache/hadoop/hbase/security/KerberosInfo.java PRE-CREATION
          src/main/java/org/apache/hadoop/hbase/security/TokenInfo.java PRE-CREATION
          src/main/java/org/apache/hadoop/hbase/security/User.java 00bd06d
          src/main/java/org/apache/hadoop/hbase/zookeeper/ZKLeaderManager.java PRE-CREATION
          src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java 4c5e844
          src/main/resources/hbase-default.xml 6f98f5d
          src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java c0634f4
          src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java bb9ea45
          src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKLeaderManager.java PRE-CREATION

          Diff: https://reviews.apache.org/r/1991/diff

          Testing
          -------

          A full test suite run showed failures in TestMasterFailover (timeout) and TestAvroServer. I'll look into those.

          Cluster testing (1 master + 3 slaves in ec2) with Kerberos authentication and map reduce token authentication, using YCSB, RowCounter, PerformanceEvaluation.

          Thanks,

          Gary

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1991/ ----------------------------------------------------------- (Updated 2011-11-18 01:37:36.587135) Review request for hbase. Changes ------- Updating to address Ted's comments in JIRA. Summary ------- This patch creates a new secure RPC engine for HBase, which provides Kerberos based authentication of clients, and a token-based authentication mechanism for mapreduce jobs. Primary components of the patch are: a new maven profile for secure Hadoop/HBase: hadoop-0.20S Secure Hadoop dependent classes are separated under a pseudo-module in the security/ directory. These source and test directories are only including if building the secure Hadoop profile Currently the security classes get packaged with the regular HBase build artifacts. We need a way to at least override project.version, so we can append something like a "-security" suffix indicating the additional security components. The pseudo-module here is really a half-step forward. It enables the security code to be optionally included in the build for now, and sets up the structure for a security module. But we still will want to pursue full modularization (see HBASE-4336 ), which will allow packing the security code in a separate build artifact. a new RPC engine providing kerberos and token-based authentication: org.apache.hadoop.hbase.ipc.SecureRpcEngine implementation under security/src/main/java/org/apache/hadoop/hbase/ipc/ The implementation classes extend the existing HBaseClient and HBaseServer to share as much of the RPC code as possible. The main override is of the connection classes to allow control over the SASL negotiation of secure connections existing RPC changes The existing HBaseClient and HBaseServer have been modified to make subclassing possible All references to Hadoop UserGroupInformation have been replaced with org.apache.hadoop.hbase.security.User to insulate from future dependencies on specific Hadoop versions a coprocessor endpoint for obtaining new authentication tokens: TokenProvider, and supporting classes for token generation and synchronization (incorporating HBASE-3615 ) implementation is under security/src/main/java/org/apache/hadoop/hbase/security/token/ Secret keys for token generation and verification are synchronized throughout the cluster in zookeeper, under /hbase/tokenauth/keys To enable secure RPC, add the following configuration to hbase-site.xml: <property> <name>hadoop.security.authorization</name> <value>true</value> </property> <property> <name>hadoop.security.authentication</name> <value>kerberos</value> </property> <property> <name>hbase.rpc.engine</name> <value>org.apache.hadoop.hbase.ipc.SecureRpcEngine</value> </property> <property> <name>hbase.coprocessor.region.classes</name> <value>org.apache.hadoop.hbase.security.token.TokenProvider</value> </property> In addition, the master and regionserver processes must be configured for kerberos authentication using the properties: hbase.(master|regionserver).keytab.file hbase.(master|regionserver).kerberos.principal hbase.(master|regionserver).kerberos.https.principal This addresses bug HBASE-2742 . https://issues.apache.org/jira/browse/HBASE-2742 Diffs (updated) conf/hbase-policy.xml PRE-CREATION pom.xml 2847416 security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/SecureConnectionHeader.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationKey.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationProtocol.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenIdentifier.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSelector.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/TokenProvider.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/token/ZKSecretWatcher.java PRE-CREATION security/src/test/java/org/apache/hadoop/hbase/security/token/TestTokenAuthentication.java PRE-CREATION security/src/test/java/org/apache/hadoop/hbase/security/token/TestZKSecretWatcher.java PRE-CREATION security/src/test/resources/hbase-site.xml PRE-CREATION src/assembly/all.xml 3ad8ab3 src/main/java/org/apache/hadoop/hbase/HServerAddress.java f28240c src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java cba7bd1 src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java 904c107 src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 4086829 src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java 054b92b src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 337da78 src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java d9ba0ea src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java 9e0535e src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java 0dc5bea src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java e8b2c9e src/main/java/org/apache/hadoop/hbase/ipc/RequestContext.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java 1b5629a src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java c8e030c src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java db07ed1 src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java ad88b76 src/main/java/org/apache/hadoop/hbase/master/HMaster.java 19ed5b8 src/main/java/org/apache/hadoop/hbase/security/KerberosInfo.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/security/TokenInfo.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/security/User.java 00bd06d src/main/java/org/apache/hadoop/hbase/zookeeper/ZKLeaderManager.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java 4c5e844 src/main/resources/hbase-default.xml 6f98f5d src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java c0634f4 src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java bb9ea45 src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKLeaderManager.java PRE-CREATION Diff: https://reviews.apache.org/r/1991/diff Testing ------- A full test suite run showed failures in TestMasterFailover (timeout) and TestAvroServer. I'll look into those. Cluster testing (1 master + 3 slaves in ec2) with Kerberos authentication and map reduce token authentication, using YCSB, RowCounter, PerformanceEvaluation. Thanks, Gary
          Hide
          Ted Yu added a comment -

          I am fine with aborting watcher in stepDownAsLeader().

          +1 if tests pass.

          Show
          Ted Yu added a comment - I am fine with aborting watcher in stepDownAsLeader(). +1 if tests pass.
          Hide
          Gary Helmling added a comment -

          Latest patch from review board.

          Show
          Gary Helmling added a comment - Latest patch from review board.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12504164/HBASE-2742_10.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 23 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 javac. The applied patch generated 14 javac compiler warnings (more than the trunk's current 5 warnings).

          -1 findbugs. The patch appears to introduce 60 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.client.TestAdmin
          org.apache.hadoop.hbase.client.TestShell

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/288//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/288//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/288//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12504164/HBASE-2742_10.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 23 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 14 javac compiler warnings (more than the trunk's current 5 warnings). -1 findbugs. The patch appears to introduce 60 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.client.TestAdmin org.apache.hadoop.hbase.client.TestShell Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/288//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/288//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/288//console This message is automatically generated.
          Hide
          Gary Helmling added a comment -

          TestAdmin and TestShell failures seem to be unrelated, and both pass for me locally.

          Show
          Gary Helmling added a comment - TestAdmin and TestShell failures seem to be unrelated, and both pass for me locally.
          Hide
          Gary Helmling added a comment -

          Committed to 0.92 branch and trunk. Thanks for all the reviews Andy, Stack, and Ted! This was a big one.

          Show
          Gary Helmling added a comment - Committed to 0.92 branch and trunk. Thanks for all the reviews Andy, Stack, and Ted! This was a big one.
          Hide
          Hudson added a comment -

          Integrated in HBase-0.92 #143 (See https://builds.apache.org/job/HBase-0.92/143/)
          HBASE-2742 Provide strong authentication with a secure RPC engine

          garyh :
          Files :

          • /hbase/branches/0.92/CHANGES.txt
          • /hbase/branches/0.92/conf/hbase-policy.xml
          • /hbase/branches/0.92/pom.xml
          • /hbase/branches/0.92/security
          • /hbase/branches/0.92/security/src
          • /hbase/branches/0.92/security/src/main
          • /hbase/branches/0.92/security/src/main/java
          • /hbase/branches/0.92/security/src/main/java/org
          • /hbase/branches/0.92/security/src/main/java/org/apache
          • /hbase/branches/0.92/security/src/main/java/org/apache/hadoop
          • /hbase/branches/0.92/security/src/main/java/org/apache/hadoop/hbase
          • /hbase/branches/0.92/security/src/main/java/org/apache/hadoop/hbase/ipc
          • /hbase/branches/0.92/security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java
          • /hbase/branches/0.92/security/src/main/java/org/apache/hadoop/hbase/ipc/SecureConnectionHeader.java
          • /hbase/branches/0.92/security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java
          • /hbase/branches/0.92/security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java
          • /hbase/branches/0.92/security/src/main/java/org/apache/hadoop/hbase/security
          • /hbase/branches/0.92/security/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java
          • /hbase/branches/0.92/security/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java
          • /hbase/branches/0.92/security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java
          • /hbase/branches/0.92/security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java
          • /hbase/branches/0.92/security/src/main/java/org/apache/hadoop/hbase/security/token
          • /hbase/branches/0.92/security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationKey.java
          • /hbase/branches/0.92/security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationProtocol.java
          • /hbase/branches/0.92/security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenIdentifier.java
          • /hbase/branches/0.92/security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java
          • /hbase/branches/0.92/security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSelector.java
          • /hbase/branches/0.92/security/src/main/java/org/apache/hadoop/hbase/security/token/TokenProvider.java
          • /hbase/branches/0.92/security/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java
          • /hbase/branches/0.92/security/src/main/java/org/apache/hadoop/hbase/security/token/ZKSecretWatcher.java
          • /hbase/branches/0.92/security/src/test
          • /hbase/branches/0.92/security/src/test/java
          • /hbase/branches/0.92/security/src/test/java/org
          • /hbase/branches/0.92/security/src/test/java/org/apache
          • /hbase/branches/0.92/security/src/test/java/org/apache/hadoop
          • /hbase/branches/0.92/security/src/test/java/org/apache/hadoop/hbase
          • /hbase/branches/0.92/security/src/test/java/org/apache/hadoop/hbase/security
          • /hbase/branches/0.92/security/src/test/java/org/apache/hadoop/hbase/security/token
          • /hbase/branches/0.92/security/src/test/java/org/apache/hadoop/hbase/security/token/TestTokenAuthentication.java
          • /hbase/branches/0.92/security/src/test/java/org/apache/hadoop/hbase/security/token/TestZKSecretWatcher.java
          • /hbase/branches/0.92/security/src/test/resources
          • /hbase/branches/0.92/security/src/test/resources/hbase-site.xml
          • /hbase/branches/0.92/src/assembly/all.xml
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/HServerAddress.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/ipc/RequestContext.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/security/KerberosInfo.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/security/TokenInfo.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/security/User.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKLeaderManager.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
          • /hbase/branches/0.92/src/main/resources/hbase-default.xml
          • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java
          • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java
          • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKLeaderManager.java
          Show
          Hudson added a comment - Integrated in HBase-0.92 #143 (See https://builds.apache.org/job/HBase-0.92/143/ ) HBASE-2742 Provide strong authentication with a secure RPC engine garyh : Files : /hbase/branches/0.92/CHANGES.txt /hbase/branches/0.92/conf/hbase-policy.xml /hbase/branches/0.92/pom.xml /hbase/branches/0.92/security /hbase/branches/0.92/security/src /hbase/branches/0.92/security/src/main /hbase/branches/0.92/security/src/main/java /hbase/branches/0.92/security/src/main/java/org /hbase/branches/0.92/security/src/main/java/org/apache /hbase/branches/0.92/security/src/main/java/org/apache/hadoop /hbase/branches/0.92/security/src/main/java/org/apache/hadoop/hbase /hbase/branches/0.92/security/src/main/java/org/apache/hadoop/hbase/ipc /hbase/branches/0.92/security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java /hbase/branches/0.92/security/src/main/java/org/apache/hadoop/hbase/ipc/SecureConnectionHeader.java /hbase/branches/0.92/security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java /hbase/branches/0.92/security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java /hbase/branches/0.92/security/src/main/java/org/apache/hadoop/hbase/security /hbase/branches/0.92/security/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java /hbase/branches/0.92/security/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java /hbase/branches/0.92/security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java /hbase/branches/0.92/security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java /hbase/branches/0.92/security/src/main/java/org/apache/hadoop/hbase/security/token /hbase/branches/0.92/security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationKey.java /hbase/branches/0.92/security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationProtocol.java /hbase/branches/0.92/security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenIdentifier.java /hbase/branches/0.92/security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java /hbase/branches/0.92/security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSelector.java /hbase/branches/0.92/security/src/main/java/org/apache/hadoop/hbase/security/token/TokenProvider.java /hbase/branches/0.92/security/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java /hbase/branches/0.92/security/src/main/java/org/apache/hadoop/hbase/security/token/ZKSecretWatcher.java /hbase/branches/0.92/security/src/test /hbase/branches/0.92/security/src/test/java /hbase/branches/0.92/security/src/test/java/org /hbase/branches/0.92/security/src/test/java/org/apache /hbase/branches/0.92/security/src/test/java/org/apache/hadoop /hbase/branches/0.92/security/src/test/java/org/apache/hadoop/hbase /hbase/branches/0.92/security/src/test/java/org/apache/hadoop/hbase/security /hbase/branches/0.92/security/src/test/java/org/apache/hadoop/hbase/security/token /hbase/branches/0.92/security/src/test/java/org/apache/hadoop/hbase/security/token/TestTokenAuthentication.java /hbase/branches/0.92/security/src/test/java/org/apache/hadoop/hbase/security/token/TestZKSecretWatcher.java /hbase/branches/0.92/security/src/test/resources /hbase/branches/0.92/security/src/test/resources/hbase-site.xml /hbase/branches/0.92/src/assembly/all.xml /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/HServerAddress.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/ipc/RequestContext.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/HMaster.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/security/KerberosInfo.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/security/TokenInfo.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/security/User.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKLeaderManager.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java /hbase/branches/0.92/src/main/resources/hbase-default.xml /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKLeaderManager.java
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #2455 (See https://builds.apache.org/job/HBase-TRUNK/2455/)
          HBASE-2742 Provide strong authentication with a secure RPC engine

          garyh :
          Files :

          • /hbase/trunk/CHANGES.txt
          • /hbase/trunk/conf/hbase-policy.xml
          • /hbase/trunk/pom.xml
          • /hbase/trunk/security
          • /hbase/trunk/security/src
          • /hbase/trunk/security/src/main
          • /hbase/trunk/security/src/main/java
          • /hbase/trunk/security/src/main/java/org
          • /hbase/trunk/security/src/main/java/org/apache
          • /hbase/trunk/security/src/main/java/org/apache/hadoop
          • /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase
          • /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/ipc
          • /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java
          • /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/ipc/SecureConnectionHeader.java
          • /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java
          • /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java
          • /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/security
          • /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java
          • /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java
          • /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java
          • /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java
          • /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/security/token
          • /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationKey.java
          • /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationProtocol.java
          • /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenIdentifier.java
          • /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java
          • /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSelector.java
          • /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/security/token/TokenProvider.java
          • /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java
          • /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/security/token/ZKSecretWatcher.java
          • /hbase/trunk/security/src/test
          • /hbase/trunk/security/src/test/java
          • /hbase/trunk/security/src/test/java/org
          • /hbase/trunk/security/src/test/java/org/apache
          • /hbase/trunk/security/src/test/java/org/apache/hadoop
          • /hbase/trunk/security/src/test/java/org/apache/hadoop/hbase
          • /hbase/trunk/security/src/test/java/org/apache/hadoop/hbase/security
          • /hbase/trunk/security/src/test/java/org/apache/hadoop/hbase/security/token
          • /hbase/trunk/security/src/test/java/org/apache/hadoop/hbase/security/token/TestTokenAuthentication.java
          • /hbase/trunk/security/src/test/java/org/apache/hadoop/hbase/security/token/TestZKSecretWatcher.java
          • /hbase/trunk/security/src/test/resources
          • /hbase/trunk/security/src/test/resources/hbase-site.xml
          • /hbase/trunk/src/assembly/all.xml
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/HServerAddress.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/RequestContext.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/KerberosInfo.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/TokenInfo.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/User.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKLeaderManager.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
          • /hbase/trunk/src/main/resources/hbase-default.xml
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKLeaderManager.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK #2455 (See https://builds.apache.org/job/HBase-TRUNK/2455/ ) HBASE-2742 Provide strong authentication with a secure RPC engine garyh : Files : /hbase/trunk/CHANGES.txt /hbase/trunk/conf/hbase-policy.xml /hbase/trunk/pom.xml /hbase/trunk/security /hbase/trunk/security/src /hbase/trunk/security/src/main /hbase/trunk/security/src/main/java /hbase/trunk/security/src/main/java/org /hbase/trunk/security/src/main/java/org/apache /hbase/trunk/security/src/main/java/org/apache/hadoop /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/ipc /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/ipc/SecureConnectionHeader.java /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/security /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/security/token /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationKey.java /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationProtocol.java /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenIdentifier.java /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSelector.java /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/security/token/TokenProvider.java /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/security/token/ZKSecretWatcher.java /hbase/trunk/security/src/test /hbase/trunk/security/src/test/java /hbase/trunk/security/src/test/java/org /hbase/trunk/security/src/test/java/org/apache /hbase/trunk/security/src/test/java/org/apache/hadoop /hbase/trunk/security/src/test/java/org/apache/hadoop/hbase /hbase/trunk/security/src/test/java/org/apache/hadoop/hbase/security /hbase/trunk/security/src/test/java/org/apache/hadoop/hbase/security/token /hbase/trunk/security/src/test/java/org/apache/hadoop/hbase/security/token/TestTokenAuthentication.java /hbase/trunk/security/src/test/java/org/apache/hadoop/hbase/security/token/TestZKSecretWatcher.java /hbase/trunk/security/src/test/resources /hbase/trunk/security/src/test/resources/hbase-site.xml /hbase/trunk/src/assembly/all.xml /hbase/trunk/src/main/java/org/apache/hadoop/hbase/HServerAddress.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/RequestContext.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/HMaster.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/KerberosInfo.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/TokenInfo.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/User.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKLeaderManager.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java /hbase/trunk/src/main/resources/hbase-default.xml /hbase/trunk/src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKLeaderManager.java
          Hide
          Gary Helmling added a comment -

          Here's a trivial addendum patch to fix intermittent failures in TestZKLeaderManager. Previously the test was always using CANDIDATES[0].getWatcher() as the ZooKeeperWatcher for verifying data in zk matched the expected leader. This could fail if CANDIDATES[0] was a previous leader stopped by the test.

          Show
          Gary Helmling added a comment - Here's a trivial addendum patch to fix intermittent failures in TestZKLeaderManager. Previously the test was always using CANDIDATES [0] .getWatcher() as the ZooKeeperWatcher for verifying data in zk matched the expected leader. This could fail if CANDIDATES [0] was a previous leader stopped by the test.
          Hide
          Ted Yu added a comment -

          +1 on addendum.

          Thanks Gary.

          Show
          Ted Yu added a comment - +1 on addendum. Thanks Gary.
          Hide
          Gary Helmling added a comment -

          Addendum patch committed to 0.92 branch and trunk. Thanks for review, Ted.

          Show
          Gary Helmling added a comment - Addendum patch committed to 0.92 branch and trunk. Thanks for review, Ted.
          Hide
          Hudson added a comment -

          Integrated in HBase-0.92 #144 (See https://builds.apache.org/job/HBase-0.92/144/)
          Amend HBASE-2742 Fix intermittent TestZKLeaderManager failure

          garyh :
          Files :

          • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKLeaderManager.java
          Show
          Hudson added a comment - Integrated in HBase-0.92 #144 (See https://builds.apache.org/job/HBase-0.92/144/ ) Amend HBASE-2742 Fix intermittent TestZKLeaderManager failure garyh : Files : /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKLeaderManager.java
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #2458 (See https://builds.apache.org/job/HBase-TRUNK/2458/)
          Amend HBASE-2742 Fix intermittent TestZKLeaderManager failure

          garyh :
          Files :

          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKLeaderManager.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK #2458 (See https://builds.apache.org/job/HBase-TRUNK/2458/ ) Amend HBASE-2742 Fix intermittent TestZKLeaderManager failure garyh : Files : /hbase/trunk/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKLeaderManager.java

            People

            • Assignee:
              Gary Helmling
              Reporter:
              Gary Helmling
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development