Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: security
    • Labels:
      None

      Description

      Consider implementing discretionary access control for HBase.

      Access control has three aspects: authentication, authorization and audit.

      • Authentication: Access is controlled by insisting on an authentication procedure to establish the identity of the user. The authentication procedure should minimally require a non-plaintext authentication factor (e.g. encrypted password with salt) and should ideally or at least optionally provide cryptographically strong confidence via public key certification.
      • Authorization: Access is controlled by specifying rights to resources via an access control list (ACL). An ACL is a list of permissions attached to an object. The list specifies who or what is allowed to access the object and what operations are allowed to be performed on the object, f.e. create, update, read, or delete.
      • Audit: Important actions taken by subjects should be logged for accountability, a chronological record which enables the full reconstruction and examination of a sequence of events, e.g. schema changes or data mutations. Logging activity should be protected from all subjects except for a restricted set with administrative privilege, perhaps to only a single super-user.

      Discretionary access control means the access policy for an object is determined by the owner of the object. Every object in the system must have a valid owner. Owners can assign access rights and permissions to other users. The initial owner of an object is the subject who created it. If subjects are deleted from a system, ownership of objects owned by them should revert to some super-user or otherwise valid default.

      HBase can enforce access policy at table, column family, or cell granularity. Cell granularity does not make much sense. An implementation which controls access at both the table and column family levels is recommended, though a first cut could consider control at the table level only. The initial set of permissions can be: Create (table schema or column family), update (table schema or column family), read (column family), delete (table or column family), execute (filters), and transfer ownership. The subject identities and access tokens could be stored in a new administrative table. ACLs on tables and column families can be stored in META.

      Access other than read access to catalog and administrative tables should be restricted to a set of administrative users or perhaps a single super-user. A data mutation on a user table by a subject without administrative or superuser privilege which results in a table split is an implicit temporary privilege elevation where the regionserver or master updates the catalog tables as necessary to support the split.

      Audit logging should be configurable on a per-table basis to avoid this overhead where it is not wanted.

      Consider supporting external authentication and subject identification mechanisms with Java library support: RADIUS/TACACS, Kerberos, LDAP.

      Consider logging audit trails to an HBase table (bigtable type schemas are natural for this) and optionally external logging options with Java library support – syslog, etc., or maybe commons-logging is sufficient and punt to administrator to set up appropriate commons-logging/log4j configurations for their needs.

      If HBASE-1002 is considered, and the option to support filtering via upload of (perhaps complex) bytecode produced by some little language compiler is implemented, the execute privilege could be extended in a manner similar to how stored procedures in SQL land execute either with the privilege of the current user or the (table/procedure) creator.

        Issue Links

          Activity

          Hide
          Laxman added a comment -

          Able to find the issue. In client machines we didn't replace the JCE policy jars.
          I think its worth documenting this.

          Show
          Laxman added a comment - Able to find the issue. In client machines we didn't replace the JCE policy jars. I think its worth documenting this.
          Hide
          Laxman added a comment -

          Thanks for the info Andrew. I'm discussing this issue with Eugene. (ZOOKEEPER-1467)

          We got struck with another problem in HBase client authentication.
          Client is not able to establish connection with HBase server successfully.

          Exception we got here:

          2012-05-22 09:42:22,627 WARN org.apache.hadoop.ipc.SecureClient: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
          2012-05-22 09:42:22,627 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:testuser (auth:KERBEROS) cause:java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
          2012-05-22 09:42:22,630 DEBUG org.apache.hadoop.ipc.SecureClient: closing ipc connection to HOST-10-18-40-19/10.18.40.19:60020: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
          java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
          	at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$1.run(SecureClient.java:227)
          	at java.security.AccessController.doPrivileged(Native Method)
          	at javax.security.auth.Subject.doAs(Subject.java:396)
          	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          	at java.lang.reflect.Method.invoke(Method.java:597)
          	at org.apache.hadoop.hbase.util.Methods.call(Methods.java:37)
          	at org.apache.hadoop.hbase.security.User.call(User.java:586)
          	at org.apache.hadoop.hbase.security.User.access$700(User.java:50)
          	at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:440)
          	at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.handleSaslConnectionFailure(SecureClient.java:194)
          	at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupIOstreams(SecureClient.java:274)
          	at org.apache.hadoop.hbase.ipc.SecureClient.getConnection(SecureClient.java:485)
          	at org.apache.hadoop.hbase.ipc.SecureClient.getConnection(SecureClient.java:69)
          	at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:897)
          	at org.apache.hadoop.hbase.ipc.SecureRpcEngine$Invoker.invoke(SecureRpcEngine.java:164)
          	at $Proxy6.getProtocolVersion(Unknown Source)
          	at org.apache.hadoop.hbase.ipc.SecureRpcEngine.getProxy(SecureRpcEngine.java:208)
          	at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:303)
          	at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:280)
          	at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:332)
          	at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:236)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:1284)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:1240)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:1227)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:936)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
          	at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
          	at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
          	at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:133)
          	at hbase.test.Hbasetest.main(Hbasetest.java:37)
          Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
          	at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:194)
          	at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:138)
          	at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupSaslConnection(SecureClient.java:176)
          	at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.access$500(SecureClient.java:84)
          	at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$2.run(SecureClient.java:267)
          	at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$2.run(SecureClient.java:264)
          	at java.security.AccessController.doPrivileged(Native Method)
          	at javax.security.auth.Subject.doAs(Subject.java:396)
          	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          	at java.lang.reflect.Method.invoke(Method.java:597)
          	at org.apache.hadoop.hbase.util.Methods.call(Methods.java:37)
          	at org.apache.hadoop.hbase.security.User.call(User.java:586)
          	at org.apache.hadoop.hbase.security.User.access$700(User.java:50)
          	at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:440)
          	at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupIOstreams(SecureClient.java:263)
          	... 23 more
          Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
          	at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:130)
          	at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:106)
          	at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:172)
          	at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:209)
          	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:195)
          	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:162)
          	at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:175)
          	... 40 more
          2012-05-22 09:42:22,636 DEBUG org.apache.hadoop.ipc.SecureClient: IPC Client (1778276127) connection to HOST-10-18-40-19/10.18.40.19:60020 from testuser: closed
          2012-05-22 09:42:22,638 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation={region=-ROOT-,,0.70236052, hostname=HOST-10-18-40-19, port=60020}, attempt=0 of 120 failed; retrying after sleep of 1000 because: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
          2012-05-22 09:42:22,640 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: Looked up root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@6ecf829d; serverName=HOST-10-18-40-19,60020,1337574445438
          2012-05-22 09:42:23,641 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: Looked up root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@6ecf829d; serverName=HOST-10-18-40-19,60020,1337574445438
          2012-05-22 09:42:23,642 DEBUG org.apache.hadoop.ipc.SecureClient: RPC Server Kerberos principal name for protocol=org.apache.hadoop.hbase.ipc.HRegionInterface is hbase/hadoop@HADOOP.COM
          

          Other details:

          HBase version: 0.94.0
          Hadoop version: 0.23.1
          Kerberos version: 1.10.1
          Java version: 1.6.0_31, 64 bit
          Linux version: SuSE 11.1 [Kernel version : 2.6.32.12-0.7-default x86_64 GNU/Linux]

          We had gone thru the solutions available @
          http://docs.oracle.com/javase/1.5.0/docs/guide/security/jgss/tutorials/Troubleshooting.html
          https://ccp.cloudera.com/display/CDHDOC/Appendix+A+-+Troubleshooting#AppendixA-Troubleshooting-Problem2%3AJavaisunabletoreadtheKerberoscredentialscachecreatedbyversionsofMITKerberos1.8.1orhigher.

          But none of then seems to work. Any clues?

          Show
          Laxman added a comment - Thanks for the info Andrew. I'm discussing this issue with Eugene. ( ZOOKEEPER-1467 ) We got struck with another problem in HBase client authentication. Client is not able to establish connection with HBase server successfully. Exception we got here: 2012-05-22 09:42:22,627 WARN org.apache.hadoop.ipc.SecureClient: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] 2012-05-22 09:42:22,627 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:testuser (auth:KERBEROS) cause:java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] 2012-05-22 09:42:22,630 DEBUG org.apache.hadoop.ipc.SecureClient: closing ipc connection to HOST-10-18-40-19/10.18.40.19:60020: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$1.run(SecureClient.java:227) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.util.Methods.call(Methods.java:37) at org.apache.hadoop.hbase.security.User.call(User.java:586) at org.apache.hadoop.hbase.security.User.access$700(User.java:50) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:440) at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.handleSaslConnectionFailure(SecureClient.java:194) at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupIOstreams(SecureClient.java:274) at org.apache.hadoop.hbase.ipc.SecureClient.getConnection(SecureClient.java:485) at org.apache.hadoop.hbase.ipc.SecureClient.getConnection(SecureClient.java:69) at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:897) at org.apache.hadoop.hbase.ipc.SecureRpcEngine$Invoker.invoke(SecureRpcEngine.java:164) at $Proxy6.getProtocolVersion(Unknown Source) at org.apache.hadoop.hbase.ipc.SecureRpcEngine.getProxy(SecureRpcEngine.java:208) at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:303) at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:280) at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:332) at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:236) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:1284) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:1240) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:1227) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:936) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801) at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234) at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174) at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:133) at hbase.test.Hbasetest.main(Hbasetest.java:37) Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:194) at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:138) at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupSaslConnection(SecureClient.java:176) at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.access$500(SecureClient.java:84) at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$2.run(SecureClient.java:267) at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$2.run(SecureClient.java:264) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.util.Methods.call(Methods.java:37) at org.apache.hadoop.hbase.security.User.call(User.java:586) at org.apache.hadoop.hbase.security.User.access$700(User.java:50) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:440) at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupIOstreams(SecureClient.java:263) ... 23 more Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:130) at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:106) at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:172) at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:209) at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:195) at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:162) at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:175) ... 40 more 2012-05-22 09:42:22,636 DEBUG org.apache.hadoop.ipc.SecureClient: IPC Client (1778276127) connection to HOST-10-18-40-19/10.18.40.19:60020 from testuser: closed 2012-05-22 09:42:22,638 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation={region=-ROOT-,,0.70236052, hostname=HOST-10-18-40-19, port=60020}, attempt=0 of 120 failed; retrying after sleep of 1000 because: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] 2012-05-22 09:42:22,640 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: Looked up root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@6ecf829d; serverName=HOST-10-18-40-19,60020,1337574445438 2012-05-22 09:42:23,641 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: Looked up root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@6ecf829d; serverName=HOST-10-18-40-19,60020,1337574445438 2012-05-22 09:42:23,642 DEBUG org.apache.hadoop.ipc.SecureClient: RPC Server Kerberos principal name for protocol=org.apache.hadoop.hbase.ipc.HRegionInterface is hbase/hadoop@HADOOP.COM Other details: HBase version: 0.94.0 Hadoop version: 0.23.1 Kerberos version: 1.10.1 Java version: 1.6.0_31, 64 bit Linux version: SuSE 11.1 [Kernel version : 2.6.32.12-0.7-default x86_64 GNU/Linux] We had gone thru the solutions available @ http://docs.oracle.com/javase/1.5.0/docs/guide/security/jgss/tutorials/Troubleshooting.html https://ccp.cloudera.com/display/CDHDOC/Appendix+A+-+Troubleshooting#AppendixA-Troubleshooting-Problem2%3AJavaisunabletoreadtheKerberoscredentialscachecreatedbyversionsofMITKerberos1.8.1orhigher . But none of then seems to work. Any clues?
          Hide
          Andrew Purtell added a comment -

          When we bind Kerberos principals with host names (as recommended in CDH 4 security guide), we are frequently & randomly hitting ZooKeeper "Auth Failed" exception.

          ZOOKEEPER-1437 perhaps? ZooKeeper wants us to tunnel SASL in the ZK protocol instead of wrapping at the socket layer as is customarily done. Internally we implemented a CountDownLatch on SASL auth but Eugene is iterating a more architecturally appropriate solution with Patrick.

          Show
          Andrew Purtell added a comment - When we bind Kerberos principals with host names (as recommended in CDH 4 security guide), we are frequently & randomly hitting ZooKeeper "Auth Failed" exception. ZOOKEEPER-1437 perhaps? ZooKeeper wants us to tunnel SASL in the ZK protocol instead of wrapping at the socket layer as is customarily done. Internally we implemented a CountDownLatch on SASL auth but Eugene is iterating a more architecturally appropriate solution with Patrick.
          Hide
          Laxman added a comment -

          We are trying to setup a secure HBase (HBase-HDFS-ZooKeeper) cluster.
          After resolving initial issues, finally we are blocked with ZOOKEEPER-1467.

          And also, I found it to be more difficult to use Kerberos principals bound to host names.
          When we bind Kerberos principals with host names (as recommended in CDH 4 security guide), we are frequently & randomly hitting ZooKeeper "Auth Failed" exception.

          So we proceed with the cluster-id instead of host name. i.e., one user - one principal - one key tab. Following are the principals we are using.

          zookeeper/clusterid - to run ZK cluster (3-node ZooKeeper cluster)
          hdfs/clusterid - to run HDFS cluster (NameNode, DataNode)
          hbase/clusterid - to run HBase cluster (Master, Region Server)

          Is this approach correct?

          Show
          Laxman added a comment - We are trying to setup a secure HBase (HBase-HDFS-ZooKeeper) cluster. After resolving initial issues, finally we are blocked with ZOOKEEPER-1467 . And also, I found it to be more difficult to use Kerberos principals bound to host names. When we bind Kerberos principals with host names (as recommended in CDH 4 security guide), we are frequently & randomly hitting ZooKeeper "Auth Failed" exception. So we proceed with the cluster-id instead of host name. i.e., one user - one principal - one key tab. Following are the principals we are using. zookeeper/clusterid - to run ZK cluster (3-node ZooKeeper cluster) hdfs/clusterid - to run HDFS cluster (NameNode, DataNode) hbase/clusterid - to run HBase cluster (Master, Region Server) Is this approach correct?
          Hide
          Laxman added a comment -

          Thanks Gary for the info on Security.
          I'm going through the current implementation.
          Soon will take up some jiras.

          Show
          Laxman added a comment - Thanks Gary for the info on Security. I'm going through the current implementation. Soon will take up some jiras.
          Hide
          Gary Helmling added a comment -

          @Laxman,

          We probably need to do some JIRA housekeeping here. There hasn't been much activity in this ticket because it's really just an umbrella issue tying together the individual efforts.

          Initial security features are in place with HBASE-3025, HBASE-2418, and HBASE-2742. These are released in 0.92. Currently access control is performed by use of ACLs stored at the global (forthcoming), table, column family, or column qualifier level.

          There are definitely additional features that can be built to contribute to our security solution:

          • adding client authentication for thrift and REST clients
          • proxying HBase access from thrift and REST servers as the authenticated clients (currently these server can simply be configured to authenticate and access HBase as their own principals)
          • supporting or moving to alternate access control schemes (RBAC)
          • row or key-value based access control
          • supporting other authentication mechanisms than kerberos and authentication tokens
          • probably many others

          Anyone interested in working on these would certainly be welcomed. But we do currently have a working security implementation that integrates nicely with HDFS and map reduce.

          Show
          Gary Helmling added a comment - @Laxman, We probably need to do some JIRA housekeeping here. There hasn't been much activity in this ticket because it's really just an umbrella issue tying together the individual efforts. Initial security features are in place with HBASE-3025 , HBASE-2418 , and HBASE-2742 . These are released in 0.92. Currently access control is performed by use of ACLs stored at the global (forthcoming), table, column family, or column qualifier level. There are definitely additional features that can be built to contribute to our security solution: adding client authentication for thrift and REST clients proxying HBase access from thrift and REST servers as the authenticated clients (currently these server can simply be configured to authenticate and access HBase as their own principals) supporting or moving to alternate access control schemes (RBAC) row or key-value based access control supporting other authentication mechanisms than kerberos and authentication tokens probably many others Anyone interested in working on these would certainly be welcomed. But we do currently have a working security implementation that integrates nicely with HDFS and map reduce.
          Hide
          Laxman added a comment -

          No updates here from long time.
          From my understanding, to make HBase secure, we need huge contributions in this area.
          Also, this involves many challenges (architectural changes, maintain/break compatibility, ...).
          In spite of these challenges, it adds more value to HBase.

          Anyone interested to look into these security issues?

          Show
          Laxman added a comment - No updates here from long time. From my understanding, to make HBase secure, we need huge contributions in this area. Also, this involves many challenges (architectural changes, maintain/break compatibility, ...). In spite of these challenges, it adds more value to HBase. Anyone interested to look into these security issues?
          Hide
          stack added a comment -

          Moving out of 0.92.0. Pull it back in if you think different.

          Show
          stack added a comment - Moving out of 0.92.0. Pull it back in if you think different.
          Hide
          Suchisubhra sinha added a comment -

          I have sent you email. Let's talk about this in detail.
          Thanks.
          ~Suchi

          Show
          Suchisubhra sinha added a comment - I have sent you email. Let's talk about this in detail. Thanks. ~Suchi
          Hide
          Gary Helmling added a comment -

          Hi Suchi,

          Thanks, that is great!

          We've discussed per-KeyValue ACLs as a future feature, but have deferred the implementation due to other HBase internal changes that would be required to support them efficiently.

          Our rough plan was to implement a new 'metacolumn' feature for HBase (HBASE-2893). A 'metacolumn' would be a special, internal only (or coprocessor only?) column family associated with each table. Table level and column family level ACLs would still apply in the normal case, but the metacolumn would allow storing per-row or per-KeyValue override ACLs directly inline with the row data.

          For read operations, scans would then probably use a special filter to allow specific KVs to be included in results back to the client. For write operations, the org.apache.hadoop.hbase.security.rbac.AccessController.prePut() method would check for entries in the row metacolumn in addition to the normal ACLs.

          See also HBASE-3435 for a discussion of adding per-column-qualifier ACLs to our scheme. I have some work in progress that would enable column-qualifier ACLs to be used. It incorporates the use of an AccessControlFilter to allow returning only those KVs matching the granted column qualifiers for a scan. I'll be posting a patch for that soon.

          So summing it all up, we'd ultimately like to have a hierarchy for applying ACLs as follows:

          global -> table -> column family -> column qualifier -> row -> key value

          The assumption so far has been that the ACLs granted at each level are additive – if I have "read" access at the table level, then you can't revoke that access for specific KVs (and we don't have to continue checking for authorization down the full hierarchy). This is mainly a performance consideration. We can discuss if that meets your needs or not.

          If you'd like to work out a way to collaborate on getting all of this done, we'd love the help! Please email me directly at garyh@apache.org.

          Best,
          Gary

          Show
          Gary Helmling added a comment - Hi Suchi, Thanks, that is great! We've discussed per-KeyValue ACLs as a future feature, but have deferred the implementation due to other HBase internal changes that would be required to support them efficiently. Our rough plan was to implement a new 'metacolumn' feature for HBase ( HBASE-2893 ). A 'metacolumn' would be a special, internal only (or coprocessor only?) column family associated with each table. Table level and column family level ACLs would still apply in the normal case, but the metacolumn would allow storing per-row or per-KeyValue override ACLs directly inline with the row data. For read operations, scans would then probably use a special filter to allow specific KVs to be included in results back to the client. For write operations, the org.apache.hadoop.hbase.security.rbac.AccessController.prePut() method would check for entries in the row metacolumn in addition to the normal ACLs. See also HBASE-3435 for a discussion of adding per-column-qualifier ACLs to our scheme. I have some work in progress that would enable column-qualifier ACLs to be used. It incorporates the use of an AccessControlFilter to allow returning only those KVs matching the granted column qualifiers for a scan. I'll be posting a patch for that soon. So summing it all up, we'd ultimately like to have a hierarchy for applying ACLs as follows: global -> table -> column family -> column qualifier -> row -> key value The assumption so far has been that the ACLs granted at each level are additive – if I have "read" access at the table level, then you can't revoke that access for specific KVs (and we don't have to continue checking for authorization down the full hierarchy). This is mainly a performance consideration. We can discuss if that meets your needs or not. If you'd like to work out a way to collaborate on getting all of this done, we'd love the help! Please email me directly at garyh@apache.org. Best, Gary
          Hide
          Suchisubhra sinha added a comment -

          Hi Andrew,

          In our platform, we need to have ACL on object level. We are planning to build it. But we like to know if you guys have any plan to build it.
          If we know your plan, we can execute it and contribute. We have resources.

          Thanks.
          ~Suchi

          Show
          Suchisubhra sinha added a comment - Hi Andrew, In our platform, we need to have ACL on object level. We are planning to build it. But we like to know if you guys have any plan to build it. If we know your plan, we can execute it and contribute. We have resources. Thanks. ~Suchi
          Hide
          stack added a comment -

          Andrew: You need something on this issue?

          Show
          stack added a comment - Andrew: You need something on this issue?
          Hide
          E. Sammer added a comment -

          As part of this addition, it would be extremely useful to be able to attribute resource consumption to individual users in addition to normal action auditing. Specifically, we'd like to be able to attach quotas to users or tables to users and quotas to tables. It would be sufficient to simply defer to HDFS for quotas provided tables could be given specific root directories and user information was pushed down to the HDFS level. In this case, a table would be owned by a specific user.

          Show
          E. Sammer added a comment - As part of this addition, it would be extremely useful to be able to attribute resource consumption to individual users in addition to normal action auditing. Specifically, we'd like to be able to attach quotas to users or tables to users and quotas to tables. It would be sufficient to simply defer to HDFS for quotas provided tables could be given specific root directories and user information was pushed down to the HDFS level. In this case, a table would be owned by a specific user.
          Hide
          stack added a comment -
          Show
          stack added a comment - This looks like it might be pertinent to this issue: http://hadoop.apache.org/zookeeper/docs/current/zookeeperProgrammers.html#sc_ZooKeeperPluggableAuthentication
          Hide
          stack added a comment -

          No objection from me. Its mostly orthogonal to other dev. Just has to be done by the time hadoop 0.21 releases (smile).

          Show
          stack added a comment - No objection from me. Its mostly orthogonal to other dev. Just has to be done by the time hadoop 0.21 releases (smile).
          Hide
          Jean-Daniel Cryans added a comment -

          +1

          Show
          Jean-Daniel Cryans added a comment - +1
          Hide
          Andrew Purtell added a comment -

          What about bringing this into 0.21 (with sufficient sponsored development)?

          Show
          Andrew Purtell added a comment - What about bringing this into 0.21 (with sufficient sponsored development)?
          Hide
          Andrew Purtell added a comment -

          Bring in to 0.22. Discussions about needing this are heating up here.

          Show
          Andrew Purtell added a comment - Bring in to 0.22. Discussions about needing this are heating up here.

            People

            • Assignee:
              Unassigned
              Reporter:
              Andrew Purtell
            • Votes:
              1 Vote for this issue
              Watchers:
              33 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development