[HBASE-12466] Document visibility scan label generator usage and behavior - ASF JIRA

Hide

Permalink

Jerry He added a comment - 22/Dec/14 18:13

Thanks for taking on this one.
It escaped my TODO list.
If you need more information, please let me know.

Show

Jerry He added a comment - 22/Dec/14 18:13 Hi, Misty Stanley-Jones Thanks for taking on this one. It escaped my TODO list. If you need more information, please let me know.

Hide

Permalink

Misty Stanley-Jones added a comment - 24/Feb/15 05:30

I realized I don't know enough about the SLG to do this, Jerry He. Can you point me at some information?

Show

Misty Stanley-Jones added a comment - 24/Feb/15 05:30 I realized I don't know enough about the SLG to do this, Jerry He . Can you point me at some information?

Hide

Permalink

Jerry He added a comment - 24/Feb/15 18:26

Hi, Misty Stanley-Jones

Here is a draft

You can use ScanLabelGenerator to control how authorization labels are fetched and filtered during the authorization process.
HBase have ScanLabelGenerator implementations to provide the default authorization behavior (documented in ~~HBASE-12468~~).
You can also provide your own implementation of the interface ScanLabelGenerator.
ScanLabelGenerators can also be stacked to provide the desired behavior.
The output from one ScanLabelGenerator will be the input for the next ScanLabelGenerator in the stack.

Show

Jerry He added a comment - 24/Feb/15 18:26 Hi, Misty Stanley-Jones Here is a draft You can use ScanLabelGenerator to control how authorization labels are fetched and filtered during the authorization process. HBase have ScanLabelGenerator implementations to provide the default authorization behavior (documented in HBASE-12468 ). You can also provide your own implementation of the interface ScanLabelGenerator. ScanLabelGenerators can also be stacked to provide the desired behavior. The output from one ScanLabelGenerator will be the input for the next ScanLabelGenerator in the stack.

Hide

Permalink

Jerry He added a comment - 26/Feb/15 20:10

Hi, Misty Stanley-Jones

It is documented in the book: section 57.3.3. Implementing Your Own Visibility Label Algorithm
That is the section you may want to update.
For the details, you can refer to the org.apache.hadoop.hbase.security.visibility.VisibilityUtils class.
The comments in the method getScanLabelGenerators() explains a lot.

Show

Jerry He added a comment - 26/Feb/15 20:10 Hi, Misty Stanley-Jones It is documented in the book: section 57.3.3. Implementing Your Own Visibility Label Algorithm That is the section you may want to update. For the details, you can refer to the org.apache.hadoop.hbase.security.visibility.VisibilityUtils class. The comments in the method getScanLabelGenerators() explains a lot.

Hide

Permalink

Misty Stanley-Jones added a comment - 02/Mar/15 01:55

I needed to merge the work from ~~HBASE-12468~~ into this patch, because I needed to create an anchor and link to it for this one.

Show

Misty Stanley-Jones added a comment - 02/Mar/15 01:55 I needed to merge the work from HBASE-12468 into this patch, because I needed to create an anchor and link to it for this one.

Hide

Permalink

Hadoop QA added a comment - 02/Mar/15 03:56

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12701761/HBASE-12466.patch
against master branch at commit dad2474f08d201d09989e36f5cf1c25d3fa4acee.
ATTACHMENT ID: 12701761

+1 @author. The patch does not contain any @author tags.

+0 tests included. The patch appears to be a documentation patch that doesn't require tests.
+1 hadoop versions. The patch compiles with all supported hadoop versions (2.4.1 2.5.2 2.6.0)

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 checkstyle. The applied patch does not increase the total number of checkstyle errors

+1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

-1 lineLengths. The patch introduces the following lines longer than 100:
+When you issue a Scan or Get, HBase uses your default set of authorizations to filter out cells that you do not have access to. A superuser can see the default set of authorizations for a given user by using the `set_auths` HBase Shell command or the link:http://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/security/visibility/VisibilityClient.htmlsgetAuths%28org.apache.hadoop.conf.Configuration,%20java.lang.String%29[setAuths()] method.
+You can specify a different authorization during the Scan or Get, by passing the AUTHORIZATIONS option in HBase Shell, or the link:http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Scan.html#setAuthorizations%28org.apache.hadoop.hbase.security.visibility.Authorizations%29[setAuthorizations()] method if you use the API. This authorization will be combined with your default set as an additional filter. It will further filter your results, rather than giving you additional authorization.
+The default implementation class is `org.apache.hadoop.hbase.security.visibility.DefaultScanLabelGenerator`, which was implemented in link:https://issues.apache.org/jira/browse/HBASE-12466[HBASE-12468]. See <<reading_cells_with_labels>>.
+You can also configure a set, or "stack", of `ScanLabelGenerator`s to be used by the system, as a comma-separated list. The output for the first `ScanLabelGenerator` will be the input for the next one, until the end of the list.

+1 site. The mvn site goal succeeds with this patch.

-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort

-1 core zombie tests. There are 1 zombie test(s): at org.apache.hadoop.hbase.TestHColumnDescriptorDefaultVersions.testCreateTableWithDefaultFromConf(TestHColumnDescriptorDefaultVersions.java:104)

Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/13026//console

This message is automatically generated.

Show

Hadoop QA added a comment - 02/Mar/15 03:56 -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12701761/HBASE-12466.patch against master branch at commit dad2474f08d201d09989e36f5cf1c25d3fa4acee. ATTACHMENT ID: 12701761 +1 @author . The patch does not contain any @author tags. +0 tests included . The patch appears to be a documentation patch that doesn't require tests. +1 hadoop versions . The patch compiles with all supported hadoop versions (2.4.1 2.5.2 2.6.0) +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 checkstyle . The applied patch does not increase the total number of checkstyle errors +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 lineLengths . The patch introduces the following lines longer than 100: +When you issue a Scan or Get, HBase uses your default set of authorizations to filter out cells that you do not have access to. A superuser can see the default set of authorizations for a given user by using the `set_auths` HBase Shell command or the link: http://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/security/visibility/VisibilityClient.htmlsgetAuths%28org.apache.hadoop.conf.Configuration,%20java.lang.String%29[setAuths( )] method. +You can specify a different authorization during the Scan or Get, by passing the AUTHORIZATIONS option in HBase Shell, or the link: http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Scan.html#setAuthorizations%28org.apache.hadoop.hbase.security.visibility.Authorizations%29[setAuthorizations( )] method if you use the API. This authorization will be combined with your default set as an additional filter. It will further filter your results, rather than giving you additional authorization. +The default implementation class is `org.apache.hadoop.hbase.security.visibility.DefaultScanLabelGenerator`, which was implemented in link: https://issues.apache.org/jira/browse/HBASE-12466[HBASE-12468 ]. See <<reading_cells_with_labels>>. +You can also configure a set, or "stack", of `ScanLabelGenerator`s to be used by the system, as a comma-separated list. The output for the first `ScanLabelGenerator` will be the input for the next one, until the end of the list. +1 site . The mvn site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort -1 core zombie tests . There are 1 zombie test(s): at org.apache.hadoop.hbase.TestHColumnDescriptorDefaultVersions.testCreateTableWithDefaultFromConf(TestHColumnDescriptorDefaultVersions.java:104) Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/13026//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13026//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13026//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13026//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13026//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13026//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13026//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13026//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13026//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13026//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13026//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13026//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/13026//artifact/patchprocess/checkstyle-aggregate.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/13026//console This message is automatically generated.

Hide

Permalink

Jerry He added a comment - 02/Mar/15 18:17

Hi, Misty Stanley-Jones

A superuser can see the default set of authorizations for a given user by using the `set_auths` HBase Shell command or the link:http://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/security/visibility/VisibilityClient.htmlsgetAuths%28org.apache.hadoop.conf.Configuration,%20java.lang.String%29[setAuths()] method.

There is a mismatch of the link (setAuths) and the method (get). Also do you mean "A superuser can set ..."?

+The default implementation class is `org.apache.hadoop.hbase.security.visibility.DefaultScanLabelGenerator`, which was implemented in link:https://issues.apache.org/jira/browse/HBASE-12466[HBASE-12468].

There is no more 'org.apache.hadoop.hbase.security.visibility.DefaultScanLabelGenerator'. It has been changed/removed. See the following comment in the org.apache.hadoop.hbase.security.visibility.VisibilityUtils#getScanLabelGenerators(). Also there is a mismatch between the JIRA link and the real real JIRA number.

    // If no SLG is specified in conf, by default we'll add two SLGs
    // 1. FeedUserAuthScanLabelGenerator
    // 2. DefinedSetFilterScanLabelGenerator
    // This stacking will achieve the following default behavior:
    // 1. If there is no Auths in the scan, we will obtain the global defined set for the user
    //    from the labels table.
    // 2. If there is Auths in the scan, we will examine the passed in Auths and filter out the
    //    labels that the user is not entitled to. Then use the resulting label set.
    if (slgs.isEmpty()) {
      slgs.add(ReflectionUtils.newInstance(FeedUserAuthScanLabelGenerator.class, conf));
      slgs.add(ReflectionUtils.newInstance(DefinedSetFilterScanLabelGenerator.class, conf));
    }
    return slgs;

Show

Jerry He added a comment - 02/Mar/15 18:17 Hi, Misty Stanley-Jones A superuser can see the default set of authorizations for a given user by using the `set_auths` HBase Shell command or the link: http://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/security/visibility/VisibilityClient.htmlsgetAuths%28org.apache.hadoop.conf.Configuration,%20java.lang.String%29[setAuths( )] method. There is a mismatch of the link (setAuths) and the method (get). Also do you mean "A superuser can set ..."? +The default implementation class is `org.apache.hadoop.hbase.security.visibility.DefaultScanLabelGenerator`, which was implemented in link: https://issues.apache.org/jira/browse/HBASE-12466[HBASE-12468 ]. There is no more 'org.apache.hadoop.hbase.security.visibility.DefaultScanLabelGenerator'. It has been changed/removed. See the following comment in the org.apache.hadoop.hbase.security.visibility.VisibilityUtils#getScanLabelGenerators(). Also there is a mismatch between the JIRA link and the real real JIRA number. // If no SLG is specified in conf, by default we'll add two SLGs // 1. FeedUserAuthScanLabelGenerator // 2. DefinedSetFilterScanLabelGenerator // This stacking will achieve the following default behavior: // 1. If there is no Auths in the scan, we will obtain the global defined set for the user // from the labels table. // 2. If there is Auths in the scan, we will examine the passed in Auths and filter out the // labels that the user is not entitled to. Then use the resulting label set. if (slgs.isEmpty()) { slgs.add(ReflectionUtils.newInstance(FeedUserAuthScanLabelGenerator.class, conf)); slgs.add(ReflectionUtils.newInstance(DefinedSetFilterScanLabelGenerator.class, conf)); } return slgs;

Hide

Permalink

Misty Stanley-Jones added a comment - 05/Mar/15 04:46

Please review the latest patch.

Show

Misty Stanley-Jones added a comment - 05/Mar/15 04:46 Please review the latest patch.

Hide

Permalink

Hadoop QA added a comment - 05/Mar/15 06:57

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12702701/HBASE-12466-v1.patch
against master branch at commit 0bdab85b065bd0876152ac30c2ec6d08adae8006.
ATTACHMENT ID: 12702701

+1 @author. The patch does not contain any @author tags.

+0 tests included. The patch appears to be a documentation patch that doesn't require tests.

+1 hadoop versions. The patch compiles with all supported hadoop versions (2.4.1 2.5.2 2.6.0)

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 checkstyle. The applied patch does not increase the total number of checkstyle errors

+1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

-1 release audit. The applied patch generated 1 release audit warnings (more than the master's current 0 warnings).

-1 lineLengths. The patch introduces the following lines longer than 100:
+When you issue a Scan or Get, HBase uses your default set of authorizations to filter out cells that you do not have access to. A superuser can set the default set of authorizations for a given user by using the `set_auths` HBase Shell command or the link:http://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/security/visibility/VisibilityClient.htmlsgetAuths%28org.apache.hadoop.conf.Configuration,%20java.lang.String%29[setAuths()] method.
+You can specify a different authorization during the Scan or Get, by passing the AUTHORIZATIONS option in HBase Shell, or the link:http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Scan.html#setAuthorizations%28org.apache.hadoop.hbase.security.visibility.Authorizations%29[setAuthorizations()] method if you use the API. This authorization will be combined with your default set as an additional filter. It will further filter your results, rather than giving you additional authorization.
+You can specify a custom plugin or plugins by using the property `hbase.regionserver.scan.visibility.label.generator.class`. The output for the first `ScanLabelGenerator` will be the input for the next one, until the end of the list.
+The default implementation, which was implemented in link:https://issues.apache.org/jira/browse/HBASE-12466[HBASE-12466], loads two plugins, `FeedUserAuthScanLabelGenerator` and `DefinedSetFilterScanLabelGenerator`. See <<reading_cells_with_labels>>.

-1 site. The patch appears to cause mvn site goal to fail.

-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.coprocessor.TestMasterObserver

-1 core zombie tests. There are 1 zombie test(s): at org.apache.hadoop.hbase.coprocessor.TestMasterObserver.testTableOperations(TestMasterObserver.java:1221)

Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/13091//console

This message is automatically generated.

Show

Hadoop QA added a comment - 05/Mar/15 06:57 -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12702701/HBASE-12466-v1.patch against master branch at commit 0bdab85b065bd0876152ac30c2ec6d08adae8006. ATTACHMENT ID: 12702701 +1 @author . The patch does not contain any @author tags. +0 tests included . The patch appears to be a documentation patch that doesn't require tests. +1 hadoop versions . The patch compiles with all supported hadoop versions (2.4.1 2.5.2 2.6.0) +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 checkstyle . The applied patch does not increase the total number of checkstyle errors +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. -1 release audit . The applied patch generated 1 release audit warnings (more than the master's current 0 warnings). -1 lineLengths . The patch introduces the following lines longer than 100: +When you issue a Scan or Get, HBase uses your default set of authorizations to filter out cells that you do not have access to. A superuser can set the default set of authorizations for a given user by using the `set_auths` HBase Shell command or the link: http://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/security/visibility/VisibilityClient.htmlsgetAuths%28org.apache.hadoop.conf.Configuration,%20java.lang.String%29[setAuths( )] method. +You can specify a different authorization during the Scan or Get, by passing the AUTHORIZATIONS option in HBase Shell, or the link: http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Scan.html#setAuthorizations%28org.apache.hadoop.hbase.security.visibility.Authorizations%29[setAuthorizations( )] method if you use the API. This authorization will be combined with your default set as an additional filter. It will further filter your results, rather than giving you additional authorization. +You can specify a custom plugin or plugins by using the property `hbase.regionserver.scan.visibility.label.generator.class`. The output for the first `ScanLabelGenerator` will be the input for the next one, until the end of the list. +The default implementation, which was implemented in link: https://issues.apache.org/jira/browse/HBASE-12466[HBASE-12466 ], loads two plugins, `FeedUserAuthScanLabelGenerator` and `DefinedSetFilterScanLabelGenerator`. See <<reading_cells_with_labels>>. -1 site . The patch appears to cause mvn site goal to fail. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.coprocessor.TestMasterObserver -1 core zombie tests . There are 1 zombie test(s): at org.apache.hadoop.hbase.coprocessor.TestMasterObserver.testTableOperations(TestMasterObserver.java:1221) Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/13091//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13091//artifact/patchprocess/patchReleaseAuditWarnings.txt Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13091//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13091//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13091//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13091//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13091//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13091//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13091//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13091//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13091//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13091//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13091//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/13091//artifact/patchprocess/checkstyle-aggregate.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/13091//console This message is automatically generated.

Hide

Permalink

Andrew Purtell added a comment - 07/Mar/15 00:19

lgtm, commit unless Jerry He has more comments I'd say

Show

Andrew Purtell added a comment - 07/Mar/15 00:19 lgtm, commit unless Jerry He has more comments I'd say

Hide

Permalink

Jerry He added a comment - 07/Mar/15 03:12

Hi, Misty

link:http://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/security/visibility/VisibilityClient.htmlsgetAuths%28org.apache.hadoop.conf.Configuration,%20java.lang.String%29[setAuths()] method

Inside the link, the method should be VisibilityClient.htmls setAuths, with the url for setAuths
http://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/security/visibility/VisibilityClient.html#setAuths(org.apache.hadoop.conf.Configuration,%20java.lang.String[],%20java.lang.String)
Same problem for ~~HBASE-12468~~

Show

Jerry He added a comment - 07/Mar/15 03:12 Hi, Misty link: http://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/security/visibility/VisibilityClient.htmlsgetAuths%28org.apache.hadoop.conf.Configuration,%20java.lang.String%29[setAuths( )] method Inside the link, the method should be VisibilityClient.htmls setAuths , with the url for setAuths http://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/security/visibility/VisibilityClient.html#setAuths(org.apache.hadoop.conf.Configuration,%20java.lang.String[],%20java.lang.String ) Same problem for HBASE-12468

Hide

Permalink

Misty Stanley-Jones added a comment - 09/Mar/15 03:54

Pushed as part of ~~HBASE-12468~~.

Show

Misty Stanley-Jones added a comment - 09/Mar/15 03:54 Pushed as part of HBASE-12468 .

Document visibility scan label generator usage and behavior

Details

Description

Attachments

Issue Links

Activity

People

Dates

Development

Agile