Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.0
    • Component/s: Coprocessors
    • Labels:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Provides ability to restrict table coprocessors based on HDFS path whitelist. (Particularly useful for allowing Phoenix coprocessors but not arbitrary user created coprocessors.)

      Description

      Today one can turn off all non-system coprocessors with hbase.coprocessor.user.enabled however, this disables very useful things like Apache Phoenix's coprocessors. Some tenants of a multi-user HBase may also need to run bespoke coprocessors. But as an operator I would not want wanton coprocessor usage. Ideally, one could do one of two things:

      • Allow coprocessors defined in hbase-site.xml – this can only be administratively changed in most cases
      • Allow coprocessors from table descriptors but only if the coprocessor is whitelisted
      1. HBASE-16700.000.patch
        11 kB
        Clay B.
      2. HBASE-16700.001.patch
        12 kB
        Clay B.
      3. HBASE-16700.002.patch
        17 kB
        Clay B.
      4. HBASE-16700.003.patch
        16 kB
        Clay B.
      5. HBASE-16700.004.patch
        18 kB
        Clay B.
      6. HBASE-16700.005.patch
        20 kB
        Clay B.
      7. HBASE-16700.006.patch
        26 kB
        Clay B.
      8. HBASE-16700.007.patch
        26 kB
        Clay B.
      9. HBASE-16700.008.patch
        25 kB
        Clay B.

        Activity

        Hide
        clayb Clay B. added a comment -

        If implementing a whitelist, one would probably need to whitelist directory paths to be some secure location opposed to class names. Another question to bring up, is how do these coprocessors get used? (One could conceivably cause havoc with a coprocessor on a table with unexpected data.) To prevent errant usage, one could conceivably verify the coprocessor was deployed to the expected table or log an error; eliminating any processor to table mapping ability in core HBase.

        Show
        clayb Clay B. added a comment - If implementing a whitelist, one would probably need to whitelist directory paths to be some secure location opposed to class names. Another question to bring up, is how do these coprocessors get used? (One could conceivably cause havoc with a coprocessor on a table with unexpected data.) To prevent errant usage, one could conceivably verify the coprocessor was deployed to the expected table or log an error; eliminating any processor to table mapping ability in core HBase.
        Hide
        yuzhihong@gmail.com Ted Yu added a comment -
        + */
        +public class CoprocessorWhitelistMasterObserver extends BaseMasterObserver {
        

        Please add annotation for Audience.

        +      Collection<String> paths =
        +          services.getConfiguration().getStringCollection(
        +              CP_COPROCESSOR_WHITELIST_PATHS_KEY);
        

        The above can be lifted outside the for loop.

        +public class TestCoprocessorWhitelistMasterObserver extends SecureTestUtil {
        

        Add test category.

        +  private static final Log LOG = LogFactory.getLog(TestAccessController.class);
        

        Change class name to match actual class.

        Show
        yuzhihong@gmail.com Ted Yu added a comment - + */ + public class CoprocessorWhitelistMasterObserver extends BaseMasterObserver { Please add annotation for Audience. + Collection< String > paths = + services.getConfiguration().getStringCollection( + CP_COPROCESSOR_WHITELIST_PATHS_KEY); The above can be lifted outside the for loop. + public class TestCoprocessorWhitelistMasterObserver extends SecureTestUtil { Add test category. + private static final Log LOG = LogFactory.getLog(TestAccessController.class); Change class name to match actual class.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 18s Docker mode activated.
        +1 hbaseanti 0m 0s Patch does not have any anti-patterns.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 3m 30s master passed
        +1 compile 0m 41s master passed
        +1 checkstyle 0m 49s master passed
        +1 mvneclipse 0m 15s master passed
        +1 findbugs 2m 0s master passed
        +1 javadoc 0m 31s master passed
        +1 mvninstall 0m 54s the patch passed
        +1 compile 0m 42s the patch passed
        +1 javac 0m 42s the patch passed
        +1 checkstyle 0m 48s the patch passed
        +1 mvneclipse 0m 14s the patch passed
        -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 hadoopcheck 28m 10s Patch does not cause any errors with Hadoop 2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1.
        +1 findbugs 2m 7s the patch passed
        +1 javadoc 0m 29s the patch passed
        -1 unit 15m 16s hbase-server in the patch failed.
        +1 asflicense 0m 9s The patch does not generate ASF License warnings.
        57m 13s



        Reason Tests
        Failed junit tests hadoop.hbase.TestCheckTestClasses



        Subsystem Report/Notes
        Docker Client=1.12.3 Server=1.12.3 Image:yetus/hbase:7bda515
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838245/HBASE-16700.000.patch
        JIRA Issue HBASE-16700
        Optional Tests asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile
        uname Linux 11c650d92609 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
        git revision master / 287358b
        Default Java 1.8.0_111
        findbugs v3.0.0
        whitespace https://builds.apache.org/job/PreCommit-HBASE-Build/4408/artifact/patchprocess/whitespace-eol.txt
        unit https://builds.apache.org/job/PreCommit-HBASE-Build/4408/artifact/patchprocess/patch-unit-hbase-server.txt
        unit test logs https://builds.apache.org/job/PreCommit-HBASE-Build/4408/artifact/patchprocess/patch-unit-hbase-server.txt
        Test Results https://builds.apache.org/job/PreCommit-HBASE-Build/4408/testReport/
        modules C: hbase-server U: hbase-server
        Console output https://builds.apache.org/job/PreCommit-HBASE-Build/4408/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 hbaseanti 0m 0s Patch does not have any anti-patterns. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 3m 30s master passed +1 compile 0m 41s master passed +1 checkstyle 0m 49s master passed +1 mvneclipse 0m 15s master passed +1 findbugs 2m 0s master passed +1 javadoc 0m 31s master passed +1 mvninstall 0m 54s the patch passed +1 compile 0m 42s the patch passed +1 javac 0m 42s the patch passed +1 checkstyle 0m 48s the patch passed +1 mvneclipse 0m 14s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 hadoopcheck 28m 10s Patch does not cause any errors with Hadoop 2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. +1 findbugs 2m 7s the patch passed +1 javadoc 0m 29s the patch passed -1 unit 15m 16s hbase-server in the patch failed. +1 asflicense 0m 9s The patch does not generate ASF License warnings. 57m 13s Reason Tests Failed junit tests hadoop.hbase.TestCheckTestClasses Subsystem Report/Notes Docker Client=1.12.3 Server=1.12.3 Image:yetus/hbase:7bda515 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838245/HBASE-16700.000.patch JIRA Issue HBASE-16700 Optional Tests asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile uname Linux 11c650d92609 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh git revision master / 287358b Default Java 1.8.0_111 findbugs v3.0.0 whitespace https://builds.apache.org/job/PreCommit-HBASE-Build/4408/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HBASE-Build/4408/artifact/patchprocess/patch-unit-hbase-server.txt unit test logs https://builds.apache.org/job/PreCommit-HBASE-Build/4408/artifact/patchprocess/patch-unit-hbase-server.txt Test Results https://builds.apache.org/job/PreCommit-HBASE-Build/4408/testReport/ modules C: hbase-server U: hbase-server Console output https://builds.apache.org/job/PreCommit-HBASE-Build/4408/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        clayb Clay B. added a comment -

        Thanks Ted! Including InterfaceAudience; moving the configuration getter out of the for loop; adding a test classification; correcting the LogFactory class; removing trailing white space. Two things I noticed: correcting to use a String() instead of Path() for the success log message and removing extraneous log message "Checking coprocessor %s".

        Show
        clayb Clay B. added a comment - Thanks Ted! Including InterfaceAudience; moving the configuration getter out of the for loop; adding a test classification; correcting the LogFactory class; removing trailing white space. Two things I noticed: correcting to use a String() instead of Path() for the success log message and removing extraneous log message "Checking coprocessor %s".
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 21s Docker mode activated.
        +1 hbaseanti 0m 0s Patch does not have any anti-patterns.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 3m 47s master passed
        +1 compile 0m 42s master passed
        +1 checkstyle 0m 48s master passed
        +1 mvneclipse 0m 15s master passed
        +1 findbugs 1m 49s master passed
        +1 javadoc 0m 27s master passed
        +1 mvninstall 0m 47s the patch passed
        +1 compile 0m 38s the patch passed
        +1 javac 0m 38s the patch passed
        +1 checkstyle 0m 44s the patch passed
        +1 mvneclipse 0m 14s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 hadoopcheck 30m 30s Patch does not cause any errors with Hadoop 2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1.
        +1 findbugs 2m 5s the patch passed
        +1 javadoc 0m 27s the patch passed
        -1 unit 105m 37s hbase-server in the patch failed.
        +1 asflicense 0m 14s The patch does not generate ASF License warnings.
        149m 50s



        Reason Tests
        Timed out junit tests org.apache.hadoop.hbase.namespace.TestNamespaceAuditor
          org.apache.hadoop.hbase.wal.TestWALSplitCompressed
          org.apache.hadoop.hbase.master.TestTableLockManager
          org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancer
          org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes
          org.apache.hadoop.hbase.wal.TestWALSplit



        Subsystem Report/Notes
        Docker Client=1.12.2 Server=1.12.2 Image:yetus/hbase:7bda515
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838267/HBASE-16700.001.patch
        JIRA Issue HBASE-16700
        Optional Tests asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile
        uname Linux d8639b3a72a2 3.13.0-100-generic #147-Ubuntu SMP Tue Oct 18 16:48:51 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
        git revision master / 8192a6b
        Default Java 1.8.0_111
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HBASE-Build/4414/artifact/patchprocess/patch-unit-hbase-server.txt
        unit test logs https://builds.apache.org/job/PreCommit-HBASE-Build/4414/artifact/patchprocess/patch-unit-hbase-server.txt
        Test Results https://builds.apache.org/job/PreCommit-HBASE-Build/4414/testReport/
        modules C: hbase-server U: hbase-server
        Console output https://builds.apache.org/job/PreCommit-HBASE-Build/4414/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated. +1 hbaseanti 0m 0s Patch does not have any anti-patterns. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 3m 47s master passed +1 compile 0m 42s master passed +1 checkstyle 0m 48s master passed +1 mvneclipse 0m 15s master passed +1 findbugs 1m 49s master passed +1 javadoc 0m 27s master passed +1 mvninstall 0m 47s the patch passed +1 compile 0m 38s the patch passed +1 javac 0m 38s the patch passed +1 checkstyle 0m 44s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 hadoopcheck 30m 30s Patch does not cause any errors with Hadoop 2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. +1 findbugs 2m 5s the patch passed +1 javadoc 0m 27s the patch passed -1 unit 105m 37s hbase-server in the patch failed. +1 asflicense 0m 14s The patch does not generate ASF License warnings. 149m 50s Reason Tests Timed out junit tests org.apache.hadoop.hbase.namespace.TestNamespaceAuditor   org.apache.hadoop.hbase.wal.TestWALSplitCompressed   org.apache.hadoop.hbase.master.TestTableLockManager   org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancer   org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes   org.apache.hadoop.hbase.wal.TestWALSplit Subsystem Report/Notes Docker Client=1.12.2 Server=1.12.2 Image:yetus/hbase:7bda515 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838267/HBASE-16700.001.patch JIRA Issue HBASE-16700 Optional Tests asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile uname Linux d8639b3a72a2 3.13.0-100-generic #147-Ubuntu SMP Tue Oct 18 16:48:51 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh git revision master / 8192a6b Default Java 1.8.0_111 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HBASE-Build/4414/artifact/patchprocess/patch-unit-hbase-server.txt unit test logs https://builds.apache.org/job/PreCommit-HBASE-Build/4414/artifact/patchprocess/patch-unit-hbase-server.txt Test Results https://builds.apache.org/job/PreCommit-HBASE-Build/4414/testReport/ modules C: hbase-server U: hbase-server Console output https://builds.apache.org/job/PreCommit-HBASE-Build/4414/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        yuzhihong@gmail.com Ted Yu added a comment -

        Does TestStochasticLoadBalancer pass locally for you ?

        Show
        yuzhihong@gmail.com Ted Yu added a comment - Does TestStochasticLoadBalancer pass locally for you ?
        Hide
        clayb Clay B. added a comment -

        Hi Ted,

        Yes, I was able to run org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancer; it took just under eight minutes for my whole mvn test '-Dtest=org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancer' run.

        Show
        clayb Clay B. added a comment - Hi Ted, Yes, I was able to run org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancer ; it took just under eight minutes for my whole mvn test '-Dtest=org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancer' run.
        Hide
        enis Enis Soztutar added a comment -

        Thanks Clay for working on this. A couple of comments:

        • You want to also inject into preCreateTable() as well since a table can be created with the coprocessors already defined.
        • We should rename hbase.coprocessor.regionserver.whitelist.paths to hbase.coprocessor.region.whitelist.paths because the coprocessors whose paths are whitelisted are "region observers", not regionserver observers.
        • From the issue title, I was assuming this is whitelisting the coprocessor classes. It seems that we are whitelisting the paths instead. Do we want to do both, or just paths is fine for your use case. Lets change the title to reflect that.
        • With this approach, existing table descriptors are not checked, but it maybe fine for now.
        • This is not used, we should remove it:
          +  public static final long VERSION = 0L;
          
        • In case of Phoenix, the coprocessors are never loaded from HDFS at all. Everything is already under the deploy directory of hbase lib dir in local file system. Is there are way we can disable all coprocessor loading from HDFS and allow only those from existing classpath?
        • Andrew Purtell do you want to take a look at this?
        Show
        enis Enis Soztutar added a comment - Thanks Clay for working on this. A couple of comments: You want to also inject into preCreateTable() as well since a table can be created with the coprocessors already defined. We should rename hbase.coprocessor.regionserver.whitelist.paths to hbase.coprocessor.region.whitelist.paths because the coprocessors whose paths are whitelisted are "region observers", not regionserver observers. From the issue title, I was assuming this is whitelisting the coprocessor classes. It seems that we are whitelisting the paths instead. Do we want to do both, or just paths is fine for your use case. Lets change the title to reflect that. With this approach, existing table descriptors are not checked, but it maybe fine for now. This is not used, we should remove it: + public static final long VERSION = 0L; In case of Phoenix, the coprocessors are never loaded from HDFS at all. Everything is already under the deploy directory of hbase lib dir in local file system. Is there are way we can disable all coprocessor loading from HDFS and allow only those from existing classpath? Andrew Purtell do you want to take a look at this?
        Hide
        clayb Clay B. added a comment -

        Enis Soztutar Thank you!

        • Forgetting preCreateTable() would have been a bad omission; that's been fixed and test cases added
        • I had not read the configuration string that way; it is now changed.
        • Here I want to allow for whitelisting coprocessors, but as once can always be sneaky (or ignorant) and use someone else's class name in a multi-tenant environment, the only permissioning point I could get a handle on was the filesystem. Shy of having class signing, it seems the class name would be too weak of a control? This does leave off any relationship between which table can use which coprocessor (probably a more reasonable control in a non-cooperative multi-tenant environment.)
        • I would expect the admin should know what was on their cluster before this coprocessor was deployed. I can not think of a way to shim in a new table descriptor without going through the master API's or having access to HDFS as the hbase user?
        • As to loading from a file:/// scheme only, I have whacked the path handling logic, so one can now use wildcards (sadly only that supported by FilenameUtils.wildcardMatch and not the more flexible Ant directory globbing style) and the scheme of the filesystem is now checked; further one can use * to allow all. This ensures one can now use file:/// for whitelisting but no hdfs:/// paths to achieve what you have asked for Phoenix (or any local coprocessors). Keep in mind file:///usr/lib/phoenix/* would likely be safer as can not a user specify a coprocessor, if available on every RS, via file:///home/clay/my_coprocessor.jar?
        Show
        clayb Clay B. added a comment - Enis Soztutar Thank you! Forgetting preCreateTable() would have been a bad omission; that's been fixed and test cases added I had not read the configuration string that way; it is now changed. Here I want to allow for whitelisting coprocessors, but as once can always be sneaky (or ignorant) and use someone else's class name in a multi-tenant environment, the only permissioning point I could get a handle on was the filesystem. Shy of having class signing, it seems the class name would be too weak of a control? This does leave off any relationship between which table can use which coprocessor (probably a more reasonable control in a non-cooperative multi-tenant environment.) I would expect the admin should know what was on their cluster before this coprocessor was deployed. I can not think of a way to shim in a new table descriptor without going through the master API's or having access to HDFS as the hbase user? As to loading from a file:/// scheme only, I have whacked the path handling logic, so one can now use wildcards (sadly only that supported by FilenameUtils.wildcardMatch and not the more flexible Ant directory globbing style) and the scheme of the filesystem is now checked; further one can use * to allow all. This ensures one can now use file:/// for whitelisting but no hdfs:/// paths to achieve what you have asked for Phoenix (or any local coprocessors). Keep in mind file:///usr/lib/phoenix/* would likely be safer as can not a user specify a coprocessor, if available on every RS, via file:///home/clay/my_coprocessor.jar ?
        Hide
        clayb Clay B. added a comment -

        Erg, removing extraneous logging.

        Show
        clayb Clay B. added a comment - Erg, removing extraneous logging.
        Hide
        clayb Clay B. added a comment -

        This fixes using * as a global whitelist. Also refactors the tests to be more clearer and adds a number of tests cases.

        Show
        clayb Clay B. added a comment - This fixes using * as a global whitelist. Also refactors the tests to be more clearer and adds a number of tests cases.
        Hide
        clayb Clay B. added a comment -

        Oof, dropped table creation test out of the last patch. Added back in now. (Aside, is there anyway I can flag that old Jenkins runs should stop so I don't burn a lot of needless CPU time with bad patches? Or is the best answer, stop submitting bad patches?)

        Show
        clayb Clay B. added a comment - Oof, dropped table creation test out of the last patch. Added back in now. (Aside, is there anyway I can flag that old Jenkins runs should stop so I don't burn a lot of needless CPU time with bad patches? Or is the best answer, stop submitting bad patches?)
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 11s Docker mode activated.
        +1 hbaseanti 0m 0s Patch does not have any anti-patterns.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 2m 49s master passed
        +1 compile 0m 32s master passed
        +1 checkstyle 0m 40s master passed
        +1 mvneclipse 0m 13s master passed
        +1 findbugs 1m 32s master passed
        +1 javadoc 0m 24s master passed
        +1 mvninstall 0m 41s the patch passed
        +1 compile 0m 33s the patch passed
        +1 javac 0m 33s the patch passed
        +1 checkstyle 0m 39s the patch passed
        +1 mvneclipse 0m 13s the patch passed
        -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 hadoopcheck 25m 24s Patch does not cause any errors with Hadoop 2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1.
        +1 findbugs 1m 41s the patch passed
        +1 javadoc 0m 24s the patch passed
        -1 unit 80m 33s hbase-server in the patch failed.
        +1 asflicense 0m 14s The patch does not generate ASF License warnings.
        117m 0s



        Reason Tests
        Timed out junit tests org.apache.hadoop.hbase.util.TestCoprocessorScanPolicy
          org.apache.hadoop.hbase.util.TestMiniClusterLoadEncoded
          org.apache.hadoop.hbase.snapshot.TestExportSnapshot
          org.apache.hadoop.hbase.util.TestMergeTable
          org.apache.hadoop.hbase.util.TestMergeTool
          org.apache.hadoop.hbase.util.TestConnectionCache



        Subsystem Report/Notes
        Docker Client=1.12.3 Server=1.12.3 Image:yetus/hbase:7bda515
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838768/HBASE-16700.002.patch
        JIRA Issue HBASE-16700
        Optional Tests asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile
        uname Linux 364e98c7bc80 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
        git revision master / 9250bf8
        Default Java 1.8.0_111
        findbugs v3.0.0
        whitespace https://builds.apache.org/job/PreCommit-HBASE-Build/4460/artifact/patchprocess/whitespace-eol.txt
        unit https://builds.apache.org/job/PreCommit-HBASE-Build/4460/artifact/patchprocess/patch-unit-hbase-server.txt
        unit test logs https://builds.apache.org/job/PreCommit-HBASE-Build/4460/artifact/patchprocess/patch-unit-hbase-server.txt
        Test Results https://builds.apache.org/job/PreCommit-HBASE-Build/4460/testReport/
        modules C: hbase-server U: hbase-server
        Console output https://builds.apache.org/job/PreCommit-HBASE-Build/4460/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 hbaseanti 0m 0s Patch does not have any anti-patterns. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 2m 49s master passed +1 compile 0m 32s master passed +1 checkstyle 0m 40s master passed +1 mvneclipse 0m 13s master passed +1 findbugs 1m 32s master passed +1 javadoc 0m 24s master passed +1 mvninstall 0m 41s the patch passed +1 compile 0m 33s the patch passed +1 javac 0m 33s the patch passed +1 checkstyle 0m 39s the patch passed +1 mvneclipse 0m 13s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 hadoopcheck 25m 24s Patch does not cause any errors with Hadoop 2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. +1 findbugs 1m 41s the patch passed +1 javadoc 0m 24s the patch passed -1 unit 80m 33s hbase-server in the patch failed. +1 asflicense 0m 14s The patch does not generate ASF License warnings. 117m 0s Reason Tests Timed out junit tests org.apache.hadoop.hbase.util.TestCoprocessorScanPolicy   org.apache.hadoop.hbase.util.TestMiniClusterLoadEncoded   org.apache.hadoop.hbase.snapshot.TestExportSnapshot   org.apache.hadoop.hbase.util.TestMergeTable   org.apache.hadoop.hbase.util.TestMergeTool   org.apache.hadoop.hbase.util.TestConnectionCache Subsystem Report/Notes Docker Client=1.12.3 Server=1.12.3 Image:yetus/hbase:7bda515 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838768/HBASE-16700.002.patch JIRA Issue HBASE-16700 Optional Tests asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile uname Linux 364e98c7bc80 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh git revision master / 9250bf8 Default Java 1.8.0_111 findbugs v3.0.0 whitespace https://builds.apache.org/job/PreCommit-HBASE-Build/4460/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HBASE-Build/4460/artifact/patchprocess/patch-unit-hbase-server.txt unit test logs https://builds.apache.org/job/PreCommit-HBASE-Build/4460/artifact/patchprocess/patch-unit-hbase-server.txt Test Results https://builds.apache.org/job/PreCommit-HBASE-Build/4460/testReport/ modules C: hbase-server U: hbase-server Console output https://builds.apache.org/job/PreCommit-HBASE-Build/4460/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 10s Docker mode activated.
        +1 hbaseanti 0m 0s Patch does not have any anti-patterns.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 3m 0s master passed
        +1 compile 0m 35s master passed
        +1 checkstyle 0m 44s master passed
        +1 mvneclipse 0m 13s master passed
        +1 findbugs 1m 39s master passed
        +1 javadoc 0m 25s master passed
        +1 mvninstall 0m 45s the patch passed
        +1 compile 0m 35s the patch passed
        +1 javac 0m 35s the patch passed
        +1 checkstyle 0m 45s the patch passed
        +1 mvneclipse 0m 13s the patch passed
        -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 hadoopcheck 27m 43s Patch does not cause any errors with Hadoop 2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1.
        +1 findbugs 1m 47s the patch passed
        +1 javadoc 0m 27s the patch passed
        -1 unit 72m 34s hbase-server in the patch failed.
        +1 asflicense 0m 13s The patch does not generate ASF License warnings.
        112m 4s



        Reason Tests
        Timed out junit tests org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDefaultVisLabelService
          org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsOpWithDifferentUsersNoACL
          org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes
          org.apache.hadoop.hbase.security.visibility.TestVisibilityWithCheckAuths



        Subsystem Report/Notes
        Docker Client=1.12.3 Server=1.12.3 Image:yetus/hbase:7bda515
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838769/HBASE-16700.003.patch
        JIRA Issue HBASE-16700
        Optional Tests asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile
        uname Linux 8e84e97a228d 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
        git revision master / 9250bf8
        Default Java 1.8.0_111
        findbugs v3.0.0
        whitespace https://builds.apache.org/job/PreCommit-HBASE-Build/4461/artifact/patchprocess/whitespace-eol.txt
        unit https://builds.apache.org/job/PreCommit-HBASE-Build/4461/artifact/patchprocess/patch-unit-hbase-server.txt
        unit test logs https://builds.apache.org/job/PreCommit-HBASE-Build/4461/artifact/patchprocess/patch-unit-hbase-server.txt
        Test Results https://builds.apache.org/job/PreCommit-HBASE-Build/4461/testReport/
        modules C: hbase-server U: hbase-server
        Console output https://builds.apache.org/job/PreCommit-HBASE-Build/4461/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 hbaseanti 0m 0s Patch does not have any anti-patterns. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 3m 0s master passed +1 compile 0m 35s master passed +1 checkstyle 0m 44s master passed +1 mvneclipse 0m 13s master passed +1 findbugs 1m 39s master passed +1 javadoc 0m 25s master passed +1 mvninstall 0m 45s the patch passed +1 compile 0m 35s the patch passed +1 javac 0m 35s the patch passed +1 checkstyle 0m 45s the patch passed +1 mvneclipse 0m 13s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 hadoopcheck 27m 43s Patch does not cause any errors with Hadoop 2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. +1 findbugs 1m 47s the patch passed +1 javadoc 0m 27s the patch passed -1 unit 72m 34s hbase-server in the patch failed. +1 asflicense 0m 13s The patch does not generate ASF License warnings. 112m 4s Reason Tests Timed out junit tests org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDefaultVisLabelService   org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsOpWithDifferentUsersNoACL   org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes   org.apache.hadoop.hbase.security.visibility.TestVisibilityWithCheckAuths Subsystem Report/Notes Docker Client=1.12.3 Server=1.12.3 Image:yetus/hbase:7bda515 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838769/HBASE-16700.003.patch JIRA Issue HBASE-16700 Optional Tests asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile uname Linux 8e84e97a228d 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh git revision master / 9250bf8 Default Java 1.8.0_111 findbugs v3.0.0 whitespace https://builds.apache.org/job/PreCommit-HBASE-Build/4461/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HBASE-Build/4461/artifact/patchprocess/patch-unit-hbase-server.txt unit test logs https://builds.apache.org/job/PreCommit-HBASE-Build/4461/artifact/patchprocess/patch-unit-hbase-server.txt Test Results https://builds.apache.org/job/PreCommit-HBASE-Build/4461/testReport/ modules C: hbase-server U: hbase-server Console output https://builds.apache.org/job/PreCommit-HBASE-Build/4461/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 9s Docker mode activated.
        +1 hbaseanti 0m 0s Patch does not have any anti-patterns.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 9m 25s master passed
        +1 compile 0m 32s master passed
        +1 checkstyle 1m 2s master passed
        +1 mvneclipse 0m 23s master passed
        +1 findbugs 1m 30s master passed
        +1 javadoc 0m 43s master passed
        +1 mvninstall 0m 40s the patch passed
        +1 compile 0m 32s the patch passed
        +1 javac 0m 32s the patch passed
        +1 checkstyle 0m 40s the patch passed
        +1 mvneclipse 0m 12s the patch passed
        -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 hadoopcheck 25m 12s Patch does not cause any errors with Hadoop 2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1.
        +1 findbugs 1m 42s the patch passed
        +1 javadoc 0m 25s the patch passed
        -1 unit 88m 26s hbase-server in the patch failed.
        +1 asflicense 0m 25s The patch does not generate ASF License warnings.
        132m 16s



        Reason Tests
        Timed out junit tests org.apache.hadoop.hbase.master.procedure.TestModifyTableProcedure
          org.apache.hadoop.hbase.master.procedure.TestSplitTableRegionProcedure
          org.apache.hadoop.hbase.master.procedure.TestRestoreSnapshotProcedure
          org.apache.hadoop.hbase.master.procedure.TestMasterProcedureWalLease



        Subsystem Report/Notes
        Docker Client=1.12.3 Server=1.12.3 Image:yetus/hbase:7bda515
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838785/HBASE-16700.005.patch
        JIRA Issue HBASE-16700
        Optional Tests asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile
        uname Linux 92a6ca264724 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
        git revision master / 9250bf8
        Default Java 1.8.0_111
        findbugs v3.0.0
        whitespace https://builds.apache.org/job/PreCommit-HBASE-Build/4463/artifact/patchprocess/whitespace-eol.txt
        unit https://builds.apache.org/job/PreCommit-HBASE-Build/4463/artifact/patchprocess/patch-unit-hbase-server.txt
        unit test logs https://builds.apache.org/job/PreCommit-HBASE-Build/4463/artifact/patchprocess/patch-unit-hbase-server.txt
        Test Results https://builds.apache.org/job/PreCommit-HBASE-Build/4463/testReport/
        modules C: hbase-server U: hbase-server
        Console output https://builds.apache.org/job/PreCommit-HBASE-Build/4463/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 9s Docker mode activated. +1 hbaseanti 0m 0s Patch does not have any anti-patterns. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 9m 25s master passed +1 compile 0m 32s master passed +1 checkstyle 1m 2s master passed +1 mvneclipse 0m 23s master passed +1 findbugs 1m 30s master passed +1 javadoc 0m 43s master passed +1 mvninstall 0m 40s the patch passed +1 compile 0m 32s the patch passed +1 javac 0m 32s the patch passed +1 checkstyle 0m 40s the patch passed +1 mvneclipse 0m 12s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 hadoopcheck 25m 12s Patch does not cause any errors with Hadoop 2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. +1 findbugs 1m 42s the patch passed +1 javadoc 0m 25s the patch passed -1 unit 88m 26s hbase-server in the patch failed. +1 asflicense 0m 25s The patch does not generate ASF License warnings. 132m 16s Reason Tests Timed out junit tests org.apache.hadoop.hbase.master.procedure.TestModifyTableProcedure   org.apache.hadoop.hbase.master.procedure.TestSplitTableRegionProcedure   org.apache.hadoop.hbase.master.procedure.TestRestoreSnapshotProcedure   org.apache.hadoop.hbase.master.procedure.TestMasterProcedureWalLease Subsystem Report/Notes Docker Client=1.12.3 Server=1.12.3 Image:yetus/hbase:7bda515 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838785/HBASE-16700.005.patch JIRA Issue HBASE-16700 Optional Tests asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile uname Linux 92a6ca264724 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh git revision master / 9250bf8 Default Java 1.8.0_111 findbugs v3.0.0 whitespace https://builds.apache.org/job/PreCommit-HBASE-Build/4463/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HBASE-Build/4463/artifact/patchprocess/patch-unit-hbase-server.txt unit test logs https://builds.apache.org/job/PreCommit-HBASE-Build/4463/artifact/patchprocess/patch-unit-hbase-server.txt Test Results https://builds.apache.org/job/PreCommit-HBASE-Build/4463/testReport/ modules C: hbase-server U: hbase-server Console output https://builds.apache.org/job/PreCommit-HBASE-Build/4463/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        clayb Clay B. added a comment -

        All the failed tests look to be timeouts?

        Show
        clayb Clay B. added a comment - All the failed tests look to be timeouts?
        Hide
        enis Enis Soztutar added a comment -

        Here I want to allow for whitelisting coprocessors, but as once can always be sneaky (or ignorant) and use someone else's class name in a multi-tenant environment, the only permissioning point I could get a handle on was the filesystem.

        I was asking whether we want to do class name white listing on top of path white listing. It should be fine for now.

        This ensures one can now use file:/// for whitelisting but no hdfs:/// paths to achieve what you have asked for Phoenix (or any local coprocessors).

        I was more thinking of only allowing coprocessors already in the classpath. Phoenix coprocessors are not defined with a path, assuming that they are already under the hbase lib dir. So, not even random stuff from the local file system. If you configure the allowed path to be a non-existing path for example, you can achieve the affect, but it would be better if there is an easier way. Something like opposite of wildcard which matches no string so that user cannot ever dynamically load any coprocessor class.

        Can you please also add some doc / javadoc on how to configure this (maybe a couple of examples).

        Show
        enis Enis Soztutar added a comment - Here I want to allow for whitelisting coprocessors, but as once can always be sneaky (or ignorant) and use someone else's class name in a multi-tenant environment, the only permissioning point I could get a handle on was the filesystem. I was asking whether we want to do class name white listing on top of path white listing. It should be fine for now. This ensures one can now use file:/// for whitelisting but no hdfs:/// paths to achieve what you have asked for Phoenix (or any local coprocessors). I was more thinking of only allowing coprocessors already in the classpath. Phoenix coprocessors are not defined with a path, assuming that they are already under the hbase lib dir. So, not even random stuff from the local file system. If you configure the allowed path to be a non-existing path for example, you can achieve the affect, but it would be better if there is an easier way. Something like opposite of wildcard which matches no string so that user cannot ever dynamically load any coprocessor class. Can you please also add some doc / javadoc on how to configure this (maybe a couple of examples).
        Hide
        clayb Clay B. added a comment -

        This adds test cases for:

        • A classpath coprocessor (here org.apache.hadoop.hbase.coprocessor.BaseRegionObserver since it's handy)
        • The above test is a table creation test
        • Adds back the accidentally dropped table creation test show the creation should fail
        • Add HBase reference documentation (I had to build this dropping the <skin> tag in src/main/site/site.xml as mvn site was breaking for me as Jenkins recently reported.

        As to:

        I was asking whether we want to do class name white listing on top of path white listing. It should be fine for now.

        I think it would make sense to allow certain tables certain paths or coprocessors but for now I think that's a pretty far down use case from what I have yet seen; certainly could be a follow-up JIRA if anyone wants it.

        Show
        clayb Clay B. added a comment - This adds test cases for: A classpath coprocessor (here org.apache.hadoop.hbase.coprocessor.BaseRegionObserver since it's handy) The above test is a table creation test Adds back the accidentally dropped table creation test show the creation should fail Add HBase reference documentation (I had to build this dropping the <skin> tag in src/main/site/site.xml as mvn site was breaking for me as Jenkins recently reported. As to: I was asking whether we want to do class name white listing on top of path white listing. It should be fine for now. I think it would make sense to allow certain tables certain paths or coprocessors but for now I think that's a pretty far down use case from what I have yet seen; certainly could be a follow-up JIRA if anyone wants it.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
        +1 hbaseanti 0m 0s Patch does not have any anti-patterns.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 11s Maven dependency ordering for branch
        +1 mvninstall 2m 48s master passed
        +1 compile 3m 7s master passed
        +1 checkstyle 1m 25s master passed
        +1 mvneclipse 1m 28s master passed
        0 findbugs 0m 0s Skipped patched modules with no Java source: .
        +1 findbugs 1m 39s master passed
        +1 javadoc 2m 20s master passed
        0 mvndep 0m 9s Maven dependency ordering for patch
        +1 mvninstall 3m 26s the patch passed
        +1 compile 3m 9s the patch passed
        +1 javac 3m 9s the patch passed
        +1 checkstyle 1m 27s the patch passed
        +1 mvneclipse 1m 26s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 hadoopcheck 26m 15s Patch does not cause any errors with Hadoop 2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1.
        0 findbugs 0m 0s Skipped patched modules with no Java source: .
        -1 findbugs 2m 7s hbase-server generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
        +1 javadoc 2m 47s the patch passed
        +1 unit 96m 44s hbase-server in the patch passed.
        +1 unit 123m 10s root in the patch passed.
        +1 asflicense 0m 35s The patch does not generate ASF License warnings.
        274m 51s



        Reason Tests
        FindBugs module:hbase-server
          Comparison of String objects using == or != in org.apache.hadoop.hbase.security.access.CoprocessorWhitelistMasterObserver.verifyCoprocessors(ObserverContext, HTableDescriptor) At CoprocessorWhitelistMasterObserver.java:== or != in org.apache.hadoop.hbase.security.access.CoprocessorWhitelistMasterObserver.verifyCoprocessors(ObserverContext, HTableDescriptor) At CoprocessorWhitelistMasterObserver.java:[line 182]



        Subsystem Report/Notes
        Docker Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12840277/HBASE-16700.006.patch
        JIRA Issue HBASE-16700
        Optional Tests asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile
        uname Linux 06101e9149e5 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
        git revision master / 9f5b8a8
        Default Java 1.8.0_111
        findbugs v3.0.0
        findbugs https://builds.apache.org/job/PreCommit-HBASE-Build/4601/artifact/patchprocess/new-findbugs-hbase-server.html
        Test Results https://builds.apache.org/job/PreCommit-HBASE-Build/4601/testReport/
        modules C: hbase-server . U: .
        Console output https://builds.apache.org/job/PreCommit-HBASE-Build/4601/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 hbaseanti 0m 0s Patch does not have any anti-patterns. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 11s Maven dependency ordering for branch +1 mvninstall 2m 48s master passed +1 compile 3m 7s master passed +1 checkstyle 1m 25s master passed +1 mvneclipse 1m 28s master passed 0 findbugs 0m 0s Skipped patched modules with no Java source: . +1 findbugs 1m 39s master passed +1 javadoc 2m 20s master passed 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 3m 26s the patch passed +1 compile 3m 9s the patch passed +1 javac 3m 9s the patch passed +1 checkstyle 1m 27s the patch passed +1 mvneclipse 1m 26s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 hadoopcheck 26m 15s Patch does not cause any errors with Hadoop 2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. 0 findbugs 0m 0s Skipped patched modules with no Java source: . -1 findbugs 2m 7s hbase-server generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) +1 javadoc 2m 47s the patch passed +1 unit 96m 44s hbase-server in the patch passed. +1 unit 123m 10s root in the patch passed. +1 asflicense 0m 35s The patch does not generate ASF License warnings. 274m 51s Reason Tests FindBugs module:hbase-server   Comparison of String objects using == or != in org.apache.hadoop.hbase.security.access.CoprocessorWhitelistMasterObserver.verifyCoprocessors(ObserverContext, HTableDescriptor) At CoprocessorWhitelistMasterObserver.java:== or != in org.apache.hadoop.hbase.security.access.CoprocessorWhitelistMasterObserver.verifyCoprocessors(ObserverContext, HTableDescriptor) At CoprocessorWhitelistMasterObserver.java: [line 182] Subsystem Report/Notes Docker Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12840277/HBASE-16700.006.patch JIRA Issue HBASE-16700 Optional Tests asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile uname Linux 06101e9149e5 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh git revision master / 9f5b8a8 Default Java 1.8.0_111 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-HBASE-Build/4601/artifact/patchprocess/new-findbugs-hbase-server.html Test Results https://builds.apache.org/job/PreCommit-HBASE-Build/4601/testReport/ modules C: hbase-server . U: . Console output https://builds.apache.org/job/PreCommit-HBASE-Build/4601/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        clayb Clay B. added a comment -

        This patch fixes the findbug equality issue reported for 006.

        Show
        clayb Clay B. added a comment - This patch fixes the findbug equality issue reported for 006.
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
        +1 hbaseanti 0m 0s Patch does not have any anti-patterns.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 12s Maven dependency ordering for branch
        +1 mvninstall 2m 48s master passed
        +1 compile 3m 10s master passed
        +1 checkstyle 1m 26s master passed
        +1 mvneclipse 1m 24s master passed
        0 findbugs 0m 0s Skipped patched modules with no Java source: .
        +1 findbugs 1m 40s master passed
        +1 javadoc 2m 21s master passed
        0 mvndep 0m 10s Maven dependency ordering for patch
        +1 mvninstall 3m 25s the patch passed
        +1 compile 3m 9s the patch passed
        +1 javac 3m 9s the patch passed
        +1 checkstyle 1m 26s the patch passed
        +1 mvneclipse 1m 25s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 hadoopcheck 25m 10s Patch does not cause any errors with Hadoop 2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1.
        0 findbugs 0m 0s Skipped patched modules with no Java source: .
        +1 findbugs 1m 46s the patch passed
        +1 javadoc 2m 20s the patch passed
        +1 unit 89m 43s hbase-server in the patch passed.
        +1 unit 109m 30s root in the patch passed.
        +1 asflicense 0m 33s The patch does not generate ASF License warnings.
        252m 12s



        Subsystem Report/Notes
        Docker Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12840592/HBASE-16700.007.patch
        JIRA Issue HBASE-16700
        Optional Tests asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile
        uname Linux a327262348d5 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
        git revision master / 8204337
        Default Java 1.8.0_111
        findbugs v3.0.0
        Test Results https://builds.apache.org/job/PreCommit-HBASE-Build/4637/testReport/
        modules C: hbase-server . U: .
        Console output https://builds.apache.org/job/PreCommit-HBASE-Build/4637/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 hbaseanti 0m 0s Patch does not have any anti-patterns. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 12s Maven dependency ordering for branch +1 mvninstall 2m 48s master passed +1 compile 3m 10s master passed +1 checkstyle 1m 26s master passed +1 mvneclipse 1m 24s master passed 0 findbugs 0m 0s Skipped patched modules with no Java source: . +1 findbugs 1m 40s master passed +1 javadoc 2m 21s master passed 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 3m 25s the patch passed +1 compile 3m 9s the patch passed +1 javac 3m 9s the patch passed +1 checkstyle 1m 26s the patch passed +1 mvneclipse 1m 25s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 hadoopcheck 25m 10s Patch does not cause any errors with Hadoop 2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. 0 findbugs 0m 0s Skipped patched modules with no Java source: . +1 findbugs 1m 46s the patch passed +1 javadoc 2m 20s the patch passed +1 unit 89m 43s hbase-server in the patch passed. +1 unit 109m 30s root in the patch passed. +1 asflicense 0m 33s The patch does not generate ASF License warnings. 252m 12s Subsystem Report/Notes Docker Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12840592/HBASE-16700.007.patch JIRA Issue HBASE-16700 Optional Tests asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile uname Linux a327262348d5 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh git revision master / 8204337 Default Java 1.8.0_111 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HBASE-Build/4637/testReport/ modules C: hbase-server . U: . Console output https://builds.apache.org/job/PreCommit-HBASE-Build/4637/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        enis Enis Soztutar added a comment -

        Thanks Clay for the updated patches. Looks pretty good to commit. Just some last items:

        • We should remove this (assuming that you added that for debugging):
          +  static {
          +    Logger.getLogger(CoprocessorWhitelistMasterObserver.class).setLevel(Level.TRACE);
          +    Logger.getLogger("org.apache.hbase.server").setLevel(Level.TRACE);
          +  }
          
        • Can you please refactor var names like coproc_path to camelCase.
        • Did you want to enable this test?
          +//  @Test
          +  @Category(MediumTests.class)
          +  public void testCreationClasspathCoprocessor() throws Exception {
          
        • great doc!
        Show
        enis Enis Soztutar added a comment - Thanks Clay for the updated patches. Looks pretty good to commit. Just some last items: We should remove this (assuming that you added that for debugging): + static { + Logger.getLogger(CoprocessorWhitelistMasterObserver.class).setLevel(Level.TRACE); + Logger.getLogger( "org.apache.hbase.server" ).setLevel(Level.TRACE); + } Can you please refactor var names like coproc_path to camelCase. Did you want to enable this test? + // @Test + @Category(MediumTests.class) + public void testCreationClasspathCoprocessor() throws Exception { great doc!
        Hide
        clayb Clay B. added a comment -

        Enis Soztutar thank you for yet another review! I've made the requested changes and clarified a few example strings (i.e. if a coprocessor should be considered whitelisted or not in the test cases).

        Show
        clayb Clay B. added a comment - Enis Soztutar thank you for yet another review! I've made the requested changes and clarified a few example strings (i.e. if a coprocessor should be considered whitelisted or not in the test cases).
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 11s Docker mode activated.
        +1 hbaseanti 0m 0s Patch does not have any anti-patterns.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 18s Maven dependency ordering for branch
        +1 mvninstall 2m 47s master passed
        +1 compile 3m 7s master passed
        +1 checkstyle 1m 27s master passed
        +1 mvneclipse 1m 22s master passed
        0 findbugs 0m 0s Skipped patched modules with no Java source: .
        +1 findbugs 1m 41s master passed
        +1 javadoc 2m 20s master passed
        0 mvndep 0m 10s Maven dependency ordering for patch
        +1 mvninstall 3m 25s the patch passed
        +1 compile 3m 7s the patch passed
        +1 javac 3m 7s the patch passed
        +1 checkstyle 1m 27s the patch passed
        +1 mvneclipse 1m 22s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 hadoopcheck 25m 14s Patch does not cause any errors with Hadoop 2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1.
        0 findbugs 0m 0s Skipped patched modules with no Java source: .
        +1 findbugs 1m 47s the patch passed
        +1 javadoc 2m 21s the patch passed
        +1 unit 88m 13s hbase-server in the patch passed.
        +1 unit 108m 31s root in the patch passed.
        +1 asflicense 0m 33s The patch does not generate ASF License warnings.
        249m 45s



        Subsystem Report/Notes
        Docker Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12841682/HBASE-16700.008.patch
        JIRA Issue HBASE-16700
        Optional Tests asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile
        uname Linux e42c68a0656e 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
        git revision master / c73180d
        Default Java 1.8.0_111
        findbugs v3.0.0
        Test Results https://builds.apache.org/job/PreCommit-HBASE-Build/4782/testReport/
        modules C: hbase-server . U: .
        Console output https://builds.apache.org/job/PreCommit-HBASE-Build/4782/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 hbaseanti 0m 0s Patch does not have any anti-patterns. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 18s Maven dependency ordering for branch +1 mvninstall 2m 47s master passed +1 compile 3m 7s master passed +1 checkstyle 1m 27s master passed +1 mvneclipse 1m 22s master passed 0 findbugs 0m 0s Skipped patched modules with no Java source: . +1 findbugs 1m 41s master passed +1 javadoc 2m 20s master passed 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 3m 25s the patch passed +1 compile 3m 7s the patch passed +1 javac 3m 7s the patch passed +1 checkstyle 1m 27s the patch passed +1 mvneclipse 1m 22s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 hadoopcheck 25m 14s Patch does not cause any errors with Hadoop 2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. 0 findbugs 0m 0s Skipped patched modules with no Java source: . +1 findbugs 1m 47s the patch passed +1 javadoc 2m 21s the patch passed +1 unit 88m 13s hbase-server in the patch passed. +1 unit 108m 31s root in the patch passed. +1 asflicense 0m 33s The patch does not generate ASF License warnings. 249m 45s Subsystem Report/Notes Docker Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12841682/HBASE-16700.008.patch JIRA Issue HBASE-16700 Optional Tests asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile uname Linux e42c68a0656e 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh git revision master / c73180d Default Java 1.8.0_111 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HBASE-Build/4782/testReport/ modules C: hbase-server . U: . Console output https://builds.apache.org/job/PreCommit-HBASE-Build/4782/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        enis Enis Soztutar added a comment -

        I've committed this. Thanks Clay for the patch.

        Show
        enis Enis Soztutar added a comment - I've committed this. Thanks Clay for the patch.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #2080 (See https://builds.apache.org/job/HBase-Trunk_matrix/2080/)
        HBASE-16700 Allow for coprocessor whitelisting (enis: rev c7b8b63cd1fc19e3be722ae6c71791d04ef48b9d)

        • (add) hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/CoprocessorWhitelistMasterObserver.java
        • (edit) src/main/asciidoc/_chapters/cp.adoc
        • (add) hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestCoprocessorWhitelistMasterObserver.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #2080 (See https://builds.apache.org/job/HBase-Trunk_matrix/2080/ ) HBASE-16700 Allow for coprocessor whitelisting (enis: rev c7b8b63cd1fc19e3be722ae6c71791d04ef48b9d) (add) hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/CoprocessorWhitelistMasterObserver.java (edit) src/main/asciidoc/_chapters/cp.adoc (add) hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestCoprocessorWhitelistMasterObserver.java
        Hide
        stack stack added a comment -

        Clay B. Thanks for patch. Suggest add a release note since this a nice new feature. Just write something that would work for an operator audience. I think you know this perspective (smile).

        I see the added test can fail: https://builds.apache.org/view/All/job/HBase-Trunk_matrix/jdk=JDK%201.8%20(latest),label=Hadoop/2135/testReport/junit/org.apache.hadoop.hbase.security.access/TestCoprocessorWhitelistMasterObserver/org_apache_hadoop_hbase_security_access_TestCoprocessorWhitelistMasterObserver/

        It failed also on a hadoopqa build. What you think? Should it be a large test so it has more time to run or did something go wrong in this run.

        Thanks.

        Show
        stack stack added a comment - Clay B. Thanks for patch. Suggest add a release note since this a nice new feature. Just write something that would work for an operator audience. I think you know this perspective (smile). I see the added test can fail: https://builds.apache.org/view/All/job/HBase-Trunk_matrix/jdk=JDK%201.8%20(latest),label=Hadoop/2135/testReport/junit/org.apache.hadoop.hbase.security.access/TestCoprocessorWhitelistMasterObserver/org_apache_hadoop_hbase_security_access_TestCoprocessorWhitelistMasterObserver/ It failed also on a hadoopqa build. What you think? Should it be a large test so it has more time to run or did something go wrong in this run. Thanks.
        Hide
        clayb Clay B. added a comment -

        stack Thanks for thinking of the release notes! I can't seem to do a better job than what you have. The best I came up with was:
        "This features allows restricting filesystem deployed user coprocessors (those defined against a table with a JAR path). Allows for classpath coprocessors (e.g. Apache Phoenix) and coprocessors deployed from whitelisted file-system paths."

        Otherwise, I would think these should be medium tests based on being nine tests (each are certainly expected to take less than 50 seconds) so that'd be a max of less than 10 minutes total serially (though they can be parallelized). As I understand a medium test it should take less than 30 minutes; but it may be true that there's not a need for all developers to end up running this?

        Unfortunately, the Jenkins link seems to have become a 404, so I can't see which test(s) timed out to understand how they may have hung up?

        Show
        clayb Clay B. added a comment - stack Thanks for thinking of the release notes! I can't seem to do a better job than what you have. The best I came up with was: "This features allows restricting filesystem deployed user coprocessors (those defined against a table with a JAR path). Allows for classpath coprocessors (e.g. Apache Phoenix) and coprocessors deployed from whitelisted file-system paths." Otherwise, I would think these should be medium tests based on being nine tests (each are certainly expected to take less than 50 seconds) so that'd be a max of less than 10 minutes total serially (though they can be parallelized). As I understand a medium test it should take less than 30 minutes; but it may be true that there's not a need for all developers to end up running this? Unfortunately, the Jenkins link seems to have become a 404, so I can't see which test(s) timed out to understand how they may have hung up?

          People

          • Assignee:
            clayb Clay B.
            Reporter:
            clayb Clay B.
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development