HBase
  1. HBase
  2. HBASE-5727

secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs'

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.95.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      If you build with the security profile – i.e. add '-P security' on the command line – you'll see that the secure build is broke since we messed in rpc.

      Assigning Deveraj to take a look. If you can't work on this now DD, just give it back to me and I'll have a go at it. Thanks.

      1. AccessController.diff
        2 kB
        Ted Yu
      2. 5727.2.patch
        3 kB
        Devaraj Das
      3. 5727.1.patch
        3 kB
        Devaraj Das
      4. 5727.patch
        5 kB
        Devaraj Das

        Issue Links

          Activity

          Hide
          stack added a comment -

          Marking closed.

          Show
          stack added a comment - Marking closed.
          Hide
          Ted Yu added a comment -

          build #166 passed.

          Show
          Ted Yu added a comment - build #166 passed.
          Hide
          Ted Yu added a comment -

          Integrated addendum to trunk.

          Jenkins is having trouble performing build at the moment.
          Will resolve once a good build is produced.

          Show
          Ted Yu added a comment - Integrated addendum to trunk. Jenkins is having trouble performing build at the moment. Will resolve once a good build is produced.
          Hide
          Ted Yu added a comment -

          If there is no objection, I will integrate the addendum in 3 hours.

          Show
          Ted Yu added a comment - If there is no objection, I will integrate the addendum in 3 hours.
          Hide
          stack added a comment -

          Patch looks right to me; its providing the region context configuration making access determination to a particular region.

          Show
          stack added a comment - Patch looks right to me; its providing the region context configuration making access determination to a particular region.
          Hide
          Ted Yu added a comment -

          The attached patch allows TestRowProcessorEndpoint to pass.

          Show
          Ted Yu added a comment - The attached patch allows TestRowProcessorEndpoint to pass.
          Hide
          Andrew Purtell added a comment -

          If your coprocessor uses the HRegionServer.conf, it should work the same as it always has. If your corprocessor uses HRegion.conf, a compilation error is better than a subtle runtime error that you have to spend a week hunting down.

          Yes I agree. I didn't mean to imply it was an incorrect change in some way, just that the change had implications (obviously...)

          Taking a look at the security code, it looks like you want to use HRegionServer.conf because both calling classes seem to access global security data.

          Yes.

          Show
          Andrew Purtell added a comment - If your coprocessor uses the HRegionServer.conf, it should work the same as it always has. If your corprocessor uses HRegion.conf, a compilation error is better than a subtle runtime error that you have to spend a week hunting down. Yes I agree. I didn't mean to imply it was an incorrect change in some way, just that the change had implications (obviously...) Taking a look at the security code, it looks like you want to use HRegionServer.conf because both calling classes seem to access global security data. Yes.
          Hide
          Nicolas Spiegelberg added a comment -

          I agree with Stack, this isn't a common occurrence and is pretty trivial to fix.

          getConf() was, sadly, a hack API that was public because one of the tests wasn't in the same namespace originally. It shouldn't have been public because the config has traditionally been at a per-server granularity. Now, with HBASE-5335, the per-server config object can be per-server or per-region/per-CF depending upon how it's used and the functionality you want. Now: HRegion.conf != HRegionServer.conf != Store.conf. If your coprocessor uses the HRegionServer.conf, it should work the same as it always has. If your corprocessor uses HRegion.conf, a compilation error is better than a subtle runtime error that you have to spend a week hunting down. Of course, this is part of why we tell people they are playing with fire if they use a coprocessor (vs. MySQL triggers).

          Taking a look at the security code, it looks like you want to use HRegionServer.conf because both calling classes seem to access global security data. See HRegionServer.getConfiguration(), which is used in other sections of the AccessController class.

          Show
          Nicolas Spiegelberg added a comment - I agree with Stack, this isn't a common occurrence and is pretty trivial to fix. getConf() was, sadly, a hack API that was public because one of the tests wasn't in the same namespace originally. It shouldn't have been public because the config has traditionally been at a per-server granularity. Now, with HBASE-5335 , the per-server config object can be per-server or per-region/per-CF depending upon how it's used and the functionality you want. Now: HRegion.conf != HRegionServer.conf != Store.conf. If your coprocessor uses the HRegionServer.conf, it should work the same as it always has. If your corprocessor uses HRegion.conf, a compilation error is better than a subtle runtime error that you have to spend a week hunting down. Of course, this is part of why we tell people they are playing with fire if they use a coprocessor (vs. MySQL triggers). Taking a look at the security code, it looks like you want to use HRegionServer.conf because both calling classes seem to access global security data. See HRegionServer.getConfiguration(), which is used in other sections of the AccessController class.
          Hide
          Andrew Purtell added a comment -

          Removing getConf() was an action that had implications for all coprocessors; AccessController just exposed it by being the only thing in tree currently that uses it.

          We can unify the build by integrating RPC security in the 0.96 RPC singularity. These issues with secure versus nonsecure build should be as short a transitional phase as possible, i.e. 0.92 and 0.94 use it to bridge Hadoop core differences until 0.96 specifies Hadoop 1.0 or equivalent security shims as a minimum requirement.

          Show
          Andrew Purtell added a comment - Removing getConf() was an action that had implications for all coprocessors; AccessController just exposed it by being the only thing in tree currently that uses it. We can unify the build by integrating RPC security in the 0.96 RPC singularity. These issues with secure versus nonsecure build should be as short a transitional phase as possible, i.e. 0.92 and 0.94 use it to bridge Hadoop core differences until 0.96 specifies Hadoop 1.0 or equivalent security shims as a minimum requirement.
          Hide
          stack added a comment -

          @Ted No. Thats why we run a secure build up on jenkins to find those rare cases where a commit on insecure breaks secure.

          Show
          stack added a comment - @Ted No. Thats why we run a secure build up on jenkins to find those rare cases where a commit on insecure breaks secure.
          Hide
          Ted Yu added a comment -

          I tend to agree with Devaraj in that HBASE-5335 should have gone through compilation against secure HBase.
          In AccessController.java, I don't see reference to HRegionServer through which the config can be obtained.

          Show
          Ted Yu added a comment - I tend to agree with Devaraj in that HBASE-5335 should have gone through compilation against secure HBase. In AccessController.java, I don't see reference to HRegionServer through which the config can be obtained.
          Hide
          stack added a comment -

          Sorry. Scratch the above suggestion.

          Show
          stack added a comment - Sorry. Scratch the above suggestion.
          Hide
          stack added a comment -

          Or just add it back w/ addendum on this patch.

          Show
          stack added a comment - Or just add it back w/ addendum on this patch.
          Hide
          Devaraj Das added a comment -

          Another option is to ask Nicolas to put the getConf() method back.

          Zhihong, that will be better IMHO, given that this patch didn't touch anything close to do with getConf(). In other words the build would have been broken even without the commit of the patch on this jira.

          Show
          Devaraj Das added a comment - Another option is to ask Nicolas to put the getConf() method back. Zhihong, that will be better IMHO, given that this patch didn't touch anything close to do with getConf(). In other words the build would have been broken even without the commit of the patch on this jira.
          Hide
          Ted Yu added a comment -

          @Devaraj:
          I should have explained in clearer terms.
          Since Nicolas removed the getConf() method in HBASE-5335, I would expect the patch in this JIRA to be rebased.

          Another option is to ask Nicolas to put the getConf() method back.

          Show
          Ted Yu added a comment - @Devaraj: I should have explained in clearer terms. Since Nicolas removed the getConf() method in HBASE-5335 , I would expect the patch in this JIRA to be rebased. Another option is to ask Nicolas to put the getConf() method back.
          Hide
          Devaraj Das added a comment -

          But this isn't related to the patch committed, right? The committed patch doesn't touch any of the affected files. Or, am i missing something?

          Show
          Devaraj Das added a comment - But this isn't related to the patch committed, right? The committed patch doesn't touch any of the affected files. Or, am i missing something?
          Hide
          Ted Yu added a comment -

          The compilation issue is real.

          Show
          Ted Yu added a comment - The compilation issue is real.
          Hide
          Ted Yu added a comment -

          Nicolas checked in HBASE-5335 a little earlier than Stack checked in this patch.
          So we now have the following compilation failure:

          [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.0.2:compile (default-compile) on project hbase: Compilation failure: Compilation failure:
          [ERROR] /Users/zhihyu/trunk-hbase/security/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java:[194,23] cannot find symbol
          [ERROR] symbol  : method getConf()
          [ERROR] location: class org.apache.hadoop.hbase.regionserver.HRegion
          
          Show
          Ted Yu added a comment - Nicolas checked in HBASE-5335 a little earlier than Stack checked in this patch. So we now have the following compilation failure: [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.0.2:compile ( default -compile) on project hbase: Compilation failure: Compilation failure: [ERROR] /Users/zhihyu/trunk-hbase/security/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java:[194,23] cannot find symbol [ERROR] symbol : method getConf() [ERROR] location: class org.apache.hadoop.hbase.regionserver.HRegion
          Hide
          Devaraj Das added a comment -

          I guess Zhihong forgot to 'resolve' the issue.

          Show
          Devaraj Das added a comment - I guess Zhihong forgot to 'resolve' the issue.
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK-security #164 (See https://builds.apache.org/job/HBase-TRUNK-security/164/)
          HBASE-5727 secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs' (Revision 1311287)

          Result = FAILURE
          stack :
          Files :

          • /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK-security #164 (See https://builds.apache.org/job/HBase-TRUNK-security/164/ ) HBASE-5727 secure hbase build broke because of ' HBASE-5451 Switch RPC call envelope/headers to PBs' (Revision 1311287) Result = FAILURE stack : Files : /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #2731 (See https://builds.apache.org/job/HBase-TRUNK/2731/)
          HBASE-5727 secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs' (Revision 1311287)

          Result = SUCCESS
          stack :
          Files :

          • /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK #2731 (See https://builds.apache.org/job/HBase-TRUNK/2731/ ) HBASE-5727 secure hbase build broke because of ' HBASE-5451 Switch RPC call envelope/headers to PBs' (Revision 1311287) Result = SUCCESS stack : Files : /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12521965/5727.2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1454//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1454//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1454//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12521965/5727.2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1454//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1454//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1454//console This message is automatically generated.
          Hide
          stack added a comment -

          Committed to trunk. Thanks DD.

          Show
          stack added a comment - Committed to trunk. Thanks DD.
          Hide
          Devaraj Das added a comment -

          Ok here is the patch with the curlies around the LOG. I don't think we need to bother hudson since the only delta in this patch is the curly-brace.

          Show
          Devaraj Das added a comment - Ok here is the patch with the curlies around the LOG. I don't think we need to bother hudson since the only delta in this patch is the curly-brace.
          Hide
          stack added a comment -

          +1 on patch. On commit should add curlies around the log statement.

          Show
          stack added a comment - +1 on patch. On commit should add curlies around the log statement.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12521896/5727.1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1451//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1451//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1451//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12521896/5727.1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1451//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1451//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1451//console This message is automatically generated.
          Hide
          Ted Yu added a comment -

          TestRowProcessorEndpoint in security profile passed smoothly with latest patch.

          +1 on integrating the patch and launching another secure build.

          Thanks for working over the weekend, Devaraj.

          Show
          Ted Yu added a comment - TestRowProcessorEndpoint in security profile passed smoothly with latest patch. +1 on integrating the patch and launching another secure build. Thanks for working over the weekend, Devaraj.
          Hide
          Devaraj Das added a comment -

          trying hudson.

          Show
          Devaraj Das added a comment - trying hudson.
          Hide
          Devaraj Das added a comment -

          This patch provides a fix to the problem. Basically the secure RPC request was calling the PB stuff (HBaseClient.Connection.sendParam) but the secure server side doesn't handle PB (yet). The patch overrides the RPC request creation on the secure client to do the earlier non-PB stuff.

          I think HBASE-5732 should aim to provide a global fix by integrating the secure/unsecure RPC implementations.

          While debugging the problem, I also hit a bug to do with ZKUtil. The problem being that an 'add' gets attempted on an immutable list (introduced in HBASE-5722). Fixed that as well in the patch.

          Show
          Devaraj Das added a comment - This patch provides a fix to the problem. Basically the secure RPC request was calling the PB stuff (HBaseClient.Connection.sendParam) but the secure server side doesn't handle PB (yet). The patch overrides the RPC request creation on the secure client to do the earlier non-PB stuff. I think HBASE-5732 should aim to provide a global fix by integrating the secure/unsecure RPC implementations. While debugging the problem, I also hit a bug to do with ZKUtil. The problem being that an 'add' gets attempted on an immutable list (introduced in HBASE-5722 ). Fixed that as well in the patch.
          Hide
          Devaraj Das added a comment -

          Thanks, Zhihong. Will take a look.

          Show
          Devaraj Das added a comment - Thanks, Zhihong. Will take a look.
          Hide
          Ted Yu added a comment -

          One test consistently fails in secure build:
          https://builds.apache.org/view/G-L/view/HBase/job/HBase-TRUNK-security/161/testReport/org.apache.hadoop.hbase.coprocessor/TestRowProcessorEndpoint/org_apache_hadoop_hbase_coprocessor_TestRowProcessorEndpoint/

          I reproduced it on MacBook. Here is the tail of test output:

          2012-04-07 08:45:17,801 WARN  [RegionServer:0;192.168.0.17,59758,1333813404805] regionserver.HRegionServer(1813): Unable to connect to master. Retrying. Error was:
          java.io.IOException: Call to /192.168.0.17:59756 failed on local exception: java.io.EOFException
            at org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:945)
            at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:914)
            at org.apache.hadoop.hbase.ipc.SecureRpcEngine$Invoker.invoke(SecureRpcEngine.java:164)
            at $Proxy18.getProtocolVersion(Unknown Source)
            at org.apache.hadoop.hbase.ipc.SecureRpcEngine.getProxy(SecureRpcEngine.java:208)
            at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:305)
            at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:282)
            at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:334)
            at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:238)
            at org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:1799)
            at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1845)
            at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:672)
            at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:140)
            at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:93)
            at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:124)
            at java.security.AccessController.doPrivileged(Native Method)
            at javax.security.auth.Subject.doAs(Subject.java:337)
            at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1075)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:597)
            at org.apache.hadoop.hbase.util.Methods.call(Methods.java:41)
            at org.apache.hadoop.hbase.security.User.call(User.java:617)
            at org.apache.hadoop.hbase.security.User.access$700(User.java:54)
            at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:457)
            at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:122)
            at java.lang.Thread.run(Thread.java:680)
          Caused by: java.io.EOFException
            at java.io.DataInputStream.readInt(DataInputStream.java:375)
            at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.receiveResponse(SecureClient.java:344)
            at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:507)
          
          2012-04-07 08:45:18,002 INFO  [RegionServer:0;192.168.0.17,59758,1333813404805] regionserver.HRegionServer(1794): Attempting connect to Master server at 192.168.0.17,59756,1333813402649
          
          2012-04-07 08:45:18,223 WARN  [IPC Reader 3 on port 59756] ipc.HBaseServer$Listener(723): IPC Server listener on 59756: readAndProcess threw exception java.lang.NullPointerException. Count of bytes read: 0
          java.lang.NullPointerException
            at org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:605)
            at org.apache.hadoop.hbase.ipc.Invocation.readFields(Invocation.java:127)
            at org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.processData(SecureServer.java:597)
            at org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.processOneRpc(SecureServer.java:575)
            at org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.readAndProcess(SecureServer.java:474)
            at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:719)
            at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:510)
            at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:485)
            at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
            at java.lang.Thread.run(Thread.java:680)
          
          Show
          Ted Yu added a comment - One test consistently fails in secure build: https://builds.apache.org/view/G-L/view/HBase/job/HBase-TRUNK-security/161/testReport/org.apache.hadoop.hbase.coprocessor/TestRowProcessorEndpoint/org_apache_hadoop_hbase_coprocessor_TestRowProcessorEndpoint/ I reproduced it on MacBook. Here is the tail of test output: 2012-04-07 08:45:17,801 WARN [RegionServer:0;192.168.0.17,59758,1333813404805] regionserver.HRegionServer(1813): Unable to connect to master. Retrying. Error was: java.io.IOException: Call to /192.168.0.17:59756 failed on local exception: java.io.EOFException at org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:945) at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:914) at org.apache.hadoop.hbase.ipc.SecureRpcEngine$Invoker.invoke(SecureRpcEngine.java:164) at $Proxy18.getProtocolVersion(Unknown Source) at org.apache.hadoop.hbase.ipc.SecureRpcEngine.getProxy(SecureRpcEngine.java:208) at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:305) at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:282) at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:334) at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:238) at org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:1799) at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1845) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:672) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:140) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:93) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:124) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:337) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1075) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.util.Methods.call(Methods.java:41) at org.apache.hadoop.hbase.security.User.call(User.java:617) at org.apache.hadoop.hbase.security.User.access$700(User.java:54) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:457) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:122) at java.lang. Thread .run( Thread .java:680) Caused by: java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:375) at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.receiveResponse(SecureClient.java:344) at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:507) 2012-04-07 08:45:18,002 INFO [RegionServer:0;192.168.0.17,59758,1333813404805] regionserver.HRegionServer(1794): Attempting connect to Master server at 192.168.0.17,59756,1333813402649 2012-04-07 08:45:18,223 WARN [IPC Reader 3 on port 59756] ipc.HBaseServer$Listener(723): IPC Server listener on 59756: readAndProcess threw exception java.lang.NullPointerException. Count of bytes read: 0 java.lang.NullPointerException at org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:605) at org.apache.hadoop.hbase.ipc.Invocation.readFields(Invocation.java:127) at org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.processData(SecureServer.java:597) at org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.processOneRpc(SecureServer.java:575) at org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.readAndProcess(SecureServer.java:474) at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:719) at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:510) at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:485) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang. Thread .run( Thread .java:680)
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #2715 (See https://builds.apache.org/job/HBase-TRUNK/2715/)
          HBASE-5727 secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs (Revision 1310073)

          Result = FAILURE
          stack :
          Files :

          • /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK #2715 (See https://builds.apache.org/job/HBase-TRUNK/2715/ ) HBASE-5727 secure hbase build broke because of ' HBASE-5451 Switch RPC call envelope/headers to PBs (Revision 1310073) Result = FAILURE stack : Files : /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK-security #158 (See https://builds.apache.org/job/HBase-TRUNK-security/158/)
          HBASE-5727 secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs (Revision 1310073)

          Result = FAILURE
          stack :
          Files :

          • /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK-security #158 (See https://builds.apache.org/job/HBase-TRUNK-security/158/ ) HBASE-5727 secure hbase build broke because of ' HBASE-5451 Switch RPC call envelope/headers to PBs (Revision 1310073) Result = FAILURE stack : Files : /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java
          Hide
          stack added a comment -

          I committed this patch and forced build of secure trunk. Won't close till that build is running again.

          Show
          stack added a comment - I committed this patch and forced build of secure trunk. Won't close till that build is running again.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12521546/5727.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.master.TestSplitLogManager

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1410//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1410//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1410//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12521546/5727.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.master.TestSplitLogManager Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1410//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1410//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1410//console This message is automatically generated.
          Hide
          Devaraj Das added a comment -

          How about running 'mvn -P security -P localTests test' locally?

          Thanks, this was what I was looking for.

          I raised HBASE-5732 as a follow up to discuss removing the secure RPC engine.

          Show
          Devaraj Das added a comment - How about running 'mvn -P security -P localTests test' locally? Thanks, this was what I was looking for. I raised HBASE-5732 as a follow up to discuss removing the secure RPC engine.
          Hide
          Andrew Purtell added a comment -

          Pushing through Hudson. Let me know if I should run any other test on this patch

          HudsonQA doesn't test the security profile IIRC. How about running 'mvn -P security -P localTests test' locally?

          Show
          Andrew Purtell added a comment - Pushing through Hudson. Let me know if I should run any other test on this patch HudsonQA doesn't test the security profile IIRC. How about running 'mvn -P security -P localTests test' locally?
          Hide
          Andrew Purtell added a comment -

          @Stack. Fixing the build as an interim step toward merging RPC is fine.

          Show
          Andrew Purtell added a comment - @Stack. Fixing the build as an interim step toward merging RPC is fine.
          Hide
          Devaraj Das added a comment -

          Pushing through Hudson. Let me know if I should run any other test on this patch.

          Show
          Devaraj Das added a comment - Pushing through Hudson. Let me know if I should run any other test on this patch.
          Hide
          Devaraj Das added a comment -

          Yes, I agree that we should merge the secure/unsecure RPC layers, as a follow up.

          Show
          Devaraj Das added a comment - Yes, I agree that we should merge the secure/unsecure RPC layers, as a follow up.
          Hide
          stack added a comment -

          @Andrew Please advise. I'm not up on whats involved. What you think of the DD patch attached?

          On the patch DD, its just what was done on HBaseServer applied to SecureServer? It looks good to me.

          Show
          stack added a comment - @Andrew Please advise. I'm not up on whats involved. What you think of the DD patch attached? On the patch DD, its just what was done on HBaseServer applied to SecureServer? It looks good to me.
          Hide
          Andrew Purtell added a comment -

          @Stack, @Devaraj: Do you want to try removing the need for separate RPC engines (maybe as a follow on issue)?

          Show
          Andrew Purtell added a comment - @Stack, @Devaraj: Do you want to try removing the need for separate RPC engines (maybe as a follow on issue)?
          Hide
          Devaraj Das added a comment -

          Attaching a patch that makes the build go through (at least compiles). I haven't run the tests with this.

          Show
          Devaraj Das added a comment - Attaching a patch that makes the build go through (at least compiles). I haven't run the tests with this.
          Hide
          Andrew Purtell added a comment -

          So, to be clear, I mean as part of this work consolidate the RPC engines assuming Hadoop 1.0 or equivalent shims. Then the -P security profile would just build coprocessors. There would be little (no?) need for a security profile at all.

          Show
          Andrew Purtell added a comment - So, to be clear, I mean as part of this work consolidate the RPC engines assuming Hadoop 1.0 or equivalent shims. Then the -P security profile would just build coprocessors. There would be little (no?) need for a security profile at all.
          Hide
          Andrew Purtell added a comment -

          I commented over on HBASE-5451: Subclassing of RPC engines was introduced to deal with incompatibilities in security related classes between Hadoop versions. If we are specifying Hadoop 1.0 (or equivalent shims) as a build requirement, then we could instead use a single RPC envelope and the User abstraction to paper over the remaining differences.

          Show
          Andrew Purtell added a comment - I commented over on HBASE-5451 : Subclassing of RPC engines was introduced to deal with incompatibilities in security related classes between Hadoop versions. If we are specifying Hadoop 1.0 (or equivalent shims) as a build requirement, then we could instead use a single RPC envelope and the User abstraction to paper over the remaining differences.
          Hide
          Devaraj Das added a comment -

          I will get to it today. Sorry that I broke the build.

          Show
          Devaraj Das added a comment - I will get to it today. Sorry that I broke the build.

            People

            • Assignee:
              Devaraj Das
              Reporter:
              stack
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development