HBase
  1. HBase
  2. HBASE-10831

IntegrationTestIngestWithACL is not setting up LoadTestTool correctly

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.98.1
    • Fix Version/s: 0.99.0, 0.98.3
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      IntegrationTestIngestWithACL is not setting up LoadTestTool correctly.

      Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 601.709 sec <<< FAILURE!
      testIngest(org.apache.hadoop.hbase.IntegrationTestIngestWithACL)  Time elapsed: 601.489 sec  <<< FAILURE!
      java.lang.AssertionError: Failed to initialize LoadTestTool expected:<0> but was:<1>
              at org.junit.Assert.fail(Assert.java:88)
              at org.junit.Assert.failNotEquals(Assert.java:743)
              at org.junit.Assert.assertEquals(Assert.java:118)
              at org.junit.Assert.assertEquals(Assert.java:555)
              at org.apache.hadoop.hbase.IntegrationTestIngest.initTable(IntegrationTestIngest.java:74)
              at org.apache.hadoop.hbase.IntegrationTestIngest.setUpCluster(IntegrationTestIngest.java:69)
              at org.apache.hadoop.hbase.IntegrationTestIngestWithACL.setUpCluster(IntegrationTestIngestWithACL.java:58)
              at org.apache.hadoop.hbase.IntegrationTestBase.setUp(IntegrationTestBase.java:89)
      

      Could be related to HBASE-10675?

      1. HBASE-10831_98_v1.patch
        11 kB
        Vandana Ayyalasomayajula
      2. HBASE-10831_trunk_v2.patch
        11 kB
        Vandana Ayyalasomayajula
      3. HBASE-10831_98_v3.patch
        12 kB
        Vandana Ayyalasomayajula
      4. HBASE-10831_trunk_v3.patch
        12 kB
        Vandana Ayyalasomayajula
      5. HBASE-10831_trunk_v4.patch
        12 kB
        Vandana Ayyalasomayajula
      6. HBASE-10831_98_v4.patch
        12 kB
        Vandana Ayyalasomayajula

        Issue Links

          Activity

          Hide
          Enis Soztutar added a comment -

          Closing this issue after 0.99.0 release.

          Show
          Enis Soztutar added a comment - Closing this issue after 0.99.0 release.
          Hide
          Andrew Purtell added a comment -

          We could use an addendum to remove these messages:

          2014-06-30 17:25:24,930 INFO  [HBaseReaderThreadWithACL_0] util.MultiThreadedReaderWithACL: Read happening from ACL true
          
          Show
          Andrew Purtell added a comment - We could use an addendum to remove these messages: 2014-06-30 17:25:24,930 INFO [HBaseReaderThreadWithACL_0] util.MultiThreadedReaderWithACL: Read happening from ACL true
          Hide
          ramkrishna.s.vasudevan added a comment -

          Yes true. Its handled.

          Show
          ramkrishna.s.vasudevan added a comment - Yes true. Its handled.
          Hide
          Anoop Sam John added a comment - - edited

          This is handled by below code in LoadTestTool right?

          String[] args;
                if (dataGen instanceof LoadTestDataGeneratorWithACL) {
                  LOG.info("ACL is on");
                  if (isSecure(conf)) {
                    LOG.info("Security is on.");
                    authnFileName = clazzAndArgs[1];
                    superUser = clazzAndArgs[2];
                    userNames = clazzAndArgs[3];
                    args = Arrays.copyOfRange(clazzAndArgs, 2, clazzAndArgs.length);
                   ........
                  } else {
                    superUser = clazzAndArgs[1];
                    userNames = clazzAndArgs[2];
                    args = Arrays.copyOfRange(clazzAndArgs, 1, clazzAndArgs.length);
                    userOwner = User.createUserForTesting(conf, superUser, new String[0]);
                  }
                } else {
                  args = clazzAndArgs.length == 1 ? new String[0] : Arrays.copyOfRange(clazzAndArgs, 1,
                      clazzAndArgs.length);
                }
                dataGen.initialize(args);
          
          Show
          Anoop Sam John added a comment - - edited This is handled by below code in LoadTestTool right? String [] args; if (dataGen instanceof LoadTestDataGeneratorWithACL) { LOG.info( "ACL is on" ); if (isSecure(conf)) { LOG.info( "Security is on." ); authnFileName = clazzAndArgs[1]; superUser = clazzAndArgs[2]; userNames = clazzAndArgs[3]; args = Arrays.copyOfRange(clazzAndArgs, 2, clazzAndArgs.length); ........ } else { superUser = clazzAndArgs[1]; userNames = clazzAndArgs[2]; args = Arrays.copyOfRange(clazzAndArgs, 1, clazzAndArgs.length); userOwner = User.createUserForTesting(conf, superUser, new String [0]); } } else { args = clazzAndArgs.length == 1 ? new String [0] : Arrays.copyOfRange(clazzAndArgs, 1, clazzAndArgs.length); } dataGen.initialize(args);
          Hide
          ramkrishna.s.vasudevan added a comment -

          I missed a thing in the review, now after this fix

           if (LoadTestTool.isSecure(getConf())) {
                sb.append(authnFileName);
                sb.append(COLON);
              }
              sb.append(superUser);
              sb.append(COLON);
              sb.append(userNames);
              sb.append(COLON);
              sb.append(Integer.toString(SPECIAL_PERM_CELL_INSERTION_FACTOR));
          

          WE need to change the code in LoadTestDataGeneratorWithACL

          super.initialize(args);
              if (args.length != 3) {
                throw new IllegalArgumentException(
                    "LoadTestDataGeneratorWithACL can have "
                        + "1st arguement which would be super user, the 2nd argument "
                        + "would be the user list and the 3rd argument should be the factor representing "
                        + "the row keys for which only write ACLs will be added.");
              }
              String temp = args[1];
              // This will be comma separated list of expressions.
              this.userNames = temp.split(COMMA);
              this.specialPermCellInsertionFactor = Integer.parseInt(args[2]);
          

          We may have to add if condition here to handle the no of args correctly.

          Show
          ramkrishna.s.vasudevan added a comment - I missed a thing in the review, now after this fix if (LoadTestTool.isSecure(getConf())) { sb.append(authnFileName); sb.append(COLON); } sb.append(superUser); sb.append(COLON); sb.append(userNames); sb.append(COLON); sb.append( Integer .toString(SPECIAL_PERM_CELL_INSERTION_FACTOR)); WE need to change the code in LoadTestDataGeneratorWithACL super .initialize(args); if (args.length != 3) { throw new IllegalArgumentException( "LoadTestDataGeneratorWithACL can have " + "1st arguement which would be super user, the 2nd argument " + "would be the user list and the 3rd argument should be the factor representing " + "the row keys for which only write ACLs will be added." ); } String temp = args[1]; // This will be comma separated list of expressions. this .userNames = temp.split(COMMA); this .specialPermCellInsertionFactor = Integer .parseInt(args[2]); We may have to add if condition here to handle the no of args correctly.
          Hide
          Hudson added a comment -

          FAILURE: Integrated in HBase-0.98 #312 (See https://builds.apache.org/job/HBase-0.98/312/)
          HBASE-10831 IntegrationTestIngestWithACL is not setting up LoadTestTool correctly (Vandana Ayyalasomayajula) (apurtell: rev d89342e4166c3f969d79e6be57d7efd0c37cd5b2)

          • hbase-it/src/test/java/org/apache/hadoop/hbase/IntegrationTestIngestWithACL.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/util/LoadTestTool.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedUpdaterWithACL.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedReaderWithACL.java
          Show
          Hudson added a comment - FAILURE: Integrated in HBase-0.98 #312 (See https://builds.apache.org/job/HBase-0.98/312/ ) HBASE-10831 IntegrationTestIngestWithACL is not setting up LoadTestTool correctly (Vandana Ayyalasomayajula) (apurtell: rev d89342e4166c3f969d79e6be57d7efd0c37cd5b2) hbase-it/src/test/java/org/apache/hadoop/hbase/IntegrationTestIngestWithACL.java hbase-server/src/test/java/org/apache/hadoop/hbase/util/LoadTestTool.java hbase-server/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedUpdaterWithACL.java hbase-server/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedReaderWithACL.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in HBase-TRUNK #5138 (See https://builds.apache.org/job/HBase-TRUNK/5138/)
          HBASE-10831 IntegrationTestIngestWithACL is not setting up LoadTestTool correctly (Vandana Ayyalasomayajula) (apurtell: rev 41691e469ae4f59aa04bfec99b1e8a19699f1d04)

          • hbase-server/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedReaderWithACL.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/util/LoadTestTool.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedUpdaterWithACL.java
          • hbase-it/src/test/java/org/apache/hadoop/hbase/IntegrationTestIngestWithACL.java
          Show
          Hudson added a comment - FAILURE: Integrated in HBase-TRUNK #5138 (See https://builds.apache.org/job/HBase-TRUNK/5138/ ) HBASE-10831 IntegrationTestIngestWithACL is not setting up LoadTestTool correctly (Vandana Ayyalasomayajula) (apurtell: rev 41691e469ae4f59aa04bfec99b1e8a19699f1d04) hbase-server/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedReaderWithACL.java hbase-server/src/test/java/org/apache/hadoop/hbase/util/LoadTestTool.java hbase-server/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedUpdaterWithACL.java hbase-it/src/test/java/org/apache/hadoop/hbase/IntegrationTestIngestWithACL.java
          Hide
          Andrew Purtell added a comment -

          Committed to trunk and 0.98. Thanks for the fix Vandana

          Show
          Andrew Purtell added a comment - Committed to trunk and 0.98. Thanks for the fix Vandana
          Hide
          Vandana Ayyalasomayajula added a comment -

          Addressed review comments. Thanks for prompt reviews.

          Show
          Vandana Ayyalasomayajula added a comment - Addressed review comments. Thanks for prompt reviews.
          Hide
          Andrew Purtell added a comment -

          Deadline for commit for 0.98.3 is tomorrow, otherwise lets push to .4.

          Show
          Andrew Purtell added a comment - Deadline for commit for 0.98.3 is tomorrow, otherwise lets push to .4.
          Hide
          Anoop Sam John added a comment -
          +    if (cmd.hasOption(OPT_AUTHN)) {
          +      authnFileName = cmd.getOptionValue(OPT_AUTHN);
          +      if(LoadTestTool.isSecure(getConf()) && (StringUtils.isEmpty(authnFileName))) {
          +        super.printUsage();
          

          When LoadTestTool.isSecure() is true we have to ensure the OPT_AUTHN is provided and having a non empty value. The above check will not happen when the option is not at all provided. We need change the check

          +    String keyTableFileLocation = conf.get(keyTabFileConfKey);
          

          Rename to keyTabFileLocation

          Else the patch looks good to me.

          Show
          Anoop Sam John added a comment - + if (cmd.hasOption(OPT_AUTHN)) { + authnFileName = cmd.getOptionValue(OPT_AUTHN); + if (LoadTestTool.isSecure(getConf()) && (StringUtils.isEmpty(authnFileName))) { + super .printUsage(); When LoadTestTool.isSecure() is true we have to ensure the OPT_AUTHN is provided and having a non empty value. The above check will not happen when the option is not at all provided. We need change the check + String keyTableFileLocation = conf.get(keyTabFileConfKey); Rename to keyTabFileLocation Else the patch looks good to me.
          Hide
          ramkrishna.s.vasudevan added a comment -

          +1 on patch.

          Show
          ramkrishna.s.vasudevan added a comment - +1 on patch.
          Hide
          Vandana Ayyalasomayajula added a comment -

          Addressed comments.

          Show
          Vandana Ayyalasomayajula added a comment - Addressed comments.
          Hide
          Vandana Ayyalasomayajula added a comment -

          Actually there was a misconfiguration on my side which caused the test to fail in local mode. It passed when I ran it again. I think its a good idea to fail the test in case there are missing properties. I will upload a new patch soon. Thanks for the quick comments.

          Show
          Vandana Ayyalasomayajula added a comment - Actually there was a misconfiguration on my side which caused the test to fail in local mode. It passed when I ran it again. I think its a good idea to fail the test in case there are missing properties. I will upload a new patch soon. Thanks for the quick comments.
          Hide
          ramkrishna.s.vasudevan added a comment -

          Patch looks great. Should we have a validation if all the properties required are specified in the file? If not found just fail the test?
          Rest looks good to me.

          However, this patch does not fix local test failure using mvn verify command.

          What is the failure you get when run in local mode?
          I tried running it in a single node but ensured that the users that are passed are created in my local machine. It worked.

          Show
          ramkrishna.s.vasudevan added a comment - Patch looks great. Should we have a validation if all the properties required are specified in the file? If not found just fail the test? Rest looks good to me. However, this patch does not fix local test failure using mvn verify command. What is the failure you get when run in local mode? I tried running it in a single node but ensured that the users that are passed are created in my local machine. It worked.
          Hide
          Vandana Ayyalasomayajula added a comment -

          Fixed compilation error with previous patch.

          Show
          Vandana Ayyalasomayajula added a comment - Fixed compilation error with previous patch.
          Hide
          Vandana Ayyalasomayajula added a comment -

          The command to use on cluster would be

          bin/hbase org.apache.hadoop.hbase.IntegrationTestIngestWithACL -superuser hbase -userlist user3 -authinfo authn.properties
          

          Contents of the "authn.properties" file, which can on hbase classpath. I put the file in conf directory.

          hbase.user3.keytab.file=<file_location>
          hbase.user3.kerberos.principal=user3/_HOST@REALM.COM
          hbase.hbase.keytab.file=<file_location>
          hbase.hbase.kerberos.principal=hbase/_HOST@REALM.COM
          
          Show
          Vandana Ayyalasomayajula added a comment - The command to use on cluster would be bin/hbase org.apache.hadoop.hbase.IntegrationTestIngestWithACL -superuser hbase -userlist user3 -authinfo authn.properties Contents of the "authn.properties" file, which can on hbase classpath. I put the file in conf directory. hbase.user3.keytab.file=<file_location> hbase.user3.kerberos.principal=user3/_HOST@REALM.COM hbase.hbase.keytab.file=<file_location> hbase.hbase.kerberos.principal=hbase/_HOST@REALM.COM
          Hide
          Vandana Ayyalasomayajula added a comment -

          Patch that works for trunk and 98. The test passes successfully on secure hadoop cluster. However, this patch does not fix local test failure using mvn verify command.

          Show
          Vandana Ayyalasomayajula added a comment - Patch that works for trunk and 98. The test passes successfully on secure hadoop cluster. However, this patch does not fix local test failure using mvn verify command.
          Hide
          ramkrishna.s.vasudevan added a comment -

          Thanks Vandana. We were planning to have a look at this one this week.

          Show
          ramkrishna.s.vasudevan added a comment - Thanks Vandana. We were planning to have a look at this one this week.
          Hide
          Vandana Ayyalasomayajula added a comment -

          I got the test running on a secure cluster. The test does not have authentication related code for authenticating super user and users ( passed as part of the user list). So the test fails with the exception mentioned in https://issues.apache.org/jira/browse/HBASE-10831?focusedCommentId=13981816&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13981816.
          when I added keytab files and principals for the each of the users, the test seems to pass. I still have not figured out why it fails while running on local cluster though. I will work on creating the patch for making the test pass on secure cluster.

          Show
          Vandana Ayyalasomayajula added a comment - I got the test running on a secure cluster. The test does not have authentication related code for authenticating super user and users ( passed as part of the user list). So the test fails with the exception mentioned in https://issues.apache.org/jira/browse/HBASE-10831?focusedCommentId=13981816&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13981816 . when I added keytab files and principals for the each of the users, the test seems to pass. I still have not figured out why it fails while running on local cluster though. I will work on creating the patch for making the test pass on secure cluster.
          Hide
          Vandana Ayyalasomayajula added a comment -

          I have tried with 0.98.1RC3 tag, with the following command:

          mvn clean install -DskipTests && cd hbase-it && mvn failsafe:integration-test -Dit.test=IntegrationTestIngestWithACL
          

          It fails. But on trunk it passes though. I tried impersonation settings on my cluster and even then they fail. If someone has got the test running on
          0.98.x version, it would be great help to let me know the command /version to make this test pass.

          Show
          Vandana Ayyalasomayajula added a comment - I have tried with 0.98.1RC3 tag, with the following command: mvn clean install -DskipTests && cd hbase-it && mvn failsafe:integration-test -Dit.test=IntegrationTestIngestWithACL It fails. But on trunk it passes though. I tried impersonation settings on my cluster and even then they fail. If someone has got the test running on 0.98.x version, it would be great help to let me know the command /version to make this test pass.
          Hide
          Andrew Purtell added a comment -

          Over on HBASE-11089 Ted Yu suggests setting up impersonation.

          The above test failure was due to the second user in the test not being able to authenticate using kerberos.

          This can be solved using impersonation which is described here : http://hadoop.apache.org/docs/r1.2.1/Secure_Impersonation.html

          The superuser needs to authenticate using kerberos. The superuser can impersonate any member of the specified groups.

          Show
          Andrew Purtell added a comment - Over on HBASE-11089 Ted Yu suggests setting up impersonation. The above test failure was due to the second user in the test not being able to authenticate using kerberos. This can be solved using impersonation which is described here : http://hadoop.apache.org/docs/r1.2.1/Secure_Impersonation.html The superuser needs to authenticate using kerberos. The superuser can impersonate any member of the specified groups.
          Hide
          ramkrishna.s.vasudevan added a comment -

          Can you take the released version of 0.98.1 and check that. I think you are using an some RC version of 0.98.1.
          In HBASE-10675 Ted, reported the same issue. Let me check once too.

          Show
          ramkrishna.s.vasudevan added a comment - Can you take the released version of 0.98.1 and check that. I think you are using an some RC version of 0.98.1. In HBASE-10675 Ted, reported the same issue. Let me check once too.
          Hide
          Vandana Ayyalasomayajula added a comment -

          I am using 0.98.1-SNAPSHOT. I did pass the super user as param. I exact command I ran was:

          hbase org.apache.hadoop.hbase.IntegrationTestIngestWithACL -superuser hbase -userlist avandana
          
          Show
          Vandana Ayyalasomayajula added a comment - I am using 0.98.1-SNAPSHOT. I did pass the super user as param. I exact command I ran was: hbase org.apache.hadoop.hbase.IntegrationTestIngestWithACL -superuser hbase -userlist avandana
          Hide
          ramkrishna.s.vasudevan added a comment -

          Which version? In HBASE-10675 you could pass the super user also as a param.

          Show
          ramkrishna.s.vasudevan added a comment - Which version? In HBASE-10675 you could pass the super user also as a param.
          Hide
          Vandana Ayyalasomayajula added a comment -

          When I try to run this test with the valid super user and user list specified: I get the following exception:

          14/04/26 00:34:15 WARN ipc.RpcClient: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
          14/04/26 00:34:15 FATAL ipc.RpcClient: SASL authentication failed. The most likely cause is missing or invalid credentials. Consider 'kinit'.
          javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
          	at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
          	at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:169)
          	at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupSaslConnection(RpcClient.java:768)
          	at org.apache.hadoop.hbase.ipc.RpcClient$Connection.access$600(RpcClient.java:357)
          	at org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:889)
          	at org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:886)
          	at java.security.AccessController.doPrivileged(Native Method)
          	at javax.security.auth.Subject.doAs(Subject.java:415)
          	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
          	at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:886)
          	at org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1536)
          	at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1435)
          	at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1654)
          	at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1712)
          	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.get(ClientProtos.java:29876)
          	at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRowOrBefore(ProtobufUtil.java:1470)
          	at org.apache.hadoop.hbase.client.HTable$2.call(HTable.java:706)
          	at org.apache.hadoop.hbase.client.HTable$2.call(HTable.java:704)
          	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:114)
          	at org.apache.hadoop.hbase.client.HTable.getRowOrBefore(HTable.java:710)
          	at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:144)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:1158)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1222)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1110)
          	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1067)
          	at org.apache.hadoop.hbase.client.AsyncProcess.findDestLocation(AsyncProcess.java:356)
          	at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:301)
          	at org.apache.hadoop.hbase.client.HTable.backgroundFlushCommits(HTable.java:955)
          	at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1239)
          	at org.apache.hadoop.hbase.client.HTable.put(HTable.java:901)
          	at org.apache.hadoop.hbase.util.MultiThreadedWriterWithACL$HBaseWriterThreadWithACL$WriteAccessAction.run(MultiThreadedWriterWithACL.java:130)
          	at java.security.AccessController.doPrivileged(Native Method)
          	at javax.security.auth.Subject.doAs(Subject.java:415)
          	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          	at java.lang.reflect.Method.invoke(Method.java:601)
          	at org.apache.hadoop.hbase.util.Methods.call(Methods.java:39)
          	at org.apache.hadoop.hbase.security.User.call(User.java:434)
          	at org.apache.hadoop.hbase.security.User.access$300(User.java:49)
          	at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:288)
          	at org.apache.hadoop.hbase.util.MultiThreadedWriterWithACL$HBaseWriterThreadWithACL.insert(MultiThreadedWriterWithACL.java:96)
          	at org.apache.hadoop.hbase.util.MultiThreadedWriter$HBaseWriterThread.run(MultiThreadedWriter.java:108)
          Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
          	at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
          	at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
          	at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
          	at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
          	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
          	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
          	at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
          	... 43 more
          
          Show
          Vandana Ayyalasomayajula added a comment - When I try to run this test with the valid super user and user list specified: I get the following exception: 14/04/26 00:34:15 WARN ipc.RpcClient: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] 14/04/26 00:34:15 FATAL ipc.RpcClient: SASL authentication failed. The most likely cause is missing or invalid credentials. Consider 'kinit'. javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212) at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:169) at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupSaslConnection(RpcClient.java:768) at org.apache.hadoop.hbase.ipc.RpcClient$Connection.access$600(RpcClient.java:357) at org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:889) at org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:886) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:886) at org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1536) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1435) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1654) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1712) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.get(ClientProtos.java:29876) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRowOrBefore(ProtobufUtil.java:1470) at org.apache.hadoop.hbase.client.HTable$2.call(HTable.java:706) at org.apache.hadoop.hbase.client.HTable$2.call(HTable.java:704) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:114) at org.apache.hadoop.hbase.client.HTable.getRowOrBefore(HTable.java:710) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:144) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:1158) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1222) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1110) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1067) at org.apache.hadoop.hbase.client.AsyncProcess.findDestLocation(AsyncProcess.java:356) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:301) at org.apache.hadoop.hbase.client.HTable.backgroundFlushCommits(HTable.java:955) at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1239) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:901) at org.apache.hadoop.hbase.util.MultiThreadedWriterWithACL$HBaseWriterThreadWithACL$WriteAccessAction.run(MultiThreadedWriterWithACL.java:130) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.hadoop.hbase.util.Methods.call(Methods.java:39) at org.apache.hadoop.hbase.security.User.call(User.java:434) at org.apache.hadoop.hbase.security.User.access$300(User.java:49) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:288) at org.apache.hadoop.hbase.util.MultiThreadedWriterWithACL$HBaseWriterThreadWithACL.insert(MultiThreadedWriterWithACL.java:96) at org.apache.hadoop.hbase.util.MultiThreadedWriter$HBaseWriterThread.run(MultiThreadedWriter.java:108) Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147) at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121) at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187) at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223) at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212) at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193) ... 43 more
          Hide
          ramkrishna.s.vasudevan added a comment -

          I tried this way

          ./hbase --config /usr/lib/hbase/conf/ org.apache.hadoop.hbase.IntegrationTestsDriver -r .*IntegrationTestIngestWithACL.*
          

          this takes the hardcoded values in IntegrationTestIngestWithACL for the users and superuser.

          ./hbase --config /usr/lib/hbase/conf/ org.apache.hadoop.hbase.IntegrationTestIngestWithACL -superuser owner -userlist user1,user2
          

          Running this way would take params for user and super user. Let me check that mvn verify type of running.

          Show
          ramkrishna.s.vasudevan added a comment - I tried this way ./hbase --config /usr/lib/hbase/conf/ org.apache.hadoop.hbase.IntegrationTestsDriver -r .*IntegrationTestIngestWithACL.* this takes the hardcoded values in IntegrationTestIngestWithACL for the users and superuser. ./hbase --config /usr/lib/hbase/conf/ org.apache.hadoop.hbase.IntegrationTestIngestWithACL -superuser owner -userlist user1,user2 Running this way would take params for user and super user. Let me check that mvn verify type of running.
          Hide
          Andrew Purtell added a comment -

          Run the test with 'mvn verify' or IntegrationTestDriver and it will fail.

          Show
          Andrew Purtell added a comment - Run the test with 'mvn verify' or IntegrationTestDriver and it will fail.
          Hide
          ramkrishna.s.vasudevan added a comment -

          Any other log regarding the error? How was the parameters passed to the system?

          Show
          ramkrishna.s.vasudevan added a comment - Any other log regarding the error? How was the parameters passed to the system?
          Hide
          Andrew Purtell added a comment -
          Show
          Andrew Purtell added a comment - Ping ramkrishna.s.vasudevan

            People

            • Assignee:
              Vandana Ayyalasomayajula
              Reporter:
              Andrew Purtell
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development