HCatalog
  1. HCatalog
  2. HCATALOG-90

HCat Thrift Server AKA Metastore won't start up successfully the first time without a kinit

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.3
    • Component/s: None
    • Labels:

      Description

      Hcat thrift server (AKA metastore) wouldn't startup because it kept failing to access secured hdfs. The server does not seem to login with kerberos even after providing the credentials in the config file. I'm assuming that's what it's supposed to use:

      hive.metastore.kerberos.keytab.file
      hive.metastore.kerberos.principal

      Looking through the hive code, the hdfs client does not seem to pickup any kerberos configurations/credentials.

      I finally got the server to start by manually logging myself in via kinit then running the start script. Is this normal operation? I didn't see it documented in installation instructions.

        Activity

        Hide
        Francis Liu added a comment -

        tried it on a snapshot of metastore 0.9, which is what hcat trunk pulls in. it works.

        Show
        Francis Liu added a comment - tried it on a snapshot of metastore 0.9, which is what hcat trunk pulls in. it works.
        Hide
        Ashutosh Chauhan added a comment -

        This is fixed with HIVE-2411

        Show
        Ashutosh Chauhan added a comment - This is fixed with HIVE-2411
        Hide
        Francis Liu added a comment -

        @Ashutosh, sorry for the delayed reply....the configuration works. Though the team feels this is more of a short-term work around. Will keep this Jira until it is fixed hive-side. Perhaps we can add the jira here for tracking?

        Show
        Francis Liu added a comment - @Ashutosh, sorry for the delayed reply....the configuration works. Though the team feels this is more of a short-term work around. Will keep this Jira until it is fixed hive-side. Perhaps we can add the jira here for tracking?
        Hide
        Ashutosh Chauhan added a comment -

        @Francis,

        Try with the following configuration:

        <property>
          <name>hive.metastore.checkForDefaultDb</name>
          <value>false</value>
        </property>
        
        Show
        Ashutosh Chauhan added a comment - @Francis, Try with the following configuration: <property> <name>hive.metastore.checkForDefaultDb</name> <value> false </value> </property>
        Hide
        Francis Liu added a comment -

        Yes it is readable.

        For more info here's the log message:

        Exception in thread "main" MetaException(message:Got exception: java.io.IOException Call to gsbl90640.blue.ygrid.yahoo.com/98
        .137.97.190:8020 failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caus
        ed by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)])
        at org.apache.hadoop.hive.metastore.MetaStoreUtils.logAndThrowMetaException(MetaStoreUtils.java:779)
        at org.apache.hadoop.hive.metastore.Warehouse.getFs(Warehouse.java:103)
        at org.apache.hadoop.hive.metastore.Warehouse.getDnsPath(Warehouse.java:133)
        at org.apache.hadoop.hive.metastore.Warehouse.getWhRoot(Warehouse.java:148)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getDefaultDatabasePath(HiveMetaStore.java:578)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB_core(HiveMetaStore.java:465)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.access$300(HiveMetaStore.java:125)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$5.run(HiveMetaStore.java:485)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$5.run(HiveMetaStore.java:482)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.executeWithRetry(HiveMetaStore.java:338)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:482)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:250)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.<init>(HiveMetaStore.java:207)
        at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:3398)
        at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:3379)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

        Show
        Francis Liu added a comment - Yes it is readable. For more info here's the log message: Exception in thread "main" MetaException(message:Got exception: java.io.IOException Call to gsbl90640.blue.ygrid.yahoo.com/98 .137.97.190:8020 failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caus ed by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]) at org.apache.hadoop.hive.metastore.MetaStoreUtils.logAndThrowMetaException(MetaStoreUtils.java:779) at org.apache.hadoop.hive.metastore.Warehouse.getFs(Warehouse.java:103) at org.apache.hadoop.hive.metastore.Warehouse.getDnsPath(Warehouse.java:133) at org.apache.hadoop.hive.metastore.Warehouse.getWhRoot(Warehouse.java:148) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getDefaultDatabasePath(HiveMetaStore.java:578) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB_core(HiveMetaStore.java:465) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.access$300(HiveMetaStore.java:125) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$5.run(HiveMetaStore.java:485) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$5.run(HiveMetaStore.java:482) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.executeWithRetry(HiveMetaStore.java:338) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:482) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:250) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.<init>(HiveMetaStore.java:207) at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:3398) at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:3379) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
        Hide
        Ashutosh Chauhan added a comment -

        Just to make sure is keytab file readable by the user starting the metastore?

        Show
        Ashutosh Chauhan added a comment - Just to make sure is keytab file readable by the user starting the metastore?

          People

          • Assignee:
            Ashutosh Chauhan
            Reporter:
            Francis Liu
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development