[LENS-907] Subsequent calls to metastore API to fetch native tables throws error after the first call

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.5
• Component/s: server
• Assignee: Deepak Barr
• Reporter: Deepak Barr
• Created: 18/Dec/15 09:38
• Updated: 24/Feb/16 11:41
• Resolved: 24/Dec/15 11:47

Description

HiveSessionImpl.java

  @Override
  public IMetaStoreClient getMetaStoreClient() throws HiveSQLException {
    if (metastoreClient == null) {
      try {
        metastoreClient = new HiveMetaStoreClient(getHiveConf());
      } catch (MetaException e) {
        throw new HiveSQLException(e);
      }
    }
    return metastoreClient;
  }

CubeMetastoreServiceImpl.java

private List<String> getNativeTablesFromDB(LensSessionHandle sessionid, String dbName, boolean prependDbName)
    throws LensException {
    IMetaStoreClient msc = null;
    try {
      msc = getSession(sessionid).getMetaStoreClient();
      List<String> tables = msc.getAllTables(
        dbName);
      List<String> result = new ArrayList<String>();
      if (tables != null && !tables.isEmpty()) {
        List<org.apache.hadoop.hive.metastore.api.Table> tblObjects =
          msc.getTableObjectsByName(dbName, tables);
        Iterator<org.apache.hadoop.hive.metastore.api.Table> it = tblObjects.iterator();
        while (it.hasNext()) {
          org.apache.hadoop.hive.metastore.api.Table tbl = it.next();
          if (tbl.getParameters().get(MetastoreConstants.TABLE_TYPE_KEY) == null) {
            if (prependDbName) {
              result.add(dbName + "." + tbl.getTableName());
            } else {
              result.add(tbl.getTableName());
            }
          }
        }
      }
      return result;
    } catch (Exception e) {
      throw new LensException("Error getting native tables from DB", e);
    } finally {
      if (null != msc) {
        msc.close();
      }
    }
  }

Scenario :

When we call the metastore resource API to fetch nativetables in a db, The getNativeTablesFromDB() method fetches the list of tables using (msc)HiveMetaStoreClient instance in HiveSessionImpl(). In finally, the msc instance is closed. The first call will succeed and return the list of tables.
Now, if we call the API again to fetch nativetables, the same instance of HiveMetaStoreClient which was closed in previous call. This results in the following error on msc.getAllTables() method call.

org.apache.thrift.transport.TTransportException: Cannot write to null outputStream
at org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:142) ~[libthrift-0.9.0.jar:0.9.0]
at org.apache.thrift.protocol.TBinaryProtocol.writeI32(TBinaryProtocol.java:163) ~[libthrift-0.9.0.jar:0.9.0]
at org.apache.thrift.protocol.TBinaryProtocol.writeMessageBegin(TBinaryProtocol.java:91) ~[libthrift-0.9.0.jar:0.9.0]
at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62) ~[libthrift-0.9.0.jar:0.9.0]
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.send_get_database(ThriftHiveMetastore.java:538) ~[hive-metastore-0.13.3-inm.jar:0.13.3-inm]
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_database(ThriftHiveMetastore.java:530) ~[hive-metastore-0.13.3-inm.jar:0.13.3-inm]
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabase(HiveMetaStoreClient.java:970) ~[hive-metastore-0.13.3-inm.jar:0.13.3-inm]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_45]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_45]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_45]
at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_45]
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:89) ~[hive-metastore-0.13.3-inm.jar:0.13.3-inm]
at com.sun.proxy.$Proxy7.getDatabase(Unknown Source) [na:na]
at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1156) [hive-exec-0.13.3-inm.jar:0.13.3-inm]
at org.apache.hadoop.hive.ql.metadata.Hive.databaseExists(Hive.java:1145) [hive-exec-0.13.3-inm.jar:0.13.3-inm]
at org.apache.lens.server.metastore.CubeMetastoreServiceImpl.getAllNativeTableNames(CubeMetastoreServiceImpl.java:1229) [classes/:na]

Solution :

There could be 2 solutions I can think of -

1. Do not close HiveMetaStoreClient object in getNativeTablesFromDB(). ( This was the case before LENS-796 )
2. Apply HIVE-8320 patch to inmobi hive codebase. We applied the patch for our system and It solves the problem.

Any thoughts ?