[LENS-796] Slow response times for /metastore/nativetables API

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.4
• Assignee: Deepak Barr
• Reporter: Deepak Barr
• Created: 29/Sep/15 16:28
• Updated: 17/Nov/15 11:49
• Resolved: 06/Oct/15 07:30

Description

To obtain list of native tables, CubeMetastoreService does the following -
1. Fetches the list of tables ( one MetastoreClient call)
2. Filters out the cube tables from the list. The filtering happens by looking at the table properties from the Table object. This table object is obtained with another Metastore call. So, If there are 'n' tables, there will be 'n' metastore calls.

Here is the code snippet :

private List<String> getTablesFromDB(LensSessionHandle sessionid,
String dbName, boolean prependDbName)
throws MetaException, UnknownDBException, HiveSQLException, TException, LensException {
List<String> tables = getSession(sessionid).getMetaStoreClient().getAllTables(
dbName);
List<String> result = new ArrayList<String>();
if (tables != null && !tables.isEmpty()) {
Iterator<String> it = tables.iterator();
while (it.hasNext()) {
String tblName = it.next();
org.apache.hadoop.hive.metastore.api.Table tbl =
getSession(sessionid).getMetaStoreClient().getTable(dbName, tblName);
if (tbl.getParameters().get(MetastoreConstants.TABLE_TYPE_KEY) == null) {
if (prependDbName)

{ result.add(dbName + "." + tblName); }

else

{ result.add(tblName); }

}
}
}
return result;
}

Instead of this, we can directly fetch the list of table objects for our list of table names in a single API call using getMetaStoreClient().getTableObjectsByName() method.

Currently, one of our databases contain 8000+ tables which leads to very very long response times.