Uploaded image for project: 'Apache Lens (Retired)'
  1. Apache Lens (Retired)
  2. LENS-796

Slow response times for /metastore/nativetables API

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.4
    • Component/s: None
    • Labels:
      None

      Description

      To obtain list of native tables, CubeMetastoreService does the following -
      1. Fetches the list of tables ( one MetastoreClient call)
      2. Filters out the cube tables from the list. The filtering happens by looking at the table properties from the Table object. This table object is obtained with another Metastore call. So, If there are 'n' tables, there will be 'n' metastore calls.

      Here is the code snippet :

      private List<String> getTablesFromDB(LensSessionHandle sessionid,
      String dbName, boolean prependDbName)
      throws MetaException, UnknownDBException, HiveSQLException, TException, LensException {
      List<String> tables = getSession(sessionid).getMetaStoreClient().getAllTables(
      dbName);
      List<String> result = new ArrayList<String>();
      if (tables != null && !tables.isEmpty()) {
      Iterator<String> it = tables.iterator();
      while (it.hasNext()) {
      String tblName = it.next();
      org.apache.hadoop.hive.metastore.api.Table tbl =
      getSession(sessionid).getMetaStoreClient().getTable(dbName, tblName);
      if (tbl.getParameters().get(MetastoreConstants.TABLE_TYPE_KEY) == null) {
      if (prependDbName)

      { result.add(dbName + "." + tblName); }

      else

      { result.add(tblName); }

      }
      }
      }
      return result;
      }

      Instead of this, we can directly fetch the list of table objects for our list of table names in a single API call using getMetaStoreClient().getTableObjectsByName() method.

      Currently, one of our databases contain 8000+ tables which leads to very very long response times.

        Attachments

        1. LENS-796.patch
          4 kB
          Deepak Barr

          Activity

            People

            • Assignee:
              deepak.barr Deepak Barr
              Reporter:
              deepak.barr Deepak Barr
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: