Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-6424

REFRESH right after invalidate metadata <table> loads file metadata twice

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: Impala 2.11.0
    • Fix Version/s: Impala 2.12.0
    • Component/s: Catalog
    • Labels:
    • Epic Color:
      ghx-label-5

      Description

      Compare with normal REFRESH, REFRESH right after Invalidate metadata <table> load file metadata twice and takes 2x time. The second refresh seems redundant.

      I0119 07:46:41.107390 26758 CatalogServiceCatalog.java:1518] Invalidating table metadata: s3.catalog_sales
      I0119 07:46:43.002053 26309 catalog-server.cc:331] Publishing update : TABLE:s3.catalog_sales@1166
      I0119 07:46:43.002068 26309 catalog-server.cc:331] Publishing update : CATALOG:b0f520a5e2ab4056:b7e2e045fa39d625@1166

      I0119 07:46:46.696725 26758 TableLoadingMgr.java:70] Loading metadata for table: s3.catalog_sales
      I0119 07:46:46.696781 26758 TableLoadingMgr.java:72] Remaining items in queue: 0. Loads in progress: 1
      I0119 07:46:46.696857 27023 TableLoader.java:58] Loading metadata for: s3.catalog_sales
      I0119 07:46:46.713222 27023 HdfsTable.java:1206] Fetching partition metadata from the Metastore: s3.catalog_sales
      I0119 07:46:46.905102 27023 HdfsTable.java:1210] Fetched partition metadata from the Metastore: s3.catalog_sales
      I0119 07:46:46.939254 27023 HdfsTable.java:834] Loading file and block metadata for 1837 paths for table s3.catalog_sales using a thread pool of size 20
      I0119 07:47:00.426975 27023 HdfsTable.java:874] Loaded file and block metadata for s3.catalog_sales
      I0119 07:47:00.427062 27023 TableLoader.java:97] Loaded metadata for: s3.catalog_sales

      I0119 07:47:00.427243 26758 CatalogServiceCatalog.java:1433] Refreshing table metadata: s3.catalog_sales
      I0119 07:47:00.441572 26758 HdfsTable.java:1193] Incrementally loading table metadata for: s3.catalog_sales
      I0119 07:47:00.456437 26758 HdfsTable.java:834] Loading file and block metadata for 1837 paths for table s3.catalog_sales using a thread pool of size 20
      I0119 07:47:14.038097 26758 HdfsTable.java:874] Loaded file and block metadata for s3.catalog_sales
      I0119 07:47:14.038132 26758 HdfsTable.java:1203] Incrementally loaded table metadata for: s3.catalog_sales
      I0119 07:47:14.038179 26758 CatalogServiceCatalog.java:1456] Refreshed table metadata: s3.catalog_sales
      I0119 07:47:14.062625 26309 catalog-server.cc:331] Publishing update : TABLE:s3.catalog_sales@1168
      I0119 07:47:14.062645 26309 catalog-server.cc:331] Publishing update : CATALOG:b0f520a5e2ab4056:b7e2e045fa39d625@1168

        Attachments

          Activity

            People

            • Assignee:
              dtsirogiannis Dimitris Tsirogiannis
              Reporter:
              jyu@cloudera.com Juan Yu
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: