Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-6424

REFRESH right after invalidate metadata <table> loads file metadata twice

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • Impala 2.11.0
    • Impala 2.12.0
    • Catalog
    • ghx-label-5

    Description

      Compare with normal REFRESH, REFRESH right after Invalidate metadata <table> load file metadata twice and takes 2x time. The second refresh seems redundant.

      I0119 07:46:41.107390 26758 CatalogServiceCatalog.java:1518] Invalidating table metadata: s3.catalog_sales
      I0119 07:46:43.002053 26309 catalog-server.cc:331] Publishing update : TABLE:s3.catalog_sales@1166
      I0119 07:46:43.002068 26309 catalog-server.cc:331] Publishing update : CATALOG:b0f520a5e2ab4056:b7e2e045fa39d625@1166

      I0119 07:46:46.696725 26758 TableLoadingMgr.java:70] Loading metadata for table: s3.catalog_sales
      I0119 07:46:46.696781 26758 TableLoadingMgr.java:72] Remaining items in queue: 0. Loads in progress: 1
      I0119 07:46:46.696857 27023 TableLoader.java:58] Loading metadata for: s3.catalog_sales
      I0119 07:46:46.713222 27023 HdfsTable.java:1206] Fetching partition metadata from the Metastore: s3.catalog_sales
      I0119 07:46:46.905102 27023 HdfsTable.java:1210] Fetched partition metadata from the Metastore: s3.catalog_sales
      I0119 07:46:46.939254 27023 HdfsTable.java:834] Loading file and block metadata for 1837 paths for table s3.catalog_sales using a thread pool of size 20
      I0119 07:47:00.426975 27023 HdfsTable.java:874] Loaded file and block metadata for s3.catalog_sales
      I0119 07:47:00.427062 27023 TableLoader.java:97] Loaded metadata for: s3.catalog_sales

      I0119 07:47:00.427243 26758 CatalogServiceCatalog.java:1433] Refreshing table metadata: s3.catalog_sales
      I0119 07:47:00.441572 26758 HdfsTable.java:1193] Incrementally loading table metadata for: s3.catalog_sales
      I0119 07:47:00.456437 26758 HdfsTable.java:834] Loading file and block metadata for 1837 paths for table s3.catalog_sales using a thread pool of size 20
      I0119 07:47:14.038097 26758 HdfsTable.java:874] Loaded file and block metadata for s3.catalog_sales
      I0119 07:47:14.038132 26758 HdfsTable.java:1203] Incrementally loaded table metadata for: s3.catalog_sales
      I0119 07:47:14.038179 26758 CatalogServiceCatalog.java:1456] Refreshed table metadata: s3.catalog_sales
      I0119 07:47:14.062625 26309 catalog-server.cc:331] Publishing update : TABLE:s3.catalog_sales@1168
      I0119 07:47:14.062645 26309 catalog-server.cc:331] Publishing update : CATALOG:b0f520a5e2ab4056:b7e2e045fa39d625@1168

      Attachments

        Activity

          People

            dtsirogiannis Dimitris Tsirogiannis
            jyu@cloudera.com Juan Yu
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: