[IMPALA-6424] REFRESH right after invalidate metadata <table> loads file metadata twice

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 2.11.0
• Fix Version/s: Impala 2.12.0
• Component/s: Catalog
• Labels: performance
• Epic Color: ghx-label-5
• Assignee: Dimitris Tsirogiannis
• Reporter: Juan Yu
• Created: 19/Jan/18 15:57
• Updated: 21/Feb/18 18:21
• Resolved: 21/Feb/18 18:20

Description

Compare with normal REFRESH, REFRESH right after Invalidate metadata <table> load file metadata twice and takes 2x time. The second refresh seems redundant.

I0119 07:46:41.107390 26758 CatalogServiceCatalog.java:1518] Invalidating table metadata: s3.catalog_sales
I0119 07:46:43.002053 26309 catalog-server.cc:331] Publishing update : TABLE:s3.catalog_sales@1166
I0119 07:46:43.002068 26309 catalog-server.cc:331] Publishing update : CATALOG:b0f520a5e2ab4056:b7e2e045fa39d625@1166

I0119 07:46:46.696725 26758 TableLoadingMgr.java:70] Loading metadata for table: s3.catalog_sales
I0119 07:46:46.696781 26758 TableLoadingMgr.java:72] Remaining items in queue: 0. Loads in progress: 1
I0119 07:46:46.696857 27023 TableLoader.java:58] Loading metadata for: s3.catalog_sales
I0119 07:46:46.713222 27023 HdfsTable.java:1206] Fetching partition metadata from the Metastore: s3.catalog_sales
I0119 07:46:46.905102 27023 HdfsTable.java:1210] Fetched partition metadata from the Metastore: s3.catalog_sales
I0119 07:46:46.939254 27023 HdfsTable.java:834] Loading file and block metadata for 1837 paths for table s3.catalog_sales using a thread pool of size 20
I0119 07:47:00.426975 27023 HdfsTable.java:874] Loaded file and block metadata for s3.catalog_sales
I0119 07:47:00.427062 27023 TableLoader.java:97] Loaded metadata for: s3.catalog_sales

I0119 07:47:00.427243 26758 CatalogServiceCatalog.java:1433] Refreshing table metadata: s3.catalog_sales
I0119 07:47:00.441572 26758 HdfsTable.java:1193] Incrementally loading table metadata for: s3.catalog_sales
I0119 07:47:00.456437 26758 HdfsTable.java:834] Loading file and block metadata for 1837 paths for table s3.catalog_sales using a thread pool of size 20
I0119 07:47:14.038097 26758 HdfsTable.java:874] Loaded file and block metadata for s3.catalog_sales
I0119 07:47:14.038132 26758 HdfsTable.java:1203] Incrementally loaded table metadata for: s3.catalog_sales
I0119 07:47:14.038179 26758 CatalogServiceCatalog.java:1456] Refreshed table metadata: s3.catalog_sales
I0119 07:47:14.062625 26309 catalog-server.cc:331] Publishing update : TABLE:s3.catalog_sales@1168
I0119 07:47:14.062645 26309 catalog-server.cc:331] Publishing update : CATALOG:b0f520a5e2ab4056:b7e2e045fa39d625@1168