[IMPALA-8877] CatalogException during stress test: Table <foo> modified while operation was in progress

• Type: Bug
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: Impala 3.3.0
• Component/s: Catalog
• Labels: catalog-v2
• Epic Color: ghx-label-4
• Assignee: Vihang Karajgaonkar
• Reporter: David Knupp
• Created: 19/Aug/19 18:18
• Updated: 16/Feb/20 20:37

Description

This was hit while running the stress tests to get a baseline on a deployed cluster.

/* Mem: 12850 MB. Coordinator: quasar-mzmnbe-6.vpc.cloudera.com. */
COMPUTE STATS catalog_sales

Query (id=924a50178a5a6146:29d58a7300000000)
  Summary
    Session ID: 5543fb9029e2b71f:f446381b1f59ed81
    Session Type: HIVESERVER2
    HiveServer2 Protocol Version: V6
    Start Time: 2019-08-19 01:26:07.292866000
    End Time: 2019-08-19 01:26:27.248053000
    Query Type: DDL
    Query State: EXCEPTION
    Query Status: CatalogException: Table 'tpcds_300_decimal_parquet.catalog_sales' was modified while operation was in progress, aborting execution.
    Impala Version: impalad version 3.3.0-SNAPSHOT RELEASE (build df3e7c051e2641524fc53a0cd07c2a14decd55f7)
    User: [e-mail]
    Connected User: [e-mail]
    Delegated User: 
    Network Address: ::ffff:10.65.6.19:39174
    Default Db: tpcds_300_decimal_parquet
    Sql Statement: /* Mem: 12850 MB. Coordinator: quasar-mzmnbe-6.vpc.cloudera.com. */
COMPUTE STATS catalog_sales
    Coordinator: quasar-mzmnbe-6.vpc.cloudera.com:22000
    Query Options (set by configuration): ABORT_ON_ERROR=1,MEM_LIMIT=13474201600,MT_DOP=4,EXEC_TIME_LIMIT_S=2147483647,TIMEZONE=America/Los_Angeles,DEFAULT_FILE_FORMAT=4,DEFAULT_TRANSACTIONAL_TYPE=1
    Query Options (set by configuration and planner): ABORT_ON_ERROR=1,MEM_LIMIT=13474201600,MT_DOP=4,EXEC_TIME_LIMIT_S=2147483647,TIMEZONE=America/Los_Angeles,DEFAULT_FILE_FORMAT=4,DEFAULT_TRANSACTIONAL_TYPE=1
    DDL Type: COMPUTE_STATS
    Query Compilation
      Metadata of all 1 tables cached: 5.62s (5622372318)
      Analysis finished: 5.62s (5622560027)
      Authorization finished (noop): 5.62s (5622568284)
      Retried query planning due to inconsistent metadata 7 of 40 times: Catalog object TCatalogObject(type:TABLE, catalog_version:94204, table:TTable(db_name:tpcds_300_decimal_parquet, tbl_name:catalog_sales)) changed version between accesses.: 5.95s (5949859598)
      Planning finished: 5.95s (5949861145)
    Query Timeline
      Query submitted: 0ns (0)
      Planning finished: 5.95s (5950024020)
      Child queries finished: 17.85s (17849072057)
      Rows available: 19.82s (19825080035)
      Unregister query: 19.95s (19955080560)
    Frontend
      - CatalogFetch.ColumnStats.Misses: 34 (34)
      - CatalogFetch.ColumnStats.Requests: 34 (34)
      - CatalogFetch.ColumnStats.Time: 0 (0)
      - CatalogFetch.Config.Hits: 1 (1)
      - CatalogFetch.Config.Requests: 1 (1)
      - CatalogFetch.Config.Time: 0 (0)
      - CatalogFetch.DatabaseList.Hits: 8 (8)
      - CatalogFetch.DatabaseList.Requests: 8 (8)
      - CatalogFetch.DatabaseList.Time: 0 (0)
      - CatalogFetch.PartitionLists.Misses: 1 (1)
      - CatalogFetch.PartitionLists.Requests: 1 (1)
      - CatalogFetch.PartitionLists.Time: 7 (7)
      - CatalogFetch.Partitions.Hits: 1837 (1837)
      - CatalogFetch.Partitions.Misses: 1837 (1837)
      - CatalogFetch.Partitions.Requests: 3674 (3674)
      - CatalogFetch.Partitions.Time: 325 (325)
      - CatalogFetch.RPCs.Bytes: 4.7 MiB (4936030)
      - CatalogFetch.RPCs.Requests: 22 (22)
      - CatalogFetch.RPCs.Time: 343 (343)
      - CatalogFetch.TableNames.Hits: 4 (4)
      - CatalogFetch.TableNames.Misses: 4 (4)
      - CatalogFetch.TableNames.Requests: 8 (8)
      - CatalogFetch.TableNames.Time: 0 (0)
      - CatalogFetch.Tables.Misses: 8 (8)
      - CatalogFetch.Tables.Requests: 8 (8)
      - CatalogFetch.Tables.Time: 74 (74)
      - InactiveTotalTime: 0ns (0)
      - TotalTime: 0ns (0)
  ImpalaServer
    - CatalogOpExecTimer: 1.97s (1972007962)
    - ClientFetchWaitTimer: 0ns (0)
    - InactiveTotalTime: 0ns (0)
    - RowMaterializationTimer: 0ns (0)
    - TotalTime: 0ns (0)
  Child Queries
    Table Stats Query (id=db4821e4aa5bb04d:d4a5ae4500000000)
    Column Stats Query (id=0444367557e3496d:f943511100000000)