[IMPALA-10082] Concurrent invalidate metadata and create/drop table cause discrepancy in metadata

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: Impala 4.0.0
• Component/s: Catalog
• Epic Color: ghx-label-5
• Assignee: Unassigned
• Reporter: Robbie Zhang
• Created: 13/Aug/20 12:59
• Updated: 21/Dec/20 20:01

Description

The symptom is similar to IMPALA-7093 but is a different issue. Here is how I reproduce it:

1) Ran the first script to keep running create/insert/drop queries

#!/bin/bash

while [ 1 ]
do
  shell/impala-shell -q "create table if not exists test(i int); insert into test(i) values(1); drop table test;" 2>&1| tee test.output
  n=`egrep "Exception" test.output | wc -l`
  if [ $n -lt 0 ]; then
    rm -f /tmp/testing
    exit
  fi
done

2) Ran the second script to keep running global invalidate metadata

#!/bin/bash

while [ 1 ]
do
  shell/impala-shell -q "invalidate metadata"
done

Sometime later, the first scrip ended with "Table default.test does not exist":

Starting Impala Shell with no authentication using Python 2.7.12
Warning: live_progress only applies to interactive shell sessions, and is being skipped for now.
Opened TCP connection to localhost:21000
Connected to localhost:21000
Server version: impalad version 4.0.0-SNAPSHOT DEBUG (build f95f7940e4a290d75ee85fd78e85bc26795f0f9f)
Query: create table if not exists test(i int)
Fetched 1 row(s) in 0.01s
Query: insert into test(i) values(1)
Query submitted at: 2020-08-13 22:57:51 (Coordinator: http://impala34:25000)
ERROR: AnalysisException: Table does not exist: default.test


Could not execute command: insert into test(i) values(1)

Even after I change to local catalog mode, this issue still exists:

Starting Impala Shell with no authentication using Python 2.7.12
Warning: live_progress only applies to interactive shell sessions, and is being skipped for now.
Opened TCP connection to localhost:21000
Connected to localhost:21000
Server version: impalad version 4.0.0-SNAPSHOT DEBUG (build f95f7940e4a290d75ee85fd78e85bc26795f0f9f)
Query: create table if not exists test(i int)
Fetched 1 row(s) in 0.07s
Query: insert into test(i) values(1)
Query submitted at: 2020-08-13 22:10:16 (Coordinator: http://impala34:25000)
ERROR: AnalysisException: org.apache.impala.catalog.TableLoadingException: Could not load table default.test from catalog
CAUSED BY: TableLoadingException: Could not load table default.test from catalog
CAUSED BY: TException: TGetPartialCatalogObjectResponse(status:TStatus(status_code:GENERAL, error_msgs:[TableLoadingException: Table default.test no longer exists in the Hive MetaStore. Run 'invalidate metadata default.test' to update the Impala catalog.]), lookup_status:OK)


Could not execute command: insert into test(i) values(1)

And in local catalog mode, the newly created table was lost but it's still visible in the coordinator. After running 'invalidate metadata default.test', the table disappeared at all.