[IMPALA-4799] Long running metadata load for large tables blocks queries/loading all other tables for long time

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: Impala 2.7.0
• Component/s: Catalog
• Environment: Version: 2.7.0-cdh5.9.0; OS: Centos.6.6
• Assignee: Dimitris Tsirogiannis
• Reporter: Antoni
• Created: 23/Jan/17 13:00
• Updated: 12/Sep/17 19:17
• Resolved: 12/Sep/17 19:17

Description

If you have some big tables history.big_table_with_many_partitions and do a refresh it may take a long time.
But this seems to block loading metadata for all other tables. That shouldn't be the case.
(I am guess it's something to do with ensuring correct version of catalog topic update ? )

For example the follow sequence can be used to duplicate the issue:

impala-shell -k -i hdp-dn01 -q "invalidate metadata history.empty_table;" # invalidate metadata to force PriorotizeLoad of table next time it's queried
impala-shell -k -i hdp-dn02 -q "refresh history.big_table_with_many_partitions;" # from different node
impala-shell -k -i hdp-dn01 -q "select count(1) from history.empty_table;" # this always finishes a second or two after refresh history.big... finishes - which maybe minutes - even if the table is empty. - this query is started a second or so after the previous one and it always finishes after the previous one.

See attached logs (see /tmp/catalogd-invalidate-metadata-example-logs.log) : in that case : empty_table is "history.sa_issue_rating" and big_table_with_many_partitions is "history.bundle"
You may notice in teh logs multiple Publish updates after the refresh (ResetMetadata for history bundle (big_table_with_many_partitions) ) finishes

Absolutely the same thing happens with drop table statement:

impala-shell -k -i hdp-dn02 -q "refresh history.big_table_with_many_partitions;" # from different node
impala-shell -k -i hdp-dn01 -q "drop table default.test_table;" # this always finishes a second or two after refresh history.big... finishes - which maybe minutes - even if the table is empty. - this query is started a second or so after the previous one

Please see impalad-drop-test-table-logs.log and /tmp/catalogd-drop-test-table.log for logs about this.

Issue Links

• is duplicated by: IMPALA-5058 Improve concurrency of DDL/DML operations during catalog updates