[IMPALA-8406] Failed REFRESH can partially modify table without bumping version number

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 3.2.0
• Fix Version/s: Impala 4.0.0
• Component/s: Catalog
• Epic Color: ghx-label-1
• Assignee: Quanlong Huang
• Reporter: Todd Lipcon
• Created: 10/Apr/19 23:58
• Updated: 21/Dec/20 18:13
• Resolved: 21/Dec/20 18:13

Description

Currently, various incremental operations in the catalogd modify Table objects in place, including REFRESH, which modifies each partition. In this case, if one partition fails to refresh (eg due to incorrect partitions or some other file access problem), other partitions can still be modified, either because they were modified first (in a non-parallel operation) or modified in parallel (for REFRESH).

In this case, the REFRESH operation will throw an Exception back to the user, but in fact it has modified the catalog entry. The version number, however, is not bumped, which breaks some invariants of the catalog that an object doesn't change without changing version numbers.

This also produces some unexpected behavior such as:

• SHOW FILES IN t;
• REFRESH t; – gets a failure
• SHOW FILES in t; – see the same result as originally
• ALTER TABLE t SET UNCACHED; – bumps the version number due to unrelated operation
• SHOW FILES IN t; – the set of files has changed due to the earlier partially-complete REFRESH