[IMPALA-9791] Support validWriteIdList in getPartialCatalogObject

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Impala 4.0.0
• Labels: impala-acid
• Epic Color: ghx-label-7
• Assignee: Vihang Karajgaonkar
• Reporter: Vihang Karajgaonkar
• Created: 28/May/20 19:02
• Updated: 10/Jul/20 19:37
• Resolved: 10/Jul/20 19:37

Description

When transactional tables are being queried, the coordinator (or any other Catalog client) can optionally provide a ValidWriteIdList of the table. In such case, catalog can return the metadata which is consistent with the given ValidWriteIdList. There are the following 3 possibilities:

1. Client provided ValidWriteIdList is more recent.
In this case, catalog should reload the table then send the metadata consistent with the provided writeIdList.

2. Client ValidWriteIdList is same.
Catalog can return the cached metadata directly.

3. ClientValidWriteIdList is stale with respect to the one in catalog.
In this case, catalog can attempt to return metadata which is consistent with respect to client's view of the writeIdList and return accordingly. Note that in case 1, it is possible that after reload, catalog moves ahead of the client's writeIdList and hence this becomes a sub-case of 1.

Having such an enhancement to the API can help support consistent read support for ACID tables (see IMPALA-8788)

Issue Links

• relates to: IMPALA-8788 Read globally consistent snapshot of ACID tables
• relates to: IMPALA-9841 Table level writeIdList represent the filemetadata snapshot