[IMPALA-10557] Allow Impala to use Kudu's multi-row transaction implementation

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Impala 4.1.0
• Component/s: Clients
• Epic Color: ghx-label-12
• Assignee: Wenzhe Zhou
• Reporter: Andrew Wong
• Created: 02/Mar/21 01:35
• Updated: 06/Jul/21 01:33
• Resolved: 06/Jul/21 01:33

Description

Once Kudu's support for multi-row transactions materializes, it'd be great if Impala could expose a query option or session-level flag to indicate that Impala should run queries that insert into Kudu in the context of a single transaction.

When the query option is set, similar to how transactions are opened for Hive ACID tables, a new Kudu transaction should be started by Impala's Frontend, and the resulting KuduTransaction object should be serialized and passed as context to the Impala executors, which should open their Kudu sessions using the KuduTransaction::CreateSession() API instead of the usual KuduClient::NewSession().

Errors returned in the session should be propagated to the Impala Frontend before committing – it seems reasonable to expect that any session errors should result in Impala aborting the transaction.

The KuduTransaction handle in the frontend has a heartbeater built in, so unlike Hive ACID transactions, which require manual tracking of heartbeating, no additional manual requests are needed to maintain the transaction.

Kudu's Java client API is defined in this patch, and C++ client API is defined in this patch. The implementation isn't fully operational quite yet – once this patch to automatically register participants lands, Impala developers should at least be able to start adding basic testing non-overlapping cases (partition locking is still in progress).

Issue Links

• depends upon: KUDU-2612 Implement multi-row transactions