[IMPALA-8473] Refactor lineage publication mechanism to allow for different consumers

• Type: Sub-task
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: Impala 3.3.0
• Component/s: Backend, Frontend
• Epic Color: ghx-label-5
• Assignee: radford nguyen
• Reporter: radford nguyen
• Created: 30/Apr/19 18:58
• Updated: 29/May/19 21:54
• Resolved: 29/May/19 21:54

Description

Impetus for this change is to allow lineage to be consumed by Atlas via Kafka.

Design Proposal

Implement a plugin approach (similar to authorization_provider) for consuming query event hooks, where downstream users can provide their own hook implementations as runtime dependencies.

Keep but deprecate existing lineage event file writing.

[e-mail] has provided a fe patch (attached) with suggested mechanism for allowing multiple hooks to be registered with the fe.  Hooks would be invoked from the be at appropriate places, e.g. https://github.com/apache/impala/blob/c1b0a073938c144e9bf33901bd4df6dcda0f09ec/be/src/service/impala-server.cc#L466.  The hooks should all be executed asynchronously, so the current thinking is that this execution should happen in the fe, since the be does not know about what hooks are registered.  IOW, the ImpalaPostExecHookFactory.executeHooks method (see patch) should probably make use of a thread-pool executor service (or something similar) in order to execute all hooks in parallel and in a non-blocking manner, returning to the be asap.

 

Code Review

https://gerrit.cloudera.org/#/c/13352/

Issue Links

• causes: IMPALA-8571 Make query-hook-execution more robust and observable
• causes: IMPALA-8574 Review test coverage for query hook feature
• causes: IMPALA-8576 Pass lineage object instead of string to query hook
• causes: IMPALA-8572 Move query hook execution to before query unregistration
• causes: IMPALA-8573 Implement timeout for query hook execution
• causes: IMPALA-8589 Fix flaky query event hook tests