[IMPALA-7959] Trace and visualize concurrent Catalog operations

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Catalog
• Labels: supportability
• Epic Link: SupportabilityRoadMap
• Epic Color: ghx-label-5
• Assignee: Unassigned
• Reporter: Bharath Vissapragada
• Created: 11/Dec/18 21:28
• Updated: 15/Jul/21 00:07

Description

The idea here is to leverage chromium tracing (about:trace) for visualizing Catalog events [1]. We could periodically dump the trace events in the JSON format [2] that can be plotted using the visualizer tool [3].

This helps us get a sense of concurrent catalog operations using a timeline view of chromium tracing.Useful for debugging contention and hangs of concurrent operations.

This idea was borrowed from the Apache Kudu project [4]

[1] https://github.com/catapult-project/catapult/tree/master/tracing
[2] https://docs.google.com/document/d/1CvAClvFfyA5R-PhYUmn5OOQtYMH4h6I0nSsKchNAySU/preview
[3] https://www.chromium.org/developers/how-tos/trace-event-profiling-tool/using-frameviewer
[4] https://kudu.apache.org/docs/troubleshooting.html#kudu_tracing

Issue Links

• is related to: IMPALA-9118 Add debug page for in-flight DDLs in catalogd
• is related to: IMPALA-6663 Expose current DDL metrics (grouped by type) in the Catalog web UI
• requires: IMPALA-337 Update Impala debug webserver to accept HTTP POST requests