[FLINK-23989] Flink SQL visibility

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Runtime / Queryable State
• Assignee: Unassigned
• Reporter: Zhihao Wang
• Created: 26/Aug/21 08:31
• Updated: 26/Aug/21 08:46

Description

It’s desired to inspect into the internal states generated by SQL especially for debug purpose. We propose to add a new feature in Flink UI console, when the users click on a Vertex of the Job DAG. We shall list all states the vertex has in a separate panel (let’s call states panel). On this panel users can query the states with some keys. The returned value share be a human readable string instead of opaque binary.

 

Particularly, we need to expose the states as queryable. But currently the user experience of queryable states is cumbersome. Only the serialized value is returned to client and users need to handle deserialization by themselves. What’s worse, the client need to construct the serializer and type manually. To improve this situation. We propose:

1. Have a new API to find all queryable states associated to a job vertex. This can be done to check against the KvStateLocationRegistry, which store the mapping between JobVertexId and states.
2. Have a new API to allow users get the types of queryable states: For a register name (String), Queryable Server will return the type of key and value (LogicalType).
3. To generate human readable string automatically with API in step 2, we can 1) generate TypeSerializer from the LogicalType, so as to handle Serde automatically. 2) to convert internal data structures to external data structures to generate printable string. (with converters DataStructureConverter )

With all these steps and some modifications to Web UI/ Rest, we can enable users to query SQL internal states.