[FLINK-25318] Improvement of scheduler and execution for Flink OLAP

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.14.0, 1.12.5, 1.13.3
• Component/s: Runtime / Coordination, Runtime / Network
• Labels: Umbrella
• Assignee: Unassigned
• Reporter: Fang Yong
• Created: 15/Dec/21 07:30
• Updated: 18/Dec/23 06:48

Description

We use flink to perform OLAP queries. We launch flink session cluster, submit batch jobs to the cluster as OLAP queries, and fetch the jobs' results. OLAP jobs are generally small queries which will finish at the seconds or milliseconds, and users always submit multiple jobs to the session cluster concurrently. We found the qps and latency of jobs will be greatly affected when there're tens jobs are running, even when there's little data in each query. We will give the result of benchmark for the latest version later.
After discussed with xtsong, and thanks for his advice, we create this issue to trace and manager Flink OLAP related improvements. More users and developers are welcome and feel free to create Flink OLAP related subtasks here, thanks

Issue Links

• is a parent of: FLINK-25027 Allow GC of a finished job's JobMaster before the slot timeout is reached
• is a parent of: FLINK-33852 FLIP-403 High Availability Services for OLAP Scenarios
• relates to: FLINK-32898 Improvement of usability for Flink OLAP
• relates to: FLINK-33683 FLIP-407 Improve Flink Client performance in interactive scenarios
• relates to: FLINK-25356 Add benchmarks for performance in OLAP scenarios

Sub-Tasks

• System classloader memory leak after loading too many codegen classes.: Open
• Improvement of reuse segments for join/agg/sort operators in TaskManager for flink olap queries: Open
• Manage and share gateways of taskmanagers between jobs in session cluster: Open
• HiveSourceFileEnumerator should fetch splits asynchronously: Open
• Improvement of connection from TM to JM in session cluster: Open
• Too many JM logs in flink session cluster for olap queries: Open
• ExecutionGraphInfoStore in session cluster should split failed and successful jobs: Open
• Improve cache hit rate of generated class: Open
• Add WebSocket in Dispatcher to support olap query submission and push results in session cluster: Open
• Improve to reuse threads in TaskManager for different tasks between jobs: Open
• Support customized listener during task manager startup: Open
• Support concurrency control when submitting OLAP jobs to Dispatcher: Open
• Use default classloader in TaskManager when there are no user jars for job: Open