[HIVE-27126] Display Yarn resources availability in real time along with Tez job progress

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: HiveServer2
• Assignee: Unassigned
• Reporter: Taraka Rama Rao Lethavadla
• Created: 08/Mar/23 07:22
• Updated: 08/Mar/23 18:02

Description

Right we don't know if the queue used in Hive query execution is having enough resources or not. We are already displaying tez job details like number of task per each vertex and how are they progressing. 

If the resources available are not good enough to execute the query in parallel or query that use to take shorter time is taking time, then we can use this new detail to understand that queue is busy and resources are not enough

 

 

 

----------------------------------------------------------------------------------------------
        VERTICES      MODE        STATUS  TOTAL  COMPLETED  RUNNING  PENDING  FAILED  KILLED
----------------------------------------------------------------------------------------------
Map 1            container       RUNNING    230          0        10      230     0       1
Reducer 2        container        INITED      1          0        0        1       0       0
----------------------------------------------------------------------------------------------
VERTICES: 00/02  [>>--------------------------] 0%    ELAPSED TIME: 58.33 s
----------------------------------------------------------------------------------------------

So it would be great to display current resource availability in that specific queue along with above info in real time would help understand why a query can be slow

 

Issue Links

• is related to: HIVE-25666 Realtime memory usage in beeline progress