[HIVE-19008] Improve Spark session id logging

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.0.0-alpha-1
• Component/s: Spark
• Target Version/s: 3.1.0
• Assignee: Sahil Takiar
• Reporter: Sahil Takiar
• Created: 21/Mar/18 17:11
• Updated: 17/Nov/22 08:54
• Resolved: 06/Aug/18 09:34

Description

HoS users have two session ids, one id for the Hive session and another id for the Spark session, both are UUIDs.

I think some improvements could be made here:

The Spark session id could just be a counter that is incremented for each new Spark session within a Hive session. Each Spark session is still globally identifiable by its associated Hive session id + its own counter. This may make more sense since the Hive session - Spark session has a 1-to-many relationship, as in a single Hive session can contain multiple Spark sessions, and each Spark session must belong to a Hive session.

Furthermore, we should include both the Hive session id and Spark session id in the console logs + the Spark Web UI.