[SPARK-44631] Remove session-based directory when the isolated session cache is evicted

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.5.0
• Fix Version/s: 3.5.0, 4.0.0
• Component/s: Connect
• Assignee: Hyukjin Kwon
• Reporter: Hyukjin Kwon
• Created: 02/Aug/23 04:13
• Updated: 02/Aug/23 07:47
• Resolved: 02/Aug/23 07:47

Description

SPARK-44078 added the cache for isolated sessions, and SPARK-44348 added the session-based directory for isolation.

 

When the isolated session cache is evicted, we should remove the session-based directory so it doesn't fail when the same session is used, see also https://github.com/apache/spark/pull/41625#discussion_r1251427466