[SPARK-49409] CONNECT_SESSION_PLAN_CACHE_SIZE is too small for certain programming patterns

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: Connect
• Labels: pull-request-available
• Assignee: Changgyoo Park
• Reporter: Changgyoo Park
• Created: 27/Aug/24 14:54
• Updated: 30/Aug/24 12:04
• Resolved: 30/Aug/24 12:04

Description

Example:

 

```

df_1 = df_a.filter(col('X').isNotNull())

df_2 = df_b.filter(col('SAFE_SU_Conv').isNotNull())

....

df_x = ...

for _ in range(0, 5):

    df_x = df_x.select(...)

...

df_3 = df_1.join(df_2, ...)

```

=> df_x completely invalidates all the cached entries.

Issue Links

• links to: GitHub Pull Request #47937