[SPARK-47660] Memory leak in StructType column's .dropFields method

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.5.1
• Component/s: Java API, SQL
• Assignee: Unassigned
• Reporter: Eylon Ronen
• Created: 31/Mar/24 10:10
• Updated: 01/Apr/24 07:26

Description

We noticed that when we're using the .dropFields method, for example by calling

df.withColum(column, col(column).dropFields("fieldToDrop")) the heap "explodes". we noticed it when our integration tests, for which 512MB heap is always enough started crashing with OutOfMemoryException (even when we increased the heap size to 8GB) and after a lot of trial and error we found that when we remove the .dropFields method the heap goes back down