Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Not A Problem
-
3.4.0, 3.5.0, 4.0.0
-
None
-
None
Description
Please apply this patch(SPARK-49091.patch
) to reproduce this issue. This issue may cause driver memory leak.
This issue was introduced by SPARK-41914.
Before SPARK-41914:
[info] BroadcastCleanerSuite: 10:30:16.228 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0, broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3 entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0, broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3 entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0, broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3 entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0, broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3 entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0, broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3 entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0, broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3 entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0, broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3 entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0, broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3 entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0, broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3 entries size: 0, names: [info] - Test broadcast cleaner (1 minute, 4 seconds) 10:31:21.552 WARN org.apache.spark.sql.BroadcastCleanerSuite:
After SPARK-41914:
entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0, broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3 entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0, broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3 entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0, broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3 entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0, broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3 entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0, broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3 entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0, broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3 entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0, broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3 entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0, broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3 entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0, broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3 entries size: 2, names: broadcast_1_piece0, broadcast_1 entries size: 2, names: broadcast_1_piece0, broadcast_1 entries size: 2, names: broadcast_1_piece0, broadcast_1 entries size: 2, names: broadcast_1_piece0, broadcast_1 entries size: 2, names: broadcast_1_piece0, broadcast_1 entries size: 2, names: broadcast_1_piece0, broadcast_1 entries size: 2, names: broadcast_1_piece0, broadcast_1 entries size: 2, names: broadcast_1_piece0, broadcast_1 entries size: 2, names: broadcast_1_piece0, broadcast_1 entries size: 2, names: broadcast_1_piece0, broadcast_1 entries size: 2, names: broadcast_1_piece0, broadcast_1 entries size: 2, names: broadcast_1_piece0, broadcast_1 entries size: 2, names: broadcast_1_piece0, broadcast_1 entries size: 2, names: broadcast_1_piece0, broadcast_1 entries size: 2, names: broadcast_1_piece0, broadcast_1 entries size: 2, names: broadcast_1_piece0, broadcast_1 entries size: 2, names: broadcast_1_piece0, broadcast_1 entries size: 2, names: broadcast_1_piece0, broadcast_1 entries size: 2, names: broadcast_1_piece0, broadcast_1 entries size: 2, names: broadcast_1_piece0, broadcast_1 entries size: 2, names: broadcast_1_piece0, broadcast_1 entries size: 2, names: broadcast_1_piece0, broadcast_1 entries size: 2, names: broadcast_1_piece0, broadcast_1 entries size: 2, names: broadcast_1_piece0, broadcast_1 entries size: 2, names: broadcast_1_piece0, broadcast_1 entries size: 2, names: broadcast_1_piece0, broadcast_1 entries size: 2, names: broadcast_1_piece0, broadcast_1 entries size: 2, names: broadcast_1_piece0, broadcast_1 entries size: 2, names: broadcast_1_piece0, broadcast_1 entries size: 2, names: broadcast_1_piece0, broadcast_1 entries size: 2, names: broadcast_1_piece0, broadcast_1 entries size: 2, names: broadcast_1_piece0, broadcast_1 entries size: 2, names: broadcast_1_piece0, broadcast_1 entries size: 2, names: broadcast_1_piece0, broadcast_1 entries size: 2, names: broadcast_1_piece0, broadcast_1 entries size: 2, names: broadcast_1_piece0, broadcast_1 entries size: 2, names: broadcast_1_piece0, broadcast_1 entries size: 2, names: broadcast_1_piece0, broadcast_1 entries size: 2, names: broadcast_1_piece0, broadcast_1 entries size: 2, names: broadcast_1_piece0, broadcast_1 [info] *** Test still running after 4 minutes, 58 seconds: suite name: BroadcastCleanerSuite, test name: Test broadcast cleaner. ...
Attachments
Attachments
Issue Links
- is caused by
-
-
- Resolved
-