[SPARK-26726] Synchronize the amount of memory used by the broadcast variable to the UI display

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 2.3.3, 2.4.1, 3.0.0
• Component/s: Spark Core
• Assignee: hantiantian
• Reporter: hantiantian
• Created: 25/Jan/19 08:51
• Updated: 01/Feb/19 01:21
• Resolved: 31/Jan/19 17:22

Description

The amount of memory used by the broadcast variable is not synchronized to the UI display，

spark-sql>  select /+ broadcast(a)/ a.id,b.id from a join b on a.id = b.id;

View the app's driver log：

2019-01-25 16:45:23,726 INFO org.apache.spark.storage.BlockManagerInfo: Added broadcast_4_piece0 in memory on 10.43.xx.xx:33907 (size: 6.6 KB, free: 2.5 GB)
2019-01-25 16:45:23,727 INFO org.apache.spark.storage.BlockManagerInfo: Added broadcast_4_piece0 in memory on 10.43.xx.xx:38399 (size: 6.6 KB, free: 2.5 GB)
2019-01-25 16:45:23,745 INFO org.apache.spark.storage.BlockManagerInfo: Added broadcast_3_piece0 in memory on 10.43.xx.xx:33907 (size: 32.1 KB, free: 2.5 GB)
2019-01-25 16:45:23,749 INFO org.apache.spark.storage.BlockManagerInfo: Added broadcast_3_piece0 in memory on 10.43.xx.xx:38399 (size: 32.1 KB, free: 2.5 GB)
2019-01-25 16:45:23,838 INFO org.apache.spark.storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on 10.43.xx.xx:38399 (size: 147.0 B, free: 2.5 GB)
2019-01-25 16:45:23,840 INFO org.apache.spark.storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on 10.43.xx.xx:33907 (size: 147.0 B, free: 2.5 GB)

 

Web UI does not have the use of memory，

Executor ID	Address	Status	RDD Blocks	Storage Memory	Disk Used	Cores	Active Tasks	Failed Tasks	Complete Tasks	Total Tasks	Task Time (GC Time)	Input	Shuffle Read	Shuffle Write	Logs	Thread Dump
0	xxx:38399	Active	0	0.0 B / 2.7 GB	0.0 B	1	0	0	2	2	4 s (0.4 s)	8 B	0.0 B	0.0 B
driver	xxx:47936	Active	0	0.0 B / 384.1 MB	0.0 B	0	0	0	0	0	0.0 ms (0.0 ms)	0.0 B	0.0 B	0.0 B
1	xxx:47414	Active	0	0.0 B / 2.7 GB	0.0 B	1	0	0	0	0	0.0 ms (0.0 ms)	0.0 B	0.0 B	0.0 B
2	xxx:33907	Active	0	0.0 B / 2.7 GB	0.0 B	1	0	0	2	2	4 s (0.2 s)	4 B	0.0 B	0.0 B

 

 

Issue Links

• links to: GitHub Pull Request #23649