Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-26726

Synchronize the amount of memory used by the broadcast variable to the UI display

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.4.0
    • Fix Version/s: 2.3.3, 2.4.1, 3.0.0
    • Component/s: Spark Core
    • Labels:
      None

      Description

      The amount of memory used by the broadcast variable is not synchronized to the UI display,

      spark-sql>  select /+ broadcast(a)/ a.id,b.id from a join b on a.id = b.id;

      View the app's driver log:

      2019-01-25 16:45:23,726 INFO org.apache.spark.storage.BlockManagerInfo: Added broadcast_4_piece0 in memory on 10.43.xx.xx:33907 (size: 6.6 KB, free: 2.5 GB)
      2019-01-25 16:45:23,727 INFO org.apache.spark.storage.BlockManagerInfo: Added broadcast_4_piece0 in memory on 10.43.xx.xx:38399 (size: 6.6 KB, free: 2.5 GB)
      2019-01-25 16:45:23,745 INFO org.apache.spark.storage.BlockManagerInfo: Added broadcast_3_piece0 in memory on 10.43.xx.xx:33907 (size: 32.1 KB, free: 2.5 GB)
      2019-01-25 16:45:23,749 INFO org.apache.spark.storage.BlockManagerInfo: Added broadcast_3_piece0 in memory on 10.43.xx.xx:38399 (size: 32.1 KB, free: 2.5 GB)
      2019-01-25 16:45:23,838 INFO org.apache.spark.storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on 10.43.xx.xx:38399 (size: 147.0 B, free: 2.5 GB)
      2019-01-25 16:45:23,840 INFO org.apache.spark.storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on 10.43.xx.xx:33907 (size: 147.0 B, free: 2.5 GB)

       

      Web UI does not have the use of memory,

      Executor ID Address Status RDD Blocks Storage Memory Disk Used Cores Active Tasks Failed Tasks Complete Tasks Total Tasks Task Time (GC Time) Input Shuffle Read Shuffle Write Logs Thread Dump
      0 xxx:38399 Active 0 0.0 B / 2.7 GB 0.0 B 1 0 0 2 2 4 s (0.4 s) 8 B 0.0 B 0.0 B    
      driver xxx:47936 Active 0 0.0 B / 384.1 MB 0.0 B 0 0 0 0 0 0.0 ms (0.0 ms) 0.0 B 0.0 B 0.0 B    
      1 xxx:47414 Active 0 0.0 B / 2.7 GB 0.0 B 1 0 0 0 0 0.0 ms (0.0 ms) 0.0 B 0.0 B 0.0 B    
      2 xxx:33907 Active 0 0.0 B / 2.7 GB 0.0 B 1 0 0 2 2 4 s (0.2 s) 4 B 0.0 B 0.0 B    

       

       

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                hantiantian hantiantian
                Reporter:
                hantiantian hantiantian
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: