[SPARK-20386] The log info "Added %s in memory on %s (size: %s, free: %s)" in function "org.apache.spark.storage.BlockManagerInfo.updateBlockInfo" is not accurate if the block exists on the slave already

• Type: Bug
• Status: Resolved
• Priority: Trivial
• Resolution: Fixed
• Affects Version/s: 2.1.0
• Fix Version/s: 2.2.0
• Component/s: Spark Core
• Assignee: eaton
• Reporter: eaton
• Created: 19/Apr/17 08:41
• Updated: 22/Apr/17 11:54
• Resolved: 22/Apr/17 11:30

Description

The log info"Added %s in memory on %s (size: %s, free: %s)" in function "org.apache.spark.storage.BlockManagerInfo.updateBlockInfo" is not accurate if the block exists on the slave already;
the current code is:
if (storageLevel.useMemory)

{ blockStatus = BlockStatus(storageLevel, memSize = memSize, diskSize = 0) _blocks.put(blockId, blockStatus) _remainingMem -= memSize logInfo("Added %s in memory on %s (size: %s, free: %s)".format( blockId, blockManagerId.hostPort, Utils.bytesToString(memSize), Utils.bytesToString(_remainingMem))) }

If the block exists on the slave already, the added memory should be memSize - originalMemSize, the originalMemSize is _blocks.get(blockId).memSize

Issue Links

• links to: [Github] Pull Request #17683 (eatoncys)