[SPARK-23394] Storage info's Cached Partitions doesn't consider the replications (but sc.getRDDStorageInfo does)

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3.0
• Fix Version/s: 2.3.0, 2.4.0
• Component/s: Spark Core
• Assignee: Attila Zsolt Piros
• Reporter: Attila Zsolt Piros
• Created: 12/Feb/18 10:23
• Updated: 23/Feb/18 06:33
• Resolved: 14/Feb/18 14:46

Description

Start spark as:

$ bin/spark-shell --master local-cluster[2,1,1024]

scala> import org.apache.spark.storage.StorageLevel._
import org.apache.spark.storage.StorageLevel._

scala> sc.parallelize((1 to 100), 10).persist(MEMORY_AND_DISK_2).count
res0: Long = 100                                                                

scala> sc.getRDDStorageInfo(0).numCachedPartitions
res1: Int = 20

Cached Partitions

On the UI at the Storage tab Cached Partitions is 10:

.

Full tab

Moreover the replicated partitions was also listed on the old 2.2.1 like:

But now it is like:

Issue Links

• links to: [Github] Pull Request #20589 (attilapiros)