[SPARK-20280] SharedInMemoryCache Weigher integer overflow

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.1.0, 2.2.0
• Fix Version/s: 2.1.1, 2.2.0
• Component/s: Spark Core
• Assignee: Bogdan Raducanu
• Reporter: Bogdan Raducanu
• Created: 10/Apr/17 11:49
• Updated: 10/Apr/17 19:57
• Resolved: 10/Apr/17 19:57

Description

in FileStatusCache.scala:

    .weigher(new Weigher[(ClientId, Path), Array[FileStatus]] {
      override def weigh(key: (ClientId, Path), value: Array[FileStatus]): Int = {
        (SizeEstimator.estimate(key) + SizeEstimator.estimate(value)).toInt
      }})

Weigher.weigh returns Int but the size of an Array[FileStatus] could be bigger than Int.maxValue. Then, a negative value is returned, leading to this exception:

* [info]   java.lang.IllegalStateException: Weights must be non-negative
* [info]   at com.google.common.base.Preconditions.checkState(Preconditions.java:149)
* [info]   at com.google.common.cache.LocalCache$Segment.setValue(LocalCache.java:2223)
* [info]   at com.google.common.cache.LocalCache$Segment.put(LocalCache.java:2944)
* [info]   at com.google.common.cache.LocalCache.put(LocalCache.java:4212)
* [info]   at com.google.common.cache.LocalCache$LocalManualCache.put(LocalCache.java:4804)
* [info]   at org.apache.spark.sql.execution.datasources.SharedInMemoryCache$$anon$3.putLeafFiles(FileStatusCache.scala:131)

Issue Links

• links to: [Github] Pull Request #17591 (bogdanrdc)