Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-20280

SharedInMemoryCache Weigher integer overflow

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.1.0, 2.2.0
    • Fix Version/s: 2.1.1, 2.2.0
    • Component/s: Spark Core
    • Labels:
      None

      Description

      in FileStatusCache.scala:

          .weigher(new Weigher[(ClientId, Path), Array[FileStatus]] {
            override def weigh(key: (ClientId, Path), value: Array[FileStatus]): Int = {
              (SizeEstimator.estimate(key) + SizeEstimator.estimate(value)).toInt
            }})
      

      Weigher.weigh returns Int but the size of an Array[FileStatus] could be bigger than Int.maxValue. Then, a negative value is returned, leading to this exception:

      * [info]   java.lang.IllegalStateException: Weights must be non-negative
      * [info]   at com.google.common.base.Preconditions.checkState(Preconditions.java:149)
      * [info]   at com.google.common.cache.LocalCache$Segment.setValue(LocalCache.java:2223)
      * [info]   at com.google.common.cache.LocalCache$Segment.put(LocalCache.java:2944)
      * [info]   at com.google.common.cache.LocalCache.put(LocalCache.java:4212)
      * [info]   at com.google.common.cache.LocalCache$LocalManualCache.put(LocalCache.java:4804)
      * [info]   at org.apache.spark.sql.execution.datasources.SharedInMemoryCache$$anon$3.putLeafFiles(FileStatusCache.scala:131)
      
      

        Attachments

          Activity

            People

            • Assignee:
              bograd Bogdan Raducanu
              Reporter:
              bograd Bogdan Raducanu
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: