Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-21794

exception about reading task serial data(broadcast) value when the storage memory is not enough to unroll

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Duplicate
    • 2.0.1, 2.1.0
    • None
    • Spark Core
    • None

    Description

      ```
      17/08/09 19:27:43 ERROR Utils: Exception encountered
      java.util.NoSuchElementException
      at org.apache.spark.util.collection.PrimitiveVector$$anon$1.next(PrimitiveVector.scala:58)
      at org.apache.spark.storage.memory.PartiallyUnrolledIterator.next(MemoryStore.scala:697)
      at org.apache.spark.util.CompletionIterator.next(CompletionIterator.scala:30)
      at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1$$anonfun$2.apply(TorrentBroadcast.scala:178)
      at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1$$anonfun$2.apply(TorrentBroadcast.scala:178)
      at scala.Option.map(Option.scala:146)
      at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1.apply(TorrentBroadcast.scala:178)
      at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1276)
      at org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:174)
      at org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(TorrentBroadcast.scala:65)
      at org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.scala:65)
      at org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:89)
      at org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70)
      at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:72)
      at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47)
      at org.apache.spark.scheduler.Task.run(Task.scala:86)
      at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:745)
      17/08/09 19:27:43 INFO UnifiedMemoryManager: Will not store broadcast_5 as the required space (1048576 bytes) exceeds our memory limit (878230 bytes)
      17/08/09 19:27:43 WARN MemoryStore: Failed to reserve initial memory threshold of 1024.0 KB for computing block broadcast_5 in memory.
      17/08/09 19:27:43 WARN MemoryStore: Not enough space to cache broadcast_5 in memory! (computed 384.0 B so far)
      17/08/09 19:27:43 INFO MemoryStore: Memory use = 857.6 KB (blocks) + 0.0 B (scratch space shared across 0 tasks(s)) = 857.6 KB. Storage limit = 857.6 KB.
      17/08/09 19:27:43 ERROR Utils: Exception encountered
      java.util.NoSuchElementException
      at org.apache.spark.util.collection.PrimitiveVector$$anon$1.next(PrimitiveVector.scala:58)
      at org.apache.spark.storage.memory.PartiallyUnrolledIterator.next(MemoryStore.scala:697)
      at org.apache.spark.util.CompletionIterator.next(CompletionIterator.scala:30)
      at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1$$anonfun$2.apply(TorrentBroadcast.scala:178)
      at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1$$anonfun$2.apply(TorrentBroadcast.scala:178)
      at scala.Option.map(Option.scala:146)
      at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1.apply(TorrentBroadcast.scala:178)
      at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1276)
      at org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:174)
      at org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(TorrentBroadcast.scala:65)
      at org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.scala:65)
      at org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:89)
      at org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70)
      at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:72)
      at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47)
      at org.apache.spark.scheduler.Task.run(Task.scala:86)
      at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:745)
      ```

      This exception is caused in `MemoryStore.putIteratorAsValues()`. When `keepUnrolling` is false in the first time, the `vector: SizeTrackingVector` is not null and is empty. So when call the iterator method of `vector`, it throws this exception.

      Attachments

        1. error stack.png
          71 kB
          roncenzhao

        Issue Links

          Activity

            People

              Unassigned Unassigned
              roncenzhao roncenzhao
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: