Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-32373

Spark Standalone - RetryingBlockFetcher tries to get block from worker even 10mins after it was de-registered from spark cluster

    XMLWordPrintableJSON

    Details

      Description

      Using spark standalone in 2.4.6 with spot ec2 instances, the .242 IP instance was terminated at 12:00:11pm, before then it appeared registered in Spark UI as ALIVE for few hours, it then appeared in Spark UI as DEAD until 12:16pm, then it disappeared from Spark UI completely. An app that started at 11:24am had below error. As you can see in below app log from another worker it is still trying to get shuffle block from .242 IP at 12:10pm (10mins after the worker was removed from the spark cluster). I would expect that once within 2mins of the worker being removed from the cluster that it would stop retrying

       

      
      2020-07-20 12:10:02,702 [Block Fetch Retry-9-3] ERROR org.apache.spark.network.shuffle.RetryingBlockFetcher - Exception while beginning fetch of 1 outstanding blocks (after 3 retries)
      java.io.IOException: Connecting to /redact.242:7337 timed out (120000 ms)
      	at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:243)
      	at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:187)
      	at org.apache.spark.network.shuffle.ExternalShuffleClient.lambda$fetchBlocks$0(ExternalShuffleClient.java:100)
      	at org.apache.spark.network.shuffle.RetryingBlockFetcher.fetchAllOutstanding(RetryingBlockFetcher.java:141)
      	at org.apache.spark.network.shuffle.RetryingBlockFetcher.lambda$initiateRetry$0(RetryingBlockFetcher.java:169)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
      	at java.lang.Thread.run(Thread.java:748)
      2020-07-20 12:07:57,700 [Block Fetch Retry-9-2] ERROR org.apache.spark.network.shuffle.RetryingBlockFetcher - Exception while beginning fetch of 1 outstanding blocks (after 2 retries)
      java.io.IOException: Connecting to /redact.242:7337 timed out (120000 ms)
      	at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:243)
      	at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:187)
      	at org.apache.spark.network.shuffle.ExternalShuffleClient.lambda$fetchBlocks$0(ExternalShuffleClient.java:100)
      	at org.apache.spark.network.shuffle.RetryingBlockFetcher.fetchAllOutstanding(RetryingBlockFetcher.java:141)
      	at org.apache.spark.network.shuffle.RetryingBlockFetcher.lambda$initiateRetry$0(RetryingBlockFetcher.java:169)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
      	at java.lang.Thread.run(Thread.java:748)
      2020-07-20 12:05:52,697 [Block Fetch Retry-9-1] ERROR org.apache.spark.network.shuffle.RetryingBlockFetcher - Exception while beginning fetch of 1 outstanding blocks (after 1 retries)
      java.io.IOException: Connecting to /redact.242:7337 timed out (120000 ms)
      	at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:243)
      	at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:187)
      	at org.apache.spark.network.shuffle.ExternalShuffleClient.lambda$fetchBlocks$0(ExternalShuffleClient.java:100)
      	at org.apache.spark.network.shuffle.RetryingBlockFetcher.fetchAllOutstanding(RetryingBlockFetcher.java:141)
      	at org.apache.spark.network.shuffle.RetryingBlockFetcher.lambda$initiateRetry$0(RetryingBlockFetcher.java:169)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
      	at java.lang.Thread.run(Thread.java:748)
      2020-07-20 12:03:47,689 [Executor task launch worker for task 217] ERROR org.apache.spark.network.shuffle.RetryingBlockFetcher - Exception while beginning fetch of 1 outstanding blocks 
      java.io.IOException: Connecting to /redact.242:7337 timed out (120000 ms)
      	at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:243)
      	at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:187)
      	at org.apache.spark.network.shuffle.ExternalShuffleClient.lambda$fetchBlocks$0(ExternalShuffleClient.java:100)
      	at org.apache.spark.network.shuffle.RetryingBlockFetcher.fetchAllOutstanding(RetryingBlockFetcher.java:141)
      	at org.apache.spark.network.shuffle.RetryingBlockFetcher.start(RetryingBlockFetcher.java:121)
      	at org.apache.spark.network.shuffle.ExternalShuffleClient.fetchBlocks(ExternalShuffleClient.java:109)
      	at org.apache.spark.storage.ShuffleBlockFetcherIterator.sendRequest(ShuffleBlockFetcherIterator.scala:260)
      	at org.apache.spark.storage.ShuffleBlockFetcherIterator.org$apache$spark$storage$ShuffleBlockFetcherIterator$$send$1(ShuffleBlockFetcherIterator.scala:531)
      	at org.apache.spark.storage.ShuffleBlockFetcherIterator.fetchUpToMaxBytes(ShuffleBlockFetcherIterator.scala:526)
      	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:489)
      	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:64)
      	at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:435)
      	at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:441)
      	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
      	at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:31)
      	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
      	at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:199)
      	at org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:102)
      	at org.apache.spark.rdd.ShuffledRDD.compute(ShuffledRDD.scala:105)
      	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346)
      	at org.apache.spark.rdd.RDD.iterator(RDD.scala:310)
      	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
      	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346)
      	at org.apache.spark.rdd.RDD.iterator(RDD.scala:310)
      	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
      	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346)
      	at org.apache.spark.rdd.RDD.iterator(RDD.scala:310)
      	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
      	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346)
      	at org.apache.spark.rdd.RDD.iterator(RDD.scala:310)
      	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
      	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346)
      	at org.apache.spark.rdd.RDD.iterator(RDD.scala:310)
      	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
      	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346)
      	at org.apache.spark.rdd.RDD.iterator(RDD.scala:310)
      	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
      	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:55)
      	at org.apache.spark.scheduler.Task.run(Task.scala:123)
      	at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
      	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
      	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              toopt4 t oo
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: