Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-23895

Job continues to run even though some tasks have been failed

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Incomplete
    • 1.6.0
    • None
    • Spark Core

    • Ubuntu 14.04.4 LTS

      Spark standalone deployment

    Description

      We are using com.databricks.spark-redshift to write dataframe into Redshift table. Internally, it writes dataframe to S3 first, and then executes query to load data from S3 to the final Redshift table. The problem occurred in the writing S3 phase, one executor was down due to JVM issue but the whole job was considered as success. It continued to run query to load the incomplete data from S3 to Redshift, which resulted in data loss.

      The executor log:

      8/04/01 15:06:25 INFO MemoryStore: Block broadcast_664 stored as values in memory (estimated size 114.6 KB, free 63.3 MB)
      18/04/01 15:06:25 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 11, fetching them
      18/04/01 15:06:25 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@172.19.103.84:43248)
      18/04/01 15:06:25 INFO MapOutputTrackerWorker: Got the output locations
      18/04/01 15:06:25 INFO ShuffleBlockFetcherIterator: Getting 200 non-empty blocks out of 200 blocks
      18/04/01 15:06:25 INFO ShuffleBlockFetcherIterator: Started 15 remote fetches in 4 ms
      18/04/01 15:06:25 INFO DefaultWriterContainer: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
      Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f823e54e000, 65536, 1) failed; error='Cannot allocate memory' (errno=12)

       

      The driver log:

      18/04/01 15:06:25 INFO BlockManagerInfo: Added rdd_743_199 in memory on ip-172-19-107-199.ec2.internal:35094 (size: 752.5 KB, free: 1939.0 MB)
      18/04/01 15:06:25 INFO TaskSetManager: Finished task 199.0 in stage 29.0 (TID 5463) in 597 ms on ip-172-19-107-199.ec2.internal (200/200)
      18/04/01 15:06:25 INFO TaskSchedulerImpl: Removed TaskSet 29.0, whose tasks have all completed, from pool
      18/04/01 15:06:25 INFO DAGScheduler: ShuffleMapStage 29 (mapPartitions at RedshiftWriter.scala:237) finished in 6.194 s
      18/04/01 15:06:25 INFO DAGScheduler: looking for newly runnable stages
      18/04/01 15:06:25 INFO DAGScheduler: running: Set()
      18/04/01 15:06:25 INFO DAGScheduler: waiting: Set(ResultStage 30)
      18/04/01 15:06:25 INFO DAGScheduler: failed: Set()
      18/04/01 15:06:25 INFO DAGScheduler: Submitting ResultStage 30 (MapPartitionsRDD[749] at createDataFrame at RedshiftWriter.scala:275), which has no missing parents
      18/04/01 15:06:25 INFO MemoryStore: Block broadcast_664 stored as values in memory (estimated size 114.6 KB, free 159.8 MB)
      18/04/01 15:06:25 INFO MemoryStore: Block broadcast_664_piece0 stored as bytes in memory (estimated size 44.1 KB, free 159.9 MB)
      18/04/01 15:06:25 INFO BlockManagerInfo: Added broadcast_664_piece0 in memory on 172.19.103.84:18128 (size: 44.1 KB, free: 492.3 MB)
      18/04/01 15:06:25 INFO SparkContext: Created broadcast 664 from broadcast at DAGScheduler.scala:1006
      18/04/01 15:06:25 INFO DAGScheduler: Submitting 16 missing tasks from ResultStage 30 (MapPartitionsRDD[749] at createDataFrame at RedshiftWriter.scala:275)
      18/04/01 15:06:25 INFO TaskSchedulerImpl: Adding task set 30.0 with 16 tasks
      18/04/01 15:06:25 INFO TaskSetManager: Starting task 0.0 in stage 30.0 (TID 5464, ip-172-19-110-243.ec2.internal, partition 0,PROCESS_LOCAL, 2061 bytes)
      18/04/01 15:06:25 INFO TaskSetManager: Starting task 1.0 in stage 30.0 (TID 5465, ip-172-19-101-76.ec2.internal, partition 1,PROCESS_LOCAL, 2061 bytes)
      18/04/01 15:06:25 INFO TaskSetManager: Starting task 2.0 in stage 30.0 (TID 5466, ip-172-19-105-117.ec2.internal, partition 2,PROCESS_LOCAL, 2061 bytes)
      18/04/01 15:06:25 INFO TaskSetManager: Starting task 3.0 in stage 30.0 (TID 5467, ip-172-19-107-199.ec2.internal, partition 3,PROCESS_LOCAL, 2061 bytes)
      18/04/01 15:06:25 INFO TaskSetManager: Starting task 4.0 in stage 30.0 (TID 5468, ip-172-19-108-144.ec2.internal, partition 4,PROCESS_LOCAL, 2061 bytes)
      18/04/01 15:06:25 INFO TaskSetManager: Starting task 5.0 in stage 30.0 (TID 5469, ip-172-19-103-249.ec2.internal, partition 5,PROCESS_LOCAL, 2061 bytes)
      18/04/01 15:06:25 INFO TaskSetManager: Starting task 6.0 in stage 30.0 (TID 5470, ip-172-19-102-6.ec2.internal, partition 6,PROCESS_LOCAL, 2061 bytes)
      18/04/01 15:06:25 INFO TaskSetManager: Starting task 7.0 in stage 30.0 (TID 5471, ip-172-19-105-221.ec2.internal, partition 7,PROCESS_LOCAL, 2061 bytes)
      18/04/01 15:06:25 INFO TaskSetManager: Starting task 8.0 in stage 30.0 (TID 5472, ip-172-19-110-149.ec2.internal, partition 8,PROCESS_LOCAL, 2061 bytes)
      18/04/01 15:06:25 INFO TaskSetManager: Starting task 9.0 in stage 30.0 (TID 5473, ip-172-19-104-236.ec2.internal, partition 9,PROCESS_LOCAL, 2061 bytes)
      18/04/01 15:06:25 INFO TaskSetManager: Starting task 10.0 in stage 30.0 (TID 5474, ip-172-19-107-198.ec2.internal, partition 10,PROCESS_LOCAL, 2061 bytes)
      18/04/01 15:06:25 INFO TaskSetManager: Starting task 11.0 in stage 30.0 (TID 5475, ip-172-19-100-11.ec2.internal, partition 11,PROCESS_LOCAL, 2061 bytes)
      18/04/01 15:06:25 INFO TaskSetManager: Starting task 12.0 in stage 30.0 (TID 5476, ip-172-19-110-176.ec2.internal, partition 12,PROCESS_LOCAL, 2061 bytes)
      18/04/01 15:06:25 INFO TaskSetManager: Starting task 13.0 in stage 30.0 (TID 5477, ip-172-19-109-92.ec2.internal, partition 13,PROCESS_LOCAL, 2061 bytes)
      18/04/01 15:06:25 INFO TaskSetManager: Starting task 14.0 in stage 30.0 (TID 5478, ip-172-19-106-52.ec2.internal, partition 14,PROCESS_LOCAL, 2061 bytes)
      18/04/01 15:06:25 INFO TaskSetManager: Starting task 15.0 in stage 30.0 (TID 5479, ip-172-19-103-87.ec2.internal, partition 15,PROCESS_LOCAL, 2061 bytes)
      18/04/01 15:06:25 INFO BlockManagerInfo: Added broadcast_664_piece0 in memory on ip-172-19-101-76.ec2.internal:16864 (size: 44.1 KB, free: 1932.7 MB)
      18/04/01 15:06:25 INFO BlockManagerInfo: Added broadcast_664_piece0 in memory on ip-172-19-103-249.ec2.internal:41486 (size: 44.1 KB, free: 1939.5 MB)
      18/04/01 15:06:25 INFO BlockManagerInfo: Added broadcast_664_piece0 in memory on ip-172-19-100-11.ec2.internal:24171 (size: 44.1 KB, free: 1935.0 MB)
      18/04/01 15:06:25 INFO BlockManagerInfo: Added broadcast_664_piece0 in memory on ip-172-19-102-6.ec2.internal:28761 (size: 44.1 KB, free: 1940.4 MB)
      18/04/01 15:06:25 INFO BlockManagerInfo: Added broadcast_664_piece0 in memory on ip-172-19-103-87.ec2.internal:62681 (size: 44.1 KB, free: 1932.3 MB)
      18/04/01 15:06:25 INFO BlockManagerInfo: Added broadcast_664_piece0 in memory on ip-172-19-110-243.ec2.internal:14787 (size: 44.1 KB, free: 1942.5 MB)
      18/04/01 15:06:25 INFO BlockManagerInfo: Added broadcast_664_piece0 in memory on ip-172-19-108-144.ec2.internal:17672 (size: 44.1 KB, free: 1934.5 MB)
      18/04/01 15:06:25 INFO BlockManagerInfo: Added broadcast_664_piece0 in memory on ip-172-19-110-149.ec2.internal:64342 (size: 44.1 KB, free: 1931.3 MB)
      18/04/01 15:06:25 INFO BlockManagerInfo: Added broadcast_664_piece0 in memory on ip-172-19-110-176.ec2.internal:48439 (size: 44.1 KB, free: 1937.9 MB)
      18/04/01 15:06:25 INFO BlockManagerInfo: Added broadcast_664_piece0 in memory on ip-172-19-105-117.ec2.internal:45766 (size: 44.1 KB, free: 1933.5 MB)
      18/04/01 15:06:25 INFO BlockManagerInfo: Added broadcast_664_piece0 in memory on ip-172-19-105-221.ec2.internal:52999 (size: 44.1 KB, free: 1934.4 MB)
      18/04/01 15:06:25 INFO BlockManagerInfo: Added broadcast_664_piece0 in memory on ip-172-19-107-199.ec2.internal:35094 (size: 44.1 KB, free: 1938.9 MB)
      18/04/01 15:06:25 INFO BlockManagerInfo: Added broadcast_664_piece0 in memory on ip-172-19-107-198.ec2.internal:11244 (size: 44.1 KB, free: 1938.3 MB)
      18/04/01 15:06:25 INFO BlockManagerInfo: Added broadcast_664_piece0 in memory on ip-172-19-106-52.ec2.internal:65123 (size: 44.1 KB, free: 1934.6 MB)
      18/04/01 15:06:25 INFO BlockManagerInfo: Added broadcast_664_piece0 in memory on ip-172-19-104-236.ec2.internal:52270 (size: 44.1 KB, free: 1934.4 MB)
      18/04/01 15:06:25 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 11 to ip-172-19-103-249.ec2.internal:57024
      18/04/01 15:06:25 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 11 to ip-172-19-100-11.ec2.internal:36641
      18/04/01 15:06:25 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 11 is 1249 bytes
      18/04/01 15:06:25 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 11 is 1249 bytes
      18/04/01 15:06:25 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 11 to ip-172-19-105-117.ec2.internal:13679
      18/04/01 15:06:25 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 11 to ip-172-19-103-87.ec2.internal:36114
      18/04/01 15:06:25 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 11 to ip-172-19-110-149.ec2.internal:53545
      18/04/01 15:06:25 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 11 to ip-172-19-105-221.ec2.internal:16426
      18/04/01 15:06:25 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 11 to ip-172-19-101-76.ec2.internal:44931
      18/04/01 15:06:25 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 11 to ip-172-19-102-6.ec2.internal:52572
      18/04/01 15:06:25 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 11 to ip-172-19-110-243.ec2.internal:37638
      18/04/01 15:06:25 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 11 to ip-172-19-107-198.ec2.internal:30635
      18/04/01 15:06:25 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 11 to ip-172-19-107-199.ec2.internal:55529
      18/04/01 15:06:25 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 11 to ip-172-19-104-236.ec2.internal:59497
      18/04/01 15:06:25 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 11 to ip-172-19-110-176.ec2.internal:33639
      18/04/01 15:06:25 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 11 to ip-172-19-108-144.ec2.internal:62978
      18/04/01 15:06:25 INFO BlockManagerInfo: Added broadcast_664_piece0 in memory on ip-172-19-109-92.ec2.internal:22910 (size: 44.1 KB, free: 1939.7 MB)
      18/04/01 15:06:25 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 11 to ip-172-19-106-52.ec2.internal:24871
      18/04/01 15:06:25 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 11 to ip-172-19-109-92.ec2.internal:39158
      18/04/01 15:06:27 ERROR TaskSchedulerImpl: Lost executor 9 on ip-172-19-109-92.ec2.internal: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.
      18/04/01 15:06:27 WARN TaskSetManager: Lost task 13.0 in stage 30.0 (TID 5477, ip-172-19-109-92.ec2.internal): ExecutorLostFailure (executor 9 exited caused by one of the running tasks) Reason: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.
      18/04/01 15:06:27 INFO DAGScheduler: Executor lost: 9 (epoch 12)
      18/04/01 15:06:27 INFO BlockManagerMasterEndpoint: Trying to remove executor 9 from BlockManagerMaster.
      18/04/01 15:06:27 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(9, ip-172-19-109-92.ec2.internal, 22910)
      18/04/01 15:06:27 INFO BlockManagerMaster: Removed 9 successfully in removeExecutor
      18/04/01 15:06:27 INFO ShuffleMapStage: ShuffleMapStage 29 is now unavailable on executor 9 (190/200, false)
      18/04/01 15:06:27 INFO ShuffleMapStage: ShuffleMapStage 28 is now unavailable on executor 9 (602/640, false)
      18/04/01 15:06:27 INFO AppClient$ClientEndpoint: Executor updated: app-20180401150045-72810/9 is now EXITED (Command exited with code 1)
      18/04/01 15:06:27 INFO SparkDeploySchedulerBackend: Executor app-20180401150045-72810/9 removed: Command exited with code 1
      18/04/01 15:06:27 INFO SparkDeploySchedulerBackend: Asked to remove non-existent executor 9
      18/04/01 15:06:27 INFO AppClient$ClientEndpoint: Executor added: app-20180401150045-72810/16 on worker-20180211071355-ip-172-19-103-87.legacy.vungle.com-7078 (ip-172-19-103-87.legacy.vungle.com:7078) with 1 cores
      18/04/01 15:06:27 INFO SparkDeploySchedulerBackend: Granted executor ID app-20180401150045-72810/16 on hostPort ip-172-19-103-87.legacy.vungle.com:7078 with 1 cores, 3.0 GB RAM
      18/04/01 15:06:27 INFO AppClient$ClientEndpoint: Executor updated: app-20180401150045-72810/16 is now RUNNING
      18/04/01 15:06:29 INFO SparkDeploySchedulerBackend: Registered executor NettyRpcEndpointRef(null) (ip-172-19-103-87.ec2.internal:36668) with ID 16
      18/04/01 15:06:29 INFO TaskSetManager: Starting task 13.1 in stage 30.0 (TID 5480, ip-172-19-103-87.ec2.internal, partition 13,PROCESS_LOCAL, 2061 bytes)
      18/04/01 15:06:29 INFO BlockManagerMasterEndpoint: Registering block manager ip-172-19-103-87.ec2.internal:13372 with 1983.0 MB RAM, BlockManagerId(16, ip-172-19-103-87.ec2.internal, 13372)
      18/04/01 15:06:30 INFO TaskSetManager: Finished task 7.0 in stage 30.0 (TID 5471) in 4354 ms on ip-172-19-105-221.ec2.internal (1/16)
      18/04/01 15:06:30 INFO TaskSetManager: Finished task 8.0 in stage 30.0 (TID 5472) in 4486 ms on ip-172-19-110-149.ec2.internal (2/16)
      18/04/01 15:06:30 INFO TaskSetManager: Finished task 5.0 in stage 30.0 (TID 5469) in 4527 ms on ip-172-19-103-249.ec2.internal (3/16)
      18/04/01 15:06:30 INFO TaskSetManager: Finished task 2.0 in stage 30.0 (TID 5466) in 4536 ms on ip-172-19-105-117.ec2.internal (4/16)
      18/04/01 15:06:30 INFO TaskSetManager: Finished task 4.0 in stage 30.0 (TID 5468) in 4805 ms on ip-172-19-108-144.ec2.internal (5/16)
      18/04/01 15:06:30 INFO TaskSetManager: Finished task 9.0 in stage 30.0 (TID 5473) in 4818 ms on ip-172-19-104-236.ec2.internal (6/16)
      18/04/01 15:06:30 INFO TaskSetManager: Finished task 1.0 in stage 30.0 (TID 5465) in 4927 ms on ip-172-19-101-76.ec2.internal (7/16)
      18/04/01 15:06:30 INFO TaskSetManager: Finished task 10.0 in stage 30.0 (TID 5474) in 4932 ms on ip-172-19-107-198.ec2.internal (8/16)
      18/04/01 15:06:31 INFO TaskSetManager: Finished task 0.0 in stage 30.0 (TID 5464) in 5469 ms on ip-172-19-110-243.ec2.internal (9/16)
      18/04/01 15:06:31 INFO TaskSetManager: Finished task 15.0 in stage 30.0 (TID 5479) in 5503 ms on ip-172-19-103-87.ec2.internal (10/16)
      18/04/01 15:06:31 INFO BlockManagerInfo: Added broadcast_664_piece0 in memory on ip-172-19-103-87.ec2.internal:13372 (size: 44.1 KB, free: 1983.0 MB)
      18/04/01 15:06:31 INFO TaskSetManager: Finished task 12.0 in stage 30.0 (TID 5476) in 5822 ms on ip-172-19-110-176.ec2.internal (11/16)
      18/04/01 15:06:32 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 11 to ip-172-19-103-87.ec2.internal:36668
      18/04/01 15:06:32 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 11 is 1208 bytes
      18/04/01 15:06:32 WARN TaskSetManager: Lost task 13.1 in stage 30.0 (TID 5480, ip-172-19-103-87.ec2.internal): FetchFailed(null, shuffleId=11, mapId=-1, reduceId=13, message=
      org.apache.spark.shuffle.MetadataFetchFailedException: Missing an output location for shuffle 11
          at org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:542)
          at org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:538)
          at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:778)
          at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
          at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
          at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:777)
          at org.apache.spark.MapOutputTracker$.org$apache$spark$MapOutputTracker$$convertMapStatuses(MapOutputTracker.scala:538)
          at org.apache.spark.MapOutputTracker.getMapSizesByExecutorId(MapOutputTracker.scala:155)
          at org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:47)
          at org.apache.spark.sql.execution.ShuffledRowRDD.compute(ShuffledRowRDD.scala:166)
          at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
          at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
          at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
          at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
          at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
          at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
          at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
          at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
          at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
          at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
          at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
          at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
          at org.apache.spark.scheduler.Task.run(Task.scala:89)
          at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at java.lang.Thread.run(Thread.java:748)

      )
      18/04/01 15:06:32 INFO DAGScheduler: Marking ResultStage 30 (save at RedshiftWriter.scala:278) as failed due to a fetch failure from ShuffleMapStage 29 (mapPartitions at RedshiftWriter.scala:237)
      18/04/01 15:06:32 INFO DAGScheduler: ResultStage 30 (save at RedshiftWriter.scala:278) failed in 6.558 s
      18/04/01 15:06:32 INFO DAGScheduler: Resubmitting ShuffleMapStage 29 (mapPartitions at RedshiftWriter.scala:237) and ResultStage 30 (save at RedshiftWriter.scala:278) due to fetch failure
      18/04/01 15:06:32 INFO DAGScheduler: Resubmitting failed stages
      18/04/01 15:06:32 INFO DAGScheduler: Submitting ShuffleMapStage 28 (MapPartitionsRDD[739] at cache at IdspAggregator.scala:214), which has no missing parents
      18/04/01 15:06:32 INFO MemoryStore: Block broadcast_665 stored as values in memory (estimated size 111.8 KB, free 160.0 MB)
      18/04/01 15:06:32 INFO MemoryStore: Block broadcast_665_piece0 stored as bytes in memory (estimated size 32.4 KB, free 160.0 MB)
      18/04/01 15:06:32 INFO BlockManagerInfo: Added broadcast_665_piece0 in memory on 172.19.103.84:18128 (size: 32.4 KB, free: 492.3 MB)
      18/04/01 15:06:32 INFO SparkContext: Created broadcast 665 from broadcast at DAGScheduler.scala:1006
      18/04/01 15:06:32 INFO DAGScheduler: Submitting 38 missing tasks from ShuffleMapStage 28 (MapPartitionsRDD[739] at cache at IdspAggregator.scala:214)
      18/04/01 15:06:32 INFO TaskSchedulerImpl: Adding task set 28.1 with 38 tasks
      18/04/01 15:06:32 INFO TaskSetManager: Starting task 0.0 in stage 28.1 (TID 5481, ip-172-19-107-198.ec2.internal, partition 5,PROCESS_LOCAL, 2449 bytes)
      18/04/01 15:06:32 INFO TaskSetManager: Starting task 1.0 in stage 28.1 (TID 5482, ip-172-19-110-176.ec2.internal, partition 25,PROCESS_LOCAL, 2449 bytes)
      18/04/01 15:06:32 INFO TaskSetManager: Starting task 2.0 in stage 28.1 (TID 5483, ip-172-19-105-221.ec2.internal, partition 40,PROCESS_LOCAL, 2449 bytes)
      18/04/01 15:06:32 INFO TaskSetManager: Starting task 3.0 in stage 28.1 (TID 5484, ip-172-19-110-149.ec2.internal, partition 59,PROCESS_LOCAL, 2449 bytes)
      18/04/01 15:06:32 INFO TaskSetManager: Starting task 4.0 in stage 28.1 (TID 5485, ip-172-19-103-87.ec2.internal, partition 76,PROCESS_LOCAL, 2449 bytes)
      18/04/01 15:06:32 INFO TaskSetManager: Starting task 5.0 in stage 28.1 (TID 5486, ip-172-19-105-117.ec2.internal, partition 100,PROCESS_LOCAL, 2449 bytes)
      18/04/01 15:06:32 INFO TaskSetManager: Starting task 6.0 in stage 28.1 (TID 5487, ip-172-19-103-249.ec2.internal, partition 117,PROCESS_LOCAL, 2449 bytes)
      18/04/01 15:06:32 INFO TaskSetManager: Starting task 7.0 in stage 28.1 (TID 5488, ip-172-19-101-76.ec2.internal, partition 133,PROCESS_LOCAL, 2449 bytes)
      18/04/01 15:06:32 INFO TaskSetManager: Starting task 8.0 in stage 28.1 (TID 5489, ip-172-19-104-236.ec2.internal, partition 154,PROCESS_LOCAL, 2449 bytes)
      18/04/01 15:06:32 INFO TaskSetManager: Starting task 9.0 in stage 28.1 (TID 5490, ip-172-19-110-243.ec2.internal, partition 169,PROCESS_LOCAL, 2449 bytes)
      18/04/01 15:06:32 INFO TaskSetManager: Starting task 10.0 in stage 28.1 (TID 5491, ip-172-19-108-144.ec2.internal, partition 189,PROCESS_LOCAL, 2449 bytes)
      18/04/01 15:06:32 INFO TaskSetManager: Starting task 11.0 in stage 28.1 (TID 5492, ip-172-19-103-87.ec2.internal, partition 205,PROCESS_LOCAL, 2449 bytes)
      18/04/01 15:06:32 INFO BlockManagerInfo: Added broadcast_665_piece0 in memory on ip-172-19-105-221.ec2.internal:52999 (size: 32.4 KB, free: 1934.4 MB)
      18/04/01 15:06:32 INFO BlockManagerInfo: Added broadcast_665_piece0 in memory on ip-172-19-104-236.ec2.internal:52270 (size: 32.4 KB, free: 1934.4 MB)
      18/04/01 15:06:32 INFO BlockManagerInfo: Added broadcast_665_piece0 in memory on ip-172-19-103-249.ec2.internal:41486 (size: 32.4 KB, free: 1939.4 MB)
      18/04/01 15:06:32 INFO BlockManagerInfo: Added broadcast_665_piece0 in memory on ip-172-19-103-87.ec2.internal:62681 (size: 32.4 KB, free: 1932.3 MB)
      18/04/01 15:06:32 INFO BlockManagerInfo: Added broadcast_665_piece0 in memory on ip-172-19-110-243.ec2.internal:14787 (size: 32.4 KB, free: 1942.4 MB)
      18/04/01 15:06:32 INFO BlockManagerInfo: Added broadcast_665_piece0 in memory on ip-172-19-105-117.ec2.internal:45766 (size: 32.4 KB, free: 1933.5 MB)
      18/04/01 15:06:32 INFO BlockManagerInfo: Added broadcast_665_piece0 in memory on ip-172-19-108-144.ec2.internal:17672 (size: 32.4 KB, free: 1934.5 MB)
      18/04/01 15:06:32 INFO BlockManagerInfo: Added broadcast_665_piece0 in memory on ip-172-19-107-198.ec2.internal:11244 (size: 32.4 KB, free: 1938.3 MB)
      18/04/01 15:06:32 INFO BlockManagerInfo: Added broadcast_665_piece0 in memory on ip-172-19-101-76.ec2.internal:16864 (size: 32.4 KB, free: 1932.7 MB)
      18/04/01 15:06:32 INFO BlockManagerInfo: Added broadcast_665_piece0 in memory on ip-172-19-110-149.ec2.internal:64342 (size: 32.4 KB, free: 1931.3 MB)
      18/04/01 15:06:32 INFO BlockManagerInfo: Added broadcast_665_piece0 in memory on ip-172-19-110-176.ec2.internal:48439 (size: 32.4 KB, free: 1937.9 MB)
      18/04/01 15:06:32 INFO BlockManagerInfo: Added broadcast_397_piece0 in memory on ip-172-19-103-87.ec2.internal:62681 (size: 19.6 KB, free: 1932.2 MB)
      18/04/01 15:06:32 INFO BlockManagerInfo: Added broadcast_438_piece0 in memory on ip-172-19-103-249.ec2.internal:41486 (size: 19.6 KB, free: 1939.4 MB)
      18/04/01 15:06:32 INFO BlockManagerInfo: Added broadcast_454_piece0 in memory on ip-172-19-101-76.ec2.internal:16864 (size: 19.6 KB, free: 1932.7 MB)
      18/04/01 15:06:32 INFO BlockManagerInfo: Added broadcast_361_piece0 in memory on ip-172-19-105-221.ec2.internal:52999 (size: 19.6 KB, free: 1934.3 MB)
      18/04/01 15:06:32 INFO BlockManagerInfo: Added broadcast_475_piece0 in memory on ip-172-19-104-236.ec2.internal:52270 (size: 19.6 KB, free: 1934.4 MB)
      18/04/01 15:06:32 INFO BlockManagerInfo: Added broadcast_421_piece0 in memory on ip-172-19-105-117.ec2.internal:45766 (size: 19.6 KB, free: 1933.4 MB)
      18/04/01 15:06:32 INFO BlockManagerInfo: Added broadcast_346_piece0 in memory on ip-172-19-110-176.ec2.internal:48439 (size: 19.6 KB, free: 1937.9 MB)
      18/04/01 15:06:32 INFO BlockManagerInfo: Added broadcast_510_piece0 in memory on ip-172-19-108-144.ec2.internal:17672 (size: 19.6 KB, free: 1934.5 MB)
      18/04/01 15:06:32 INFO BlockManagerInfo: Added broadcast_490_piece0 in memory on ip-172-19-110-243.ec2.internal:14787 (size: 19.6 KB, free: 1942.4 MB)
      18/04/01 15:06:32 INFO BlockManagerInfo: Added broadcast_326_piece0 in memory on ip-172-19-107-198.ec2.internal:11244 (size: 19.6 KB, free: 1938.3 MB)
      18/04/01 15:06:32 INFO BlockManagerInfo: Added broadcast_380_piece0 in memory on ip-172-19-110-149.ec2.internal:64342 (size: 19.6 KB, free: 1931.3 MB)
      18/04/01 15:06:32 INFO BlockManagerInfo: Added broadcast_665_piece0 in memory on ip-172-19-103-87.ec2.internal:13372 (size: 32.4 KB, free: 1982.9 MB)
      18/04/01 15:06:32 INFO BlockManagerInfo: Added broadcast_526_piece0 in memory on ip-172-19-103-87.ec2.internal:13372 (size: 19.6 KB, free: 1982.9 MB)
      18/04/01 15:06:33 INFO BlockManagerInfo: Added rdd_661_189 in memory on ip-172-19-108-144.ec2.internal:17672 (size: 1523.9 KB, free: 1933.0 MB)
      18/04/01 15:06:33 INFO BlockManagerInfo: Added rdd_661_40 in memory on ip-172-19-105-221.ec2.internal:52999 (size: 1510.0 KB, free: 1932.9 MB)
      18/04/01 15:06:33 INFO BlockManagerInfo: Added rdd_661_154 in memory on ip-172-19-104-236.ec2.internal:52270 (size: 1533.5 KB, free: 1932.9 MB)
      18/04/01 15:06:33 INFO TaskSetManager: Starting task 12.0 in stage 28.1 (TID 5493, ip-172-19-108-144.ec2.internal, partition 225,PROCESS_LOCAL, 2449 bytes)
      18/04/01 15:06:33 INFO TaskSetManager: Finished task 10.0 in stage 28.1 (TID 5491) in 1277 ms on ip-172-19-108-144.ec2.internal (1/38)
      18/04/01 15:06:33 INFO TaskSetManager: Starting task 13.0 in stage 28.1 (TID 5494, ip-172-19-105-221.ec2.internal, partition 242,PROCESS_LOCAL, 2449 bytes)
      18/04/01 15:06:33 INFO TaskSetManager: Finished task 2.0 in stage 28.1 (TID 5483) in 1280 ms on ip-172-19-105-221.ec2.internal (2/38)
      18/04/01 15:06:33 INFO BlockManagerInfo: Added rdd_661_100 in memory on ip-172-19-105-117.ec2.internal:45766 (size: 1517.9 KB, free: 1932.0 MB)
      18/04/01 15:06:33 INFO BlockManagerInfo: Added broadcast_546_piece0 in memory on ip-172-19-108-144.ec2.internal:17672 (size: 19.6 KB, free: 1933.0 MB)
      18/04/01 15:06:33 INFO BlockManagerInfo: Added broadcast_563_piece0 in memory on ip-172-19-105-221.ec2.internal:52999 (size: 19.6 KB, free: 1932.8 MB)
      18/04/01 15:06:33 INFO TaskSetManager: Starting task 14.0 in stage 28.1 (TID 5495, ip-172-19-104-236.ec2.internal, partition 255,PROCESS_LOCAL, 2449 bytes)
      18/04/01 15:06:33 INFO TaskSetManager: Finished task 8.0 in stage 28.1 (TID 5489) in 1298 ms on ip-172-19-104-236.ec2.internal (3/38)
      18/04/01 15:06:33 INFO BlockManagerInfo: Added broadcast_576_piece0 in memory on ip-172-19-104-236.ec2.internal:52270 (size: 19.6 KB, free: 1932.8 MB)
      18/04/01 15:06:33 INFO BlockManagerInfo: Added rdd_661_133 in memory on ip-172-19-101-76.ec2.internal:16864 (size: 1533.1 KB, free: 1931.2 MB)
      18/04/01 15:06:33 INFO BlockManagerInfo: Added rdd_661_59 in memory on ip-172-19-110-149.ec2.internal:64342 (size: 1525.7 KB, free: 1929.8 MB)
      18/04/01 15:06:34 INFO BlockManagerInfo: Added rdd_661_25 in memory on ip-172-19-110-176.ec2.internal:48439 (size: 1517.6 KB, free: 1936.4 MB)
      18/04/01 15:06:34 INFO BlockManagerInfo: Added rdd_661_5 in memory on ip-172-19-107-198.ec2.internal:11244 (size: 1539.1 KB, free: 1936.8 MB)
      18/04/01 15:06:34 INFO TaskSetManager: Starting task 15.0 in stage 28.1 (TID 5496, ip-172-19-105-117.ec2.internal, partition 266,PROCESS_LOCAL, 2449 bytes)
      18/04/01 15:06:34 INFO TaskSetManager: Finished task 5.0 in stage 28.1 (TID 5486) in 1419 ms on ip-172-19-105-117.ec2.internal (4/38)
      18/04/01 15:06:34 INFO BlockManagerInfo: Added broadcast_587_piece0 in memory on ip-172-19-105-117.ec2.internal:45766 (size: 19.6 KB, free: 1931.9 MB)
      18/04/01 15:06:34 INFO TaskSetManager: Starting task 16.0 in stage 28.1 (TID 5497, ip-172-19-101-76.ec2.internal, partition 282,PROCESS_LOCAL, 2449 bytes)
      18/04/01 15:06:34 INFO TaskSetManager: Finished task 7.0 in stage 28.1 (TID 5488) in 1456 ms on ip-172-19-101-76.ec2.internal (5/38)
      18/04/01 15:06:34 INFO TaskSetManager: Starting task 17.0 in stage 28.1 (TID 5498, ip-172-19-110-149.ec2.internal, partition 299,PROCESS_LOCAL, 2449 bytes)
      18/04/01 15:06:34 INFO TaskSetManager: Finished task 3.0 in stage 28.1 (TID 5484) in 1460 ms on ip-172-19-110-149.ec2.internal (6/38)
      18/04/01 15:06:34 INFO BlockManagerInfo: Added broadcast_603_piece0 in memory on ip-172-19-101-76.ec2.internal:16864 (size: 19.6 KB, free: 1931.2 MB)
      18/04/01 15:06:34 INFO BlockManagerInfo: Added broadcast_620_piece0 in memory on ip-172-19-110-149.ec2.internal:64342 (size: 19.6 KB, free: 1929.8 MB)
      18/04/01 15:06:34 INFO BlockManagerInfo: Added rdd_661_169 in memory on ip-172-19-110-243.ec2.internal:14787 (size: 1532.6 KB, free: 1940.9 MB)
      18/04/01 15:06:34 INFO TaskSetManager: Starting task 18.0 in stage 28.1 (TID 5499, ip-172-19-110-176.ec2.internal, partition 315,PROCESS_LOCAL, 2449 bytes)
      18/04/01 15:06:34 INFO TaskSetManager: Finished task 1.0 in stage 28.1 (TID 5482) in 1489 ms on ip-172-19-110-176.ec2.internal (7/38)
      18/04/01 15:06:34 INFO BlockManagerInfo: Added broadcast_636_piece0 in memory on ip-172-19-110-176.ec2.internal:48439 (size: 19.6 KB, free: 1936.4 MB)
      18/04/01 15:06:34 INFO TaskSetManager: Starting task 19.0 in stage 28.1 (TID 5500, ip-172-19-107-198.ec2.internal, partition 325,PROCESS_LOCAL, 2449 bytes)
      18/04/01 15:06:34 INFO TaskSetManager: Finished task 0.0 in stage 28.1 (TID 5481) in 1509 ms on ip-172-19-107-198.ec2.internal (8/38)
      18/04/01 15:06:34 INFO BlockManagerInfo: Added rdd_661_117 in memory on ip-172-19-103-249.ec2.internal:41486 (size: 1534.0 KB, free: 1937.9 MB)
      18/04/01 15:06:34 INFO TaskSetManager: Starting task 20.0 in stage 28.1 (TID 5501, ip-172-19-110-243.ec2.internal, partition 345,PROCESS_LOCAL, 2449 bytes)
      18/04/01 15:06:34 INFO TaskSetManager: Finished task 9.0 in stage 28.1 (TID 5490) in 1612 ms on ip-172-19-110-243.ec2.internal (9/38)
      18/04/01 15:06:34 INFO BlockManagerInfo: Added broadcast_346_piece0 in memory on ip-172-19-110-243.ec2.internal:14787 (size: 19.6 KB, free: 1940.9 MB)
      18/04/01 15:06:34 INFO TaskSetManager: Starting task 21.0 in stage 28.1 (TID 5502, ip-172-19-103-249.ec2.internal, partition 360,PROCESS_LOCAL, 2449 bytes)
      18/04/01 15:06:34 INFO TaskSetManager: Finished task 6.0 in stage 28.1 (TID 5487) in 1821 ms on ip-172-19-103-249.ec2.internal (10/38)
      18/04/01 15:06:34 INFO BlockManagerInfo: Added broadcast_361_piece0 in memory on ip-172-19-103-249.ec2.internal:41486 (size: 19.6 KB, free: 1937.9 MB)
      18/04/01 15:06:35 INFO BlockManagerInfo: Added rdd_661_225 in memory on ip-172-19-108-144.ec2.internal:17672 (size: 1531.9 KB, free: 1931.5 MB)
      18/04/01 15:06:35 INFO BlockManagerInfo: Added rdd_661_255 in memory on ip-172-19-104-236.ec2.internal:52270 (size: 1519.2 KB, free: 1931.4 MB)
      18/04/01 15:06:35 INFO BlockManagerInfo: Added rdd_661_242 in memory on ip-172-19-105-221.ec2.internal:52999 (size: 1520.8 KB, free: 1931.4 MB)
      18/04/01 15:06:35 INFO TaskSetManager: Starting task 22.0 in stage 28.1 (TID 5503, ip-172-19-108-144.ec2.internal, partition 379,PROCESS_LOCAL, 2449 bytes)
      18/04/01 15:06:35 INFO TaskSetManager: Finished task 12.0 in stage 28.1 (TID 5493) in 1204 ms on ip-172-19-108-144.ec2.internal (11/38)
      18/04/01 15:06:35 INFO BlockManagerInfo: Added broadcast_380_piece0 in memory on ip-172-19-108-144.ec2.internal:17672 (size: 19.6 KB, free: 1931.5 MB)
      18/04/01 15:06:35 INFO TaskSetManager: Starting task 23.0 in stage 28.1 (TID 5504, ip-172-19-104-236.ec2.internal, partition 396,PROCESS_LOCAL, 2449 bytes)
      18/04/01 15:06:35 INFO TaskSetManager: Finished task 14.0 in stage 28.1 (TID 5495) in 1233 ms on ip-172-19-104-236.ec2.internal (12/38)
      18/04/01 15:06:35 INFO BlockManagerInfo: Added broadcast_397_piece0 in memory on ip-172-19-104-236.ec2.internal:52270 (size: 19.6 KB, free: 1931.3 MB)
      18/04/01 15:06:35 INFO TaskSetManager: Starting task 24.0 in stage 28.1 (TID 5505, ip-172-19-105-221.ec2.internal, partition 420,PROCESS_LOCAL, 2449 bytes)
      18/04/01 15:06:35 INFO TaskSetManager: Finished task 13.0 in stage 28.1 (TID 5494) in 1308 ms on ip-172-19-105-221.ec2.internal (13/38)
      18/04/01 15:06:35 INFO BlockManagerInfo: Added broadcast_421_piece0 in memory on ip-172-19-105-221.ec2.internal:52999 (size: 19.6 KB, free: 1931.3 MB)
      18/04/01 15:06:35 INFO BlockManagerInfo: Added rdd_661_325 in memory on ip-172-19-107-198.ec2.internal:11244 (size: 16.2 KB, free: 1936.8 MB)
      18/04/01 15:06:35 INFO TaskSetManager: Starting task 25.0 in stage 28.1 (TID 5506, ip-172-19-107-198.ec2.internal, partition 437,PROCESS_LOCAL, 2449 bytes)
      18/04/01 15:06:35 INFO TaskSetManager: Finished task 19.0 in stage 28.1 (TID 5500) in 1153 ms on ip-172-19-107-198.ec2.internal (14/38)
      18/04/01 15:06:35 INFO BlockManagerInfo: Added broadcast_438_piece0 in memory on ip-172-19-107-198.ec2.internal:11244 (size: 19.6 KB, free: 1936.7 MB)
      18/04/01 15:06:35 INFO BlockManagerInfo: Added rdd_661_345 in memory on ip-172-19-110-243.ec2.internal:14787 (size: 19.8 KB, free: 1940.9 MB)
      18/04/01 15:06:35 INFO BlockManagerInfo: Added rdd_661_282 in memory on ip-172-19-101-76.ec2.internal:16864 (size: 1539.9 KB, free: 1929.7 MB)
      18/04/01 15:06:35 INFO TaskSetManager: Starting task 26.0 in stage 28.1 (TID 5507, ip-172-19-110-243.ec2.internal, partition 453,PROCESS_LOCAL, 2449 bytes)
      18/04/01 15:06:35 INFO TaskSetManager: Finished task 20.0 in stage 28.1 (TID 5501) in 1113 ms on ip-172-19-110-243.ec2.internal (15/38)
      18/04/01 15:06:35 INFO BlockManagerInfo: Added rdd_661_266 in memory on ip-172-19-105-117.ec2.internal:45766 (size: 1521.4 KB, free: 1930.5 MB)
      18/04/01 15:06:35 INFO BlockManagerInfo: Added broadcast_454_piece0 in memory on ip-172-19-110-243.ec2.internal:14787 (size: 19.6 KB, free: 1940.9 MB)
      18/04/01 15:06:35 INFO BlockManagerInfo: Added rdd_661_299 in memory on ip-172-19-110-149.ec2.internal:64342 (size: 1532.3 KB, free: 1928.3 MB)
      18/04/01 15:06:35 INFO TaskSetManager: Starting task 27.0 in stage 28.1 (TID 5508, ip-172-19-101-76.ec2.internal, partition 474,PROCESS_LOCAL, 2449 bytes)
      18/04/01 15:06:35 INFO TaskSetManager: Finished task 16.0 in stage 28.1 (TID 5497) in 1400 ms on ip-172-19-101-76.ec2.internal (16/38)
      18/04/01 15:06:35 INFO BlockManagerInfo: Added broadcast_475_piece0 in memory on ip-172-19-101-76.ec2.internal:16864 (size: 19.6 KB, free: 1929.6 MB)
      18/04/01 15:06:35 INFO TaskSetManager: Starting task 28.0 in stage 28.1 (TID 5509, ip-172-19-105-117.ec2.internal, partition 489,PROCESS_LOCAL, 2449 bytes)
      18/04/01 15:06:35 INFO TaskSetManager: Finished task 15.0 in stage 28.1 (TID 5496) in 1449 ms on ip-172-19-105-117.ec2.internal (17/38)
      18/04/01 15:06:35 INFO BlockManagerInfo: Added broadcast_490_piece0 in memory on ip-172-19-105-117.ec2.internal:45766 (size: 19.6 KB, free: 1930.4 MB)
      18/04/01 15:06:35 INFO TaskSetManager: Starting task 29.0 in stage 28.1 (TID 5510, ip-172-19-110-149.ec2.internal, partition 509,PROCESS_LOCAL, 2449 bytes)
      18/04/01 15:06:35 INFO TaskSetManager: Finished task 17.0 in stage 28.1 (TID 5498) in 1434 ms on ip-172-19-110-149.ec2.internal (18/38)
      18/04/01 15:06:35 INFO BlockManagerInfo: Added broadcast_510_piece0 in memory on ip-172-19-110-149.ec2.internal:64342 (size: 19.6 KB, free: 1928.2 MB)
      18/04/01 15:06:35 INFO BlockManagerInfo: Added rdd_661_315 in memory on ip-172-19-110-176.ec2.internal:48439 (size: 1524.8 KB, free: 1934.9 MB)
      18/04/01 15:06:35 INFO BlockManagerInfo: Added rdd_661_205 in memory on ip-172-19-103-87.ec2.internal:13372 (size: 1541.2 KB, free: 1981.4 MB)
      18/04/01 15:06:35 INFO BlockManagerInfo: Added rdd_661_360 in memory on ip-172-19-103-249.ec2.internal:41486 (size: 18.0 KB, free: 1937.9 MB)
      18/04/01 15:06:35 INFO TaskSetManager: Starting task 30.0 in stage 28.1 (TID 5511, ip-172-19-103-249.ec2.internal, partition 525,PROCESS_LOCAL, 2449 bytes)
      18/04/01 15:06:35 INFO TaskSetManager: Finished task 21.0 in stage 28.1 (TID 5502) in 1402 ms on ip-172-19-103-249.ec2.internal (19/38)
      18/04/01 15:06:35 INFO TaskSetManager: Starting task 31.0 in stage 28.1 (TID 5512, ip-172-19-110-176.ec2.internal, partition 545,PROCESS_LOCAL, 2449 bytes)
      18/04/01 15:06:35 INFO TaskSetManager: Finished task 18.0 in stage 28.1 (TID 5499) in 1737 ms on ip-172-19-110-176.ec2.internal (20/38)
      18/04/01 15:06:35 INFO BlockManagerInfo: Added broadcast_546_piece0 in memory on ip-172-19-110-176.ec2.internal:48439 (size: 19.6 KB, free: 1934.9 MB)
      18/04/01 15:06:35 INFO BlockManagerInfo: Added broadcast_526_piece0 in memory on ip-172-19-103-249.ec2.internal:41486 (size: 19.6 KB, free: 1937.9 MB)
      18/04/01 15:06:36 INFO BlockManagerInfo: Added rdd_661_379 in memory on ip-172-19-108-144.ec2.internal:17672 (size: 18.6 KB, free: 1931.4 MB)
      18/04/01 15:06:36 INFO BlockManagerInfo: Added rdd_661_76 in memory on ip-172-19-103-87.ec2.internal:62681 (size: 1538.6 KB, free: 1930.7 MB)
      18/04/01 15:06:36 INFO TaskSetManager: Starting task 32.0 in stage 28.1 (TID 5513, ip-172-19-108-144.ec2.internal, partition 562,PROCESS_LOCAL, 2449 bytes)
      18/04/01 15:06:36 INFO TaskSetManager: Finished task 22.0 in stage 28.1 (TID 5503) in 928 ms on ip-172-19-108-144.ec2.internal (21/38)
      18/04/01 15:06:36 INFO BlockManagerInfo: Added broadcast_563_piece0 in memory on ip-172-19-108-144.ec2.internal:17672 (size: 19.6 KB, free: 1931.4 MB)
      18/04/01 15:06:36 INFO BlockManagerInfo: Added rdd_661_396 in memory on ip-172-19-104-236.ec2.internal:52270 (size: 15.4 KB, free: 1931.3 MB)
      18/04/01 15:06:36 INFO TaskSetManager: Starting task 33.0 in stage 28.1 (TID 5514, ip-172-19-104-236.ec2.internal, partition 575,PROCESS_LOCAL, 2449 bytes)
      18/04/01 15:06:36 INFO TaskSetManager: Finished task 23.0 in stage 28.1 (TID 5504) in 933 ms on ip-172-19-104-236.ec2.internal (22/38)
      18/04/01 15:06:36 INFO BlockManagerInfo: Added rdd_661_420 in memory on ip-172-19-105-221.ec2.internal:52999 (size: 20.7 KB, free: 1931.3 MB)
      18/04/01 15:06:36 INFO TaskSetManager: Starting task 34.0 in stage 28.1 (TID 5515, ip-172-19-105-221.ec2.internal, partition 586,PROCESS_LOCAL, 2449 bytes)
      18/04/01 15:06:36 INFO TaskSetManager: Finished task 24.0 in stage 28.1 (TID 5505) in 906 ms on ip-172-19-105-221.ec2.internal (23/38)
      18/04/01 15:06:36 INFO BlockManagerInfo: Added broadcast_587_piece0 in memory on ip-172-19-105-221.ec2.internal:52999 (size: 19.6 KB, free: 1931.3 MB)
      18/04/01 15:06:36 INFO TaskSetManager: Starting task 35.0 in stage 28.1 (TID 5516, ip-172-19-103-87.ec2.internal, partition 602,PROCESS_LOCAL, 2449 bytes)
      18/04/01 15:06:36 INFO TaskSetManager: Finished task 4.0 in stage 28.1 (TID 5485) in 3604 ms on ip-172-19-103-87.ec2.internal (24/38)
      18/04/01 15:06:36 INFO BlockManagerInfo: Added broadcast_603_piece0 in memory on ip-172-19-103-87.ec2.internal:62681 (size: 19.6 KB, free: 1930.7 MB)
      18/04/01 15:06:36 INFO BlockManagerInfo: Added rdd_661_453 in memory on ip-172-19-110-243.ec2.internal:14787 (size: 17.6 KB, free: 1940.9 MB)
      18/04/01 15:06:36 INFO BlockManagerInfo: Added rdd_661_437 in memory on ip-172-19-107-198.ec2.internal:11244 (size: 17.5 KB, free: 1936.7 MB)
      18/04/01 15:06:36 INFO BlockManagerInfo: Added rdd_661_474 in memory on ip-172-19-101-76.ec2.internal:16864 (size: 15.8 KB, free: 1929.6 MB)
      18/04/01 15:06:36 INFO TaskSetManager: Starting task 36.0 in stage 28.1 (TID 5517, ip-172-19-110-243.ec2.internal, partition 619,PROCESS_LOCAL, 2449 bytes)
      18/04/01 15:06:36 INFO TaskSetManager: Finished task 26.0 in stage 28.1 (TID 5507) in 1127 ms on ip-172-19-110-243.ec2.internal (25/38)
      18/04/01 15:06:36 INFO TaskSetManager: Starting task 37.0 in stage 28.1 (TID 5518, ip-172-19-107-198.ec2.internal, partition 635,PROCESS_LOCAL, 2449 bytes)
      18/04/01 15:06:36 INFO TaskSetManager: Finished task 25.0 in stage 28.1 (TID 5506) in 1193 ms on ip-172-19-107-198.ec2.internal (26/38)
      18/04/01 15:06:36 INFO BlockManagerInfo: Added broadcast_620_piece0 in memory on ip-172-19-110-243.ec2.internal:14787 (size: 19.6 KB, free: 1940.8 MB)
      18/04/01 15:06:36 INFO TaskSetManager: Finished task 27.0 in stage 28.1 (TID 5508) in 1013 ms on ip-172-19-101-76.ec2.internal (27/38)
      18/04/01 15:06:36 INFO BlockManagerInfo: Added broadcast_636_piece0 in memory on ip-172-19-107-198.ec2.internal:11244 (size: 19.6 KB, free: 1936.7 MB)
      18/04/01 15:06:36 INFO BlockManagerInfo: Added rdd_661_509 in memory on ip-172-19-110-149.ec2.internal:64342 (size: 19.3 KB, free: 1928.2 MB)
      18/04/01 15:06:36 INFO TaskSetManager: Finished task 29.0 in stage 28.1 (TID 5510) in 1093 ms on ip-172-19-110-149.ec2.internal (28/38)
      18/04/01 15:06:36 INFO TaskSetManager: Finished task 11.0 in stage 28.1 (TID 5492) in 3993 ms on ip-172-19-103-87.ec2.internal (29/38)
      18/04/01 15:06:36 INFO BlockManagerInfo: Added rdd_661_562 in memory on ip-172-19-108-144.ec2.internal:17672 (size: 18.2 KB, free: 1931.4 MB)
      18/04/01 15:06:36 INFO TaskSetManager: Finished task 32.0 in stage 28.1 (TID 5513) in 867 ms on ip-172-19-108-144.ec2.internal (30/38)
      18/04/01 15:06:36 INFO BlockManagerInfo: Added rdd_661_545 in memory on ip-172-19-110-176.ec2.internal:48439 (size: 16.4 KB, free: 1934.9 MB)
      18/04/01 15:06:36 INFO TaskSetManager: Finished task 31.0 in stage 28.1 (TID 5512) in 1111 ms on ip-172-19-110-176.ec2.internal (31/38)
      18/04/01 15:06:37 INFO BlockManagerInfo: Added rdd_661_575 in memory on ip-172-19-104-236.ec2.internal:52270 (size: 17.2 KB, free: 1931.3 MB)
      18/04/01 15:06:37 INFO TaskSetManager: Finished task 33.0 in stage 28.1 (TID 5514) in 899 ms on ip-172-19-104-236.ec2.internal (32/38)
      18/04/01 15:06:37 INFO BlockManagerInfo: Added rdd_661_489 in memory on ip-172-19-105-117.ec2.internal:45766 (size: 18.5 KB, free: 1930.4 MB)
      18/04/01 15:06:37 INFO TaskSetManager: Finished task 28.0 in stage 28.1 (TID 5509) in 1781 ms on ip-172-19-105-117.ec2.internal (33/38)
      18/04/01 15:06:37 INFO BlockManagerInfo: Added rdd_661_586 in memory on ip-172-19-105-221.ec2.internal:52999 (size: 20.0 KB, free: 1931.3 MB)
      18/04/01 15:06:37 INFO TaskSetManager: Finished task 34.0 in stage 28.1 (TID 5515) in 1197 ms on ip-172-19-105-221.ec2.internal (34/38)
      18/04/01 15:06:37 INFO BlockManagerInfo: Added rdd_661_525 in memory on ip-172-19-103-249.ec2.internal:41486 (size: 19.0 KB, free: 1937.8 MB)
      18/04/01 15:06:37 INFO TaskSetManager: Finished task 30.0 in stage 28.1 (TID 5511) in 1587 ms on ip-172-19-103-249.ec2.internal (35/38)
      18/04/01 15:06:37 INFO BlockManagerInfo: Added rdd_661_602 in memory on ip-172-19-103-87.ec2.internal:62681 (size: 18.1 KB, free: 1930.7 MB)
      18/04/01 15:06:37 INFO TaskSetManager: Finished task 35.0 in stage 28.1 (TID 5516) in 1276 ms on ip-172-19-103-87.ec2.internal (36/38)
      18/04/01 15:06:37 INFO BlockManagerInfo: Added rdd_661_635 in memory on ip-172-19-107-198.ec2.internal:11244 (size: 16.9 KB, free: 1936.7 MB)
      18/04/01 15:06:37 INFO TaskSetManager: Finished task 37.0 in stage 28.1 (TID 5518) in 1085 ms on ip-172-19-107-198.ec2.internal (37/38)
      18/04/01 15:06:38 INFO BlockManagerInfo: Added rdd_661_619 in memory on ip-172-19-110-243.ec2.internal:14787 (size: 18.8 KB, free: 1940.8 MB)
      18/04/01 15:06:38 INFO TaskSetManager: Finished task 36.0 in stage 28.1 (TID 5517) in 1577 ms on ip-172-19-110-243.ec2.internal (38/38)
      18/04/01 15:06:38 INFO DAGScheduler: ShuffleMapStage 28 (cache at IdspAggregator.scala:214) finished in 5.427 s
      18/04/01 15:06:38 INFO TaskSchedulerImpl: Removed TaskSet 28.1, whose tasks have all completed, from pool
      18/04/01 15:06:38 INFO DAGScheduler: looking for newly runnable stages
      18/04/01 15:06:38 INFO DAGScheduler: running: Set()
      18/04/01 15:06:38 INFO DAGScheduler: waiting: Set(ResultStage 30, ShuffleMapStage 29)
      18/04/01 15:06:38 INFO DAGScheduler: failed: Set()
      18/04/01 15:06:38 INFO DAGScheduler: Submitting ShuffleMapStage 29 (MapPartitionsRDD[745] at mapPartitions at RedshiftWriter.scala:237), which has no missing parents
      18/04/01 15:06:38 INFO BlockManagerInfo: Removed broadcast_663_piece0 on 172.19.103.84:18128 in memory (size: 4.0 MB, free: 496.3 MB)
      18/04/01 15:06:38 INFO BlockManagerInfo: Removed broadcast_663_piece1 on 172.19.103.84:18128 in memory (size: 2.5 MB, free: 498.8 MB)
      18/04/01 15:06:38 INFO BlockManagerInfo: Removed broadcast_663_piece0 on ip-172-19-103-87.ec2.internal:62681 in memory (size: 4.0 MB, free: 1934.7 MB)
      18/04/01 15:06:38 INFO BlockManagerInfo: Removed broadcast_663_piece0 on ip-172-19-101-76.ec2.internal:16864 in memory (size: 4.0 MB, free: 1933.6 MB)
      18/04/01 15:06:38 INFO BlockManagerInfo: Removed broadcast_663_piece0 on ip-172-19-100-11.ec2.internal:24171 in memory (size: 4.0 MB, free: 1939.0 MB)
      18/04/01 15:06:38 INFO BlockManagerInfo: Removed broadcast_663_piece0 on ip-172-19-102-6.ec2.internal:28761 in memory (size: 4.0 MB, free: 1944.4 MB)
      18/04/01 15:06:38 INFO BlockManagerInfo: Removed broadcast_663_piece0 on ip-172-19-110-176.ec2.internal:48439 in memory (size: 4.0 MB, free: 1938.9 MB)
      18/04/01 15:06:38 INFO BlockManagerInfo: Removed broadcast_663_piece0 on ip-172-19-103-249.ec2.internal:41486 in memory (size: 4.0 MB, free: 1941.8 MB)
      18/04/01 15:06:38 INFO BlockManagerInfo: Removed broadcast_663_piece0 on ip-172-19-108-144.ec2.internal:17672 in memory (size: 4.0 MB, free: 1935.4 MB)
      18/04/01 15:06:38 INFO BlockManagerInfo: Removed broadcast_663_piece0 on ip-172-19-110-243.ec2.internal:14787 in memory (size: 4.0 MB, free: 1944.8 MB)
      18/04/01 15:06:38 INFO BlockManagerInfo: Removed broadcast_663_piece0 on ip-172-19-104-236.ec2.internal:52270 in memory (size: 4.0 MB, free: 1935.3 MB)
      18/04/01 15:06:38 INFO BlockManagerInfo: Removed broadcast_663_piece0 on ip-172-19-105-117.ec2.internal:45766 in memory (size: 4.0 MB, free: 1934.4 MB)
      18/04/01 15:06:38 INFO BlockManagerInfo: Removed broadcast_663_piece1 on ip-172-19-100-11.ec2.internal:24171 in memory (size: 2.5 MB, free: 1941.5 MB)
      18/04/01 15:06:38 INFO BlockManagerInfo: Removed broadcast_663_piece0 on ip-172-19-107-199.ec2.internal:35094 in memory (size: 4.0 MB, free: 1942.9 MB)
      18/04/01 15:06:38 INFO BlockManagerInfo: Removed broadcast_663_piece0 on ip-172-19-105-221.ec2.internal:52999 in memory (size: 4.0 MB, free: 1935.3 MB)
      18/04/01 15:06:38 INFO BlockManagerInfo: Removed broadcast_663_piece0 on ip-172-19-107-198.ec2.internal:11244 in memory (size: 4.0 MB, free: 1940.7 MB)
      18/04/01 15:06:38 INFO BlockManagerInfo: Removed broadcast_663_piece1 on ip-172-19-102-6.ec2.internal:28761 in memory (size: 2.5 MB, free: 1946.9 MB)
      18/04/01 15:06:38 INFO BlockManagerInfo: Removed broadcast_663_piece1 on ip-172-19-101-76.ec2.internal:16864 in memory (size: 2.5 MB, free: 1936.1 MB)
      18/04/01 15:06:38 INFO BlockManagerInfo: Removed broadcast_663_piece1 on ip-172-19-103-87.ec2.internal:62681 in memory (size: 2.5 MB, free: 1937.2 MB)
      18/04/01 15:06:38 INFO BlockManagerInfo: Removed broadcast_663_piece1 on ip-172-19-103-249.ec2.internal:41486 in memory (size: 2.5 MB, free: 1944.3 MB)
      18/04/01 15:06:38 INFO BlockManagerInfo: Removed broadcast_663_piece0 on ip-172-19-106-52.ec2.internal:65123 in memory (size: 4.0 MB, free: 1938.6 MB)
      18/04/01 15:06:38 INFO BlockManagerInfo: Removed broadcast_663_piece1 on ip-172-19-110-243.ec2.internal:14787 in memory (size: 2.5 MB, free: 1947.3 MB)
      18/04/01 15:06:38 INFO BlockManagerInfo: Removed broadcast_663_piece0 on ip-172-19-110-149.ec2.internal:64342 in memory (size: 4.0 MB, free: 1932.2 MB)
      18/04/01 15:06:38 INFO BlockManagerInfo: Removed broadcast_663_piece1 on ip-172-19-108-144.ec2.internal:17672 in memory (size: 2.5 MB, free: 1937.9 MB)
      18/04/01 15:06:38 INFO BlockManagerInfo: Removed broadcast_663_piece1 on ip-172-19-110-176.ec2.internal:48439 in memory (size: 2.5 MB, free: 1941.4 MB)
      18/04/01 15:06:38 INFO BlockManagerInfo: Removed broadcast_663_piece1 on ip-172-19-104-236.ec2.internal:52270 in memory (size: 2.5 MB, free: 1937.8 MB)
      18/04/01 15:06:38 INFO BlockManagerInfo: Removed broadcast_663_piece1 on ip-172-19-105-221.ec2.internal:52999 in memory (size: 2.5 MB, free: 1937.8 MB)
      18/04/01 15:06:38 INFO BlockManagerInfo: Removed broadcast_663_piece1 on ip-172-19-107-199.ec2.internal:35094 in memory (size: 2.5 MB, free: 1945.5 MB)
      18/04/01 15:06:38 INFO BlockManagerInfo: Removed broadcast_663_piece1 on ip-172-19-105-117.ec2.internal:45766 in memory (size: 2.5 MB, free: 1936.9 MB)
      18/04/01 15:06:38 INFO BlockManagerInfo: Removed broadcast_663_piece1 on ip-172-19-107-198.ec2.internal:11244 in memory (size: 2.5 MB, free: 1943.2 MB)
      18/04/01 15:06:38 INFO BlockManagerInfo: Removed broadcast_663_piece1 on ip-172-19-110-149.ec2.internal:64342 in memory (size: 2.5 MB, free: 1934.7 MB)
      18/04/01 15:06:38 INFO BlockManagerInfo: Removed broadcast_663_piece1 on ip-172-19-106-52.ec2.internal:65123 in memory (size: 2.5 MB, free: 1941.1 MB)
      18/04/01 15:06:38 INFO MemoryStore: Block broadcast_666 stored as values in memory (estimated size 9.9 MB, free 153.5 MB)
      18/04/01 15:06:38 INFO MemoryStore: Block broadcast_666_piece0 stored as bytes in memory (estimated size 4.0 MB, free 157.5 MB)
      18/04/01 15:06:38 INFO BlockManagerInfo: Added broadcast_666_piece0 in memory on 172.19.103.84:18128 (size: 4.0 MB, free: 494.8 MB)
      18/04/01 15:06:38 INFO MemoryStore: Block broadcast_666_piece1 stored as bytes in memory (estimated size 2.5 MB, free 160.0 MB)
      18/04/01 15:06:38 INFO BlockManagerInfo: Added broadcast_666_piece1 in memory on 172.19.103.84:18128 (size: 2.5 MB, free: 492.3 MB)
      18/04/01 15:06:38 INFO SparkContext: Created broadcast 666 from broadcast at DAGScheduler.scala:1006
      18/04/01 15:06:38 INFO DAGScheduler: Submitting 10 missing tasks from ShuffleMapStage 29 (MapPartitionsRDD[745] at mapPartitions at RedshiftWriter.scala:237)
      18/04/01 15:06:38 INFO TaskSchedulerImpl: Adding task set 29.1 with 10 tasks
      18/04/01 15:06:38 INFO TaskSetManager: Starting task 0.0 in stage 29.1 (TID 5519, ip-172-19-108-144.ec2.internal, partition 5,PROCESS_LOCAL, 2050 bytes)
      18/04/01 15:06:38 INFO TaskSetManager: Starting task 1.0 in stage 29.1 (TID 5520, ip-172-19-107-198.ec2.internal, partition 17,PROCESS_LOCAL, 2050 bytes)
      18/04/01 15:06:38 INFO TaskSetManager: Starting task 2.0 in stage 29.1 (TID 5521, ip-172-19-110-176.ec2.internal, partition 35,PROCESS_LOCAL, 2050 bytes)
      18/04/01 15:06:38 INFO TaskSetManager: Starting task 3.0 in stage 29.1 (TID 5522, ip-172-19-105-117.ec2.internal, partition 52,PROCESS_LOCAL, 2050 bytes)
      18/04/01 15:06:38 INFO TaskSetManager: Starting task 4.0 in stage 29.1 (TID 5523, ip-172-19-110-243.ec2.internal, partition 70,PROCESS_LOCAL, 2050 bytes)
      18/04/01 15:06:38 INFO TaskSetManager: Starting task 5.0 in stage 29.1 (TID 5524, ip-172-19-103-87.ec2.internal, partition 86,PROCESS_LOCAL, 2050 bytes)
      18/04/01 15:06:38 INFO TaskSetManager: Starting task 6.0 in stage 29.1 (TID 5525, ip-172-19-104-236.ec2.internal, partition 114,PROCESS_LOCAL, 2050 bytes)
      18/04/01 15:06:38 INFO TaskSetManager: Starting task 7.0 in stage 29.1 (TID 5526, ip-172-19-103-249.ec2.internal, partition 137,PROCESS_LOCAL, 2050 bytes)
      18/04/01 15:06:38 INFO TaskSetManager: Starting task 8.0 in stage 29.1 (TID 5527, ip-172-19-103-87.ec2.internal, partition 167,PROCESS_LOCAL, 2050 bytes)
      18/04/01 15:06:38 INFO TaskSetManager: Starting task 9.0 in stage 29.1 (TID 5528, ip-172-19-101-76.ec2.internal, partition 187,PROCESS_LOCAL, 2050 bytes)
      18/04/01 15:06:38 INFO BlockManagerInfo: Added broadcast_666_piece1 in memory on ip-172-19-101-76.ec2.internal:16864 (size: 2.5 MB, free: 1933.6 MB)
      18/04/01 15:06:38 INFO BlockManagerInfo: Added broadcast_666_piece1 in memory on ip-172-19-108-144.ec2.internal:17672 (size: 2.5 MB, free: 1935.4 MB)
      18/04/01 15:06:38 INFO BlockManagerInfo: Added broadcast_666_piece1 in memory on ip-172-19-110-243.ec2.internal:14787 (size: 2.5 MB, free: 1944.8 MB)
      18/04/01 15:06:38 INFO BlockManagerInfo: Added broadcast_666_piece0 in memory on ip-172-19-103-249.ec2.internal:41486 (size: 4.0 MB, free: 1940.3 MB)
      18/04/01 15:06:38 INFO BlockManagerInfo: Added broadcast_666_piece0 in memory on ip-172-19-103-87.ec2.internal:62681 (size: 4.0 MB, free: 1933.2 MB)
      18/04/01 15:06:38 INFO BlockManagerInfo: Added broadcast_666_piece1 in memory on ip-172-19-103-87.ec2.internal:13372 (size: 2.5 MB, free: 1978.9 MB)
      18/04/01 15:06:38 INFO BlockManagerInfo: Added broadcast_666_piece0 in memory on ip-172-19-107-198.ec2.internal:11244 (size: 4.0 MB, free: 1939.2 MB)
      18/04/01 15:06:38 INFO BlockManagerInfo: Added broadcast_666_piece0 in memory on ip-172-19-110-176.ec2.internal:48439 (size: 4.0 MB, free: 1937.4 MB)
      18/04/01 15:06:38 INFO BlockManagerInfo: Added broadcast_666_piece0 in memory on ip-172-19-101-76.ec2.internal:16864 (size: 4.0 MB, free: 1929.6 MB)
      18/04/01 15:06:38 INFO BlockManagerInfo: Added broadcast_666_piece0 in memory on ip-172-19-108-144.ec2.internal:17672 (size: 4.0 MB, free: 1931.4 MB)
      18/04/01 15:06:38 INFO BlockManagerInfo: Added broadcast_666_piece1 in memory on ip-172-19-103-249.ec2.internal:41486 (size: 2.5 MB, free: 1937.8 MB)
      18/04/01 15:06:38 INFO BlockManagerInfo: Added broadcast_666_piece0 in memory on ip-172-19-105-117.ec2.internal:45766 (size: 4.0 MB, free: 1932.9 MB)
      18/04/01 15:06:38 INFO BlockManagerInfo: Added broadcast_666_piece1 in memory on ip-172-19-103-87.ec2.internal:62681 (size: 2.5 MB, free: 1930.7 MB)
      18/04/01 15:06:38 INFO BlockManagerInfo: Added broadcast_666_piece1 in memory on ip-172-19-107-198.ec2.internal:11244 (size: 2.5 MB, free: 1936.7 MB)
      18/04/01 15:06:38 INFO BlockManagerInfo: Added broadcast_666_piece0 in memory on ip-172-19-103-87.ec2.internal:13372 (size: 4.0 MB, free: 1974.9 MB)
      18/04/01 15:06:38 INFO BlockManagerInfo: Added broadcast_666_piece1 in memory on ip-172-19-110-176.ec2.internal:48439 (size: 2.5 MB, free: 1934.9 MB)
      18/04/01 15:06:38 INFO BlockManagerInfo: Added broadcast_666_piece0 in memory on ip-172-19-110-243.ec2.internal:14787 (size: 4.0 MB, free: 1940.8 MB)
      18/04/01 15:06:38 INFO BlockManagerInfo: Added broadcast_666_piece1 in memory on ip-172-19-105-117.ec2.internal:45766 (size: 2.5 MB, free: 1930.4 MB)
      18/04/01 15:06:38 INFO BlockManagerInfo: Added broadcast_666_piece0 in memory on ip-172-19-104-236.ec2.internal:52270 (size: 4.0 MB, free: 1933.8 MB)
      18/04/01 15:06:38 INFO BlockManagerInfo: Added broadcast_666_piece1 in memory on ip-172-19-104-236.ec2.internal:52270 (size: 2.5 MB, free: 1931.3 MB)
      18/04/01 15:06:38 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 10 to ip-172-19-103-249.ec2.internal:57024
      18/04/01 15:06:38 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 10 to ip-172-19-110-176.ec2.internal:33639
      18/04/01 15:06:38 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 10 to ip-172-19-107-198.ec2.internal:30635
      18/04/01 15:06:38 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 10 is 44293 bytes
      18/04/01 15:06:38 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 10 to ip-172-19-108-144.ec2.internal:62978
      18/04/01 15:06:38 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 10 to ip-172-19-101-76.ec2.internal:44931
      18/04/01 15:06:38 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 10 to ip-172-19-110-243.ec2.internal:37638
      18/04/01 15:06:38 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 10 is 44293 bytes
      18/04/01 15:06:38 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 10 is 44293 bytes
      18/04/01 15:06:38 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 10 to ip-172-19-104-236.ec2.internal:59497
      18/04/01 15:06:38 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 10 to ip-172-19-103-87.ec2.internal:36114
      18/04/01 15:06:38 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 10 to ip-172-19-105-117.ec2.internal:13679
      18/04/01 15:06:38 INFO BlockManagerInfo: Added rdd_743_187 in memory on ip-172-19-101-76.ec2.internal:16864 (size: 756.8 KB, free: 1928.9 MB)
      18/04/01 15:06:38 INFO BlockManagerInfo: Added rdd_743_70 in memory on ip-172-19-110-243.ec2.internal:14787 (size: 748.7 KB, free: 1940.1 MB)
      18/04/01 15:06:38 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 10 to ip-172-19-103-87.ec2.internal:36668
      18/04/01 15:06:38 INFO BlockManagerInfo: Added rdd_743_137 in memory on ip-172-19-103-249.ec2.internal:41486 (size: 755.8 KB, free: 1937.1 MB)
      18/04/01 15:06:38 INFO TaskSetManager: Finished task 9.0 in stage 29.1 (TID 5528) in 445 ms on ip-172-19-101-76.ec2.internal (1/10)
      18/04/01 15:06:38 INFO BlockManagerInfo: Added rdd_743_5 in memory on ip-172-19-108-144.ec2.internal:17672 (size: 751.9 KB, free: 1930.7 MB)
      18/04/01 15:06:38 INFO TaskSetManager: Finished task 4.0 in stage 29.1 (TID 5523) in 465 ms on ip-172-19-110-243.ec2.internal (2/10)
      18/04/01 15:06:38 INFO TaskSetManager: Finished task 7.0 in stage 29.1 (TID 5526) in 477 ms on ip-172-19-103-249.ec2.internal (3/10)
      18/04/01 15:06:38 INFO TaskSetManager: Finished task 0.0 in stage 29.1 (TID 5519) in 484 ms on ip-172-19-108-144.ec2.internal (4/10)
      18/04/01 15:06:38 INFO BlockManagerInfo: Added rdd_743_17 in memory on ip-172-19-107-198.ec2.internal:11244 (size: 745.7 KB, free: 1936.0 MB)
      18/04/01 15:06:38 INFO BlockManagerInfo: Added rdd_743_114 in memory on ip-172-19-104-236.ec2.internal:52270 (size: 744.6 KB, free: 1930.6 MB)
      18/04/01 15:06:38 INFO TaskSetManager: Finished task 1.0 in stage 29.1 (TID 5520) in 526 ms on ip-172-19-107-198.ec2.internal (5/10)
      18/04/01 15:06:38 INFO BlockManagerInfo: Added rdd_743_35 in memory on ip-172-19-110-176.ec2.internal:48439 (size: 766.2 KB, free: 1934.1 MB)
      18/04/01 15:06:38 INFO TaskSetManager: Finished task 6.0 in stage 29.1 (TID 5525) in 549 ms on ip-172-19-104-236.ec2.internal (6/10)
      18/04/01 15:06:38 INFO TaskSetManager: Finished task 2.0 in stage 29.1 (TID 5521) in 561 ms on ip-172-19-110-176.ec2.internal (7/10)
      18/04/01 15:06:38 INFO BlockManagerInfo: Added rdd_743_86 in memory on ip-172-19-103-87.ec2.internal:62681 (size: 758.4 KB, free: 1930.0 MB)
      18/04/01 15:06:38 INFO TaskSetManager: Finished task 5.0 in stage 29.1 (TID 5524) in 638 ms on ip-172-19-103-87.ec2.internal (8/10)
      18/04/01 15:06:38 INFO BlockManagerInfo: Added rdd_743_52 in memory on ip-172-19-105-117.ec2.internal:45766 (size: 749.4 KB, free: 1929.7 MB)
      18/04/01 15:06:39 INFO TaskSetManager: Finished task 3.0 in stage 29.1 (TID 5522) in 689 ms on ip-172-19-105-117.ec2.internal (9/10)
      18/04/01 15:06:39 INFO BlockManagerInfo: Added rdd_743_167 in memory on ip-172-19-103-87.ec2.internal:13372 (size: 761.2 KB, free: 1974.2 MB)
      18/04/01 15:06:39 INFO TaskSetManager: Finished task 8.0 in stage 29.1 (TID 5527) in 1578 ms on ip-172-19-103-87.ec2.internal (10/10)
      18/04/01 15:06:39 INFO TaskSchedulerImpl: Removed TaskSet 29.1, whose tasks have all completed, from pool
      18/04/01 15:06:39 INFO DAGScheduler: ShuffleMapStage 29 (mapPartitions at RedshiftWriter.scala:237) finished in 1.579 s
      18/04/01 15:06:39 INFO DAGScheduler: looking for newly runnable stages
      18/04/01 15:06:39 INFO DAGScheduler: running: Set()
      18/04/01 15:06:39 INFO DAGScheduler: waiting: Set(ResultStage 30)
      18/04/01 15:06:39 INFO DAGScheduler: failed: Set()
      18/04/01 15:06:39 INFO DAGScheduler: Submitting ResultStage 30 (MapPartitionsRDD[749] at createDataFrame at RedshiftWriter.scala:275), which has no missing parents
      18/04/01 15:06:39 INFO MemoryStore: Block broadcast_667 stored as values in memory (estimated size 114.6 KB, free 160.1 MB)
      18/04/01 15:06:39 INFO MemoryStore: Block broadcast_667_piece0 stored as bytes in memory (estimated size 44.1 KB, free 160.2 MB)
      18/04/01 15:06:39 INFO BlockManagerInfo: Added broadcast_667_piece0 in memory on 172.19.103.84:18128 (size: 44.1 KB, free: 492.3 MB)
      18/04/01 15:06:39 INFO SparkContext: Created broadcast 667 from broadcast at DAGScheduler.scala:1006
      18/04/01 15:06:39 INFO DAGScheduler: Submitting 5 missing tasks from ResultStage 30 (MapPartitionsRDD[749] at createDataFrame at RedshiftWriter.scala:275)
      18/04/01 15:06:39 INFO TaskSchedulerImpl: Adding task set 30.1 with 5 tasks
      18/04/01 15:06:39 INFO TaskSetManager: Starting task 0.0 in stage 30.1 (TID 5529, ip-172-19-103-87.ec2.internal, partition 3,PROCESS_LOCAL, 2061 bytes)
      18/04/01 15:06:39 INFO TaskSetManager: Starting task 1.0 in stage 30.1 (TID 5530, ip-172-19-105-221.ec2.internal, partition 6,PROCESS_LOCAL, 2061 bytes)
      18/04/01 15:06:39 INFO TaskSetManager: Starting task 2.0 in stage 30.1 (TID 5531, ip-172-19-101-76.ec2.internal, partition 11,PROCESS_LOCAL, 2061 bytes)
      18/04/01 15:06:39 INFO TaskSetManager: Starting task 3.0 in stage 30.1 (TID 5532, ip-172-19-103-87.ec2.internal, partition 13,PROCESS_LOCAL, 2061 bytes)
      18/04/01 15:06:39 INFO TaskSetManager: Starting task 4.0 in stage 30.1 (TID 5533, ip-172-19-105-117.ec2.internal, partition 14,PROCESS_LOCAL, 2061 bytes)
      18/04/01 15:06:39 INFO BlockManagerInfo: Added broadcast_667_piece0 in memory on ip-172-19-101-76.ec2.internal:16864 (size: 44.1 KB, free: 1928.8 MB)
      18/04/01 15:06:39 INFO BlockManagerInfo: Added broadcast_667_piece0 in memory on ip-172-19-103-87.ec2.internal:62681 (size: 44.1 KB, free: 1929.9 MB)
      18/04/01 15:06:39 INFO BlockManagerInfo: Added broadcast_667_piece0 in memory on ip-172-19-105-221.ec2.internal:52999 (size: 44.1 KB, free: 1937.7 MB)
      18/04/01 15:06:39 INFO BlockManagerInfo: Added broadcast_667_piece0 in memory on ip-172-19-105-117.ec2.internal:45766 (size: 44.1 KB, free: 1929.6 MB)
      18/04/01 15:06:39 INFO BlockManagerInfo: Added broadcast_667_piece0 in memory on ip-172-19-103-87.ec2.internal:13372 (size: 44.1 KB, free: 1974.1 MB)
      18/04/01 15:06:39 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 11 to ip-172-19-101-76.ec2.internal:44931
      18/04/01 15:06:39 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 11 is 1229 bytes
      18/04/01 15:06:39 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 11 to ip-172-19-105-221.ec2.internal:16426
      18/04/01 15:06:39 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 11 to ip-172-19-103-87.ec2.internal:36114
      18/04/01 15:06:39 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 11 to ip-172-19-105-117.ec2.internal:13679
      18/04/01 15:06:39 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 11 to ip-172-19-103-87.ec2.internal:36668
      18/04/01 15:06:43 WARN TaskSetManager: Lost task 6.0 in stage 30.0 (TID 5470, ip-172-19-102-6.ec2.internal): org.apache.spark.SparkException: Task failed while writing rows.
          at org.apache.spark.sql.execution.datasources.DefaultWriterContainer.writeRows(WriterContainer.scala:272)
          at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1$$anonfun$apply$mcV$sp$3.apply(InsertIntoHadoopFsRelation.scala:150)
          at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1$$anonfun$apply$mcV$sp$3.apply(InsertIntoHadoopFsRelation.scala:150)
          at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
          at org.apache.spark.scheduler.Task.run(Task.scala:89)
          at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at java.lang.Thread.run(Thread.java:748)
      Caused by: org.apache.spark.shuffle.FetchFailedException: Failed to connect to ip-172-19-109-92.ec2.internal/172.19.109.92:22910
          at org.apache.spark.storage.ShuffleBlockFetcherIterator.throwFetchFailedException(ShuffleBlockFetcherIterator.scala:323)
          at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:300)
          at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:51)
          at scala.collection.Iterator$$anon$11.next(Iterator.scala:370)
          at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:396)
          at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:369)
          at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
          at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
          at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:369)
          at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:369)
          at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:369)
          at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:369)
          at org.apache.spark.sql.execution.datasources.DefaultWriterContainer.writeRows(WriterContainer.scala:262)
          ... 8 more
      Caused by: java.io.IOException: Failed to connect to ip-172-19-109-92.ec2.internal/172.19.109.92:22910
          at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:216)
          at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:167)
          at org.apache.spark.network.netty.NettyBlockTransferService$$anon$1.createAndStart(NettyBlockTransferService.scala:90)
          at org.apache.spark.network.shuffle.RetryingBlockFetcher.fetchAllOutstanding(RetryingBlockFetcher.java:140)
          at org.apache.spark.network.shuffle.RetryingBlockFetcher.access$200(RetryingBlockFetcher.java:43)
          at org.apache.spark.network.shuffle.RetryingBlockFetcher$1.run(RetryingBlockFetcher.java:170)
          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
          at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          ... 3 more
      Caused by: java.net.ConnectException: Connection refused: ip-172-19-109-92.ec2.internal/172.19.109.92:22910
          at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
          at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
          at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:224)
          at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:289)
          at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528)
          at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
          at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
          at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
          at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
          ... 1 more

      18/04/01 15:06:43 INFO TaskSetManager: Lost task 11.0 in stage 30.0 (TID 5475) on executor ip-172-19-100-11.ec2.internal: org.apache.spark.SparkException (Task failed while writing rows.) [duplicate 1]
      18/04/01 15:06:43 INFO TaskSetManager: Finished task 0.0 in stage 30.1 (TID 5529) in 3999 ms on ip-172-19-103-87.ec2.internal (1/5)
      18/04/01 15:06:44 INFO TaskSetManager: Lost task 3.0 in stage 30.0 (TID 5467) on executor ip-172-19-107-199.ec2.internal: org.apache.spark.SparkException (Task failed while writing rows.) [duplicate 2]
      18/04/01 15:06:44 INFO TaskSetManager: Finished task 4.0 in stage 30.1 (TID 5533) in 4322 ms on ip-172-19-105-117.ec2.internal (2/5)
      18/04/01 15:06:44 INFO TaskSetManager: Lost task 14.0 in stage 30.0 (TID 5478) on executor ip-172-19-106-52.ec2.internal: org.apache.spark.SparkException (Task failed while writing rows.) [duplicate 3]
      18/04/01 15:06:44 INFO TaskSchedulerImpl: Removed TaskSet 30.0, whose tasks have all completed, from pool
      18/04/01 15:06:44 INFO TaskSetManager: Finished task 2.0 in stage 30.1 (TID 5531) in 4742 ms on ip-172-19-101-76.ec2.internal (3/5)
      18/04/01 15:06:44 INFO TaskSetManager: Finished task 1.0 in stage 30.1 (TID 5530) in 4866 ms on ip-172-19-105-221.ec2.internal (4/5)
      18/04/01 15:06:46 INFO TaskSetManager: Finished task 3.0 in stage 30.1 (TID 5532) in 6694 ms on ip-172-19-103-87.ec2.internal (5/5)
      18/04/01 15:06:46 INFO TaskSchedulerImpl: Removed TaskSet 30.1, whose tasks have all completed, from pool
      18/04/01 15:06:46 INFO DAGScheduler: ResultStage 30 (save at RedshiftWriter.scala:278) finished in 6.694 s
      18/04/01 15:06:46 INFO DAGScheduler: Job 12 finished: save at RedshiftWriter.scala:278, took 33.277686 s
      18/04/01 15:07:10 INFO DefaultWriterContainer: Job job_201804011506_0000 committed.
      18/04/01 15:07:10 INFO AvroRelation: Listing s3a://xxxx/20180401150612/21d29cb0-67b1-41fa-9a3b-f0833d7254b1 on driver
      18/04/01 15:07:10 INFO AvroRelation: Listing s3a://xxxx/20180401150612/21d29cb0-67b1-41fa-9a3b-f0833d7254b1 on driver
      18/04/01 15:07:11 INFO RedshiftWriter: Loading new Redshift data to: "PUBLIC"."_staging_9299a65e4e8c4e54bc4cee4e3eef7df2"

      Attachments

        Activity

          People

            Unassigned Unassigned
            liutzvin Huiqiang Liu
            Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: