Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-18750

spark should be able to control the number of executor and should not throw stack overslow

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.3, 2.1.1, 2.2.0
    • Component/s: None
    • Labels:
      None

      Description

      When running Sql queries on large datasets. Job fails with stack overflow warning and it shows it is requesting lots of executors.

      Looks like there is no limit to number of executors or not even having an upperbound based on yarn available resources.

      16/11/29 15:47:47 INFO impl.ContainerManagementProtocolProxy: Opening proxy : bdtcstr61n5.svr.us.jpmchase.net:8041 
      16/11/29 15:47:47 INFO impl.ContainerManagementProtocolProxy: Opening proxy : bdtcstr61n8.svr.us.jpmchase.net:8041 
      16/11/29 15:47:47 INFO impl.ContainerManagementProtocolProxy: Opening proxy : bdtcstr61n2.svr.us.jpmchase.net:8041 
      16/11/29 15:47:47 INFO yarn.YarnAllocator: Driver requested a total number of 32770 executor(s). 
      16/11/29 15:47:47 INFO yarn.YarnAllocator: Will request 24576 executor containers, each with 1 cores and 6758 MB memory including 614 MB overhead 
      16/11/29 15:49:11 INFO yarn.YarnAllocator: Driver requested a total number of 52902 executor(s). 
      16/11/29 15:47:47 INFO impl.ContainerManagementProtocolProxy: Opening proxy : bdtcstr61n5.svr.us.jpmchase.net:8041
      16/11/29 15:47:47 INFO impl.ContainerManagementProtocolProxy: Opening proxy : bdtcstr61n8.svr.us.jpmchase.net:8041
      16/11/29 15:47:47 INFO impl.ContainerManagementProtocolProxy: Opening proxy : bdtcstr61n2.svr.us.jpmchase.net:8041
      16/11/29 15:47:47 INFO yarn.YarnAllocator: Driver requested a total number of 32770 executor(s).
      16/11/29 15:47:47 INFO yarn.YarnAllocator: Will request 24576 executor containers, each with 1 cores and 6758 MB memory including 614 MB overhead
      16/11/29 15:49:11 INFO yarn.YarnAllocator: Driver requested a total number of 52902 executor(s).
      16/11/29 15:49:11 WARN yarn.ApplicationMaster: Reporter thread fails 1 time(s) in a row.
      java.lang.StackOverflowError
      	at scala.collection.immutable.HashMap.$plus(HashMap.scala:57)
      	at scala.collection.immutable.HashMap.$plus(HashMap.scala:36)
      	at scala.collection.mutable.MapBuilder.$plus$eq(MapBuilder.scala:28)
      	at scala.collection.mutable.MapBuilder.$plus$eq(MapBuilder.scala:24)
      	at scala.collection.generic.Growable$$anonfun$$plus$plus$eq$1.apply(Growable.scala:48)
      	at scala.collection.generic.Growable$$anonfun$$plus$plus$eq$1.apply(Growable.scala:48)
      	at scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:224)
      	at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:403)
      	at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:403)
      	at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
      	at scala.collection.mutable.MapBuilder.$plus$plus$eq(MapBuilder.scala:24)
      	at scala.collection.TraversableLike$class.$plus$plus(TraversableLike.scala:156)
      	at scala.collection.AbstractTraversable.$plus$plus(Traversable.scala:105)
      	at scala.collection.immutable.HashMap.$plus(HashMap.scala:60)
      	at scala.collection.immutable.Map$Map4.updated(Map.scala:172)
      	at scala.collection.immutable.Map$Map4.$plus(Map.scala:173)
      	at scala.collection.immutable.Map$Map4.$plus(Map.scala:158)
      	at scala.collection.mutable.MapBuilder.$plus$eq(MapBuilder.scala:28)
      	at scala.collection.mutable.MapBuilder.$plus$eq(MapBuilder.scala:24)
      	at scala.collection.TraversableLike$$anonfun$filter$1.apply(TraversableLike.scala:264)
      	at scala.collection.MapLike$MappedValues$$anonfun$foreach$3.apply(MapLike.scala:245)
      	at scala.collection.MapLike$MappedValues$$anonfun$foreach$3.apply(MapLike.scala:245)
      	at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
      	at scala.collection.MapLike$MappedValues$$anonfun$foreach$3.apply(MapLike.scala:245)
      	at scala.collection.MapLike$MappedValues$$anonfun$foreach$3.apply(MapLike.scala:245)
      	at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
      	at scala.collection.MapLike$MappedValues$$anonfun$foreach$3.apply(MapLike.scala:245)
      	at scala.collection.MapLike$MappedValues$$anonfun$foreach$3.apply(MapLike.scala:245)
      	at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
      	at scala.collection.MapLike$MappedValues$$anonfun$foreach$3.apply(MapLike.scala:245)
      	at scala.collection.MapLike$MappedValues$$anonfun$foreach$3.apply(MapLike.scala:245)
      	at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
      	at scala.collection.MapLike$MappedValues$$anonfun$foreach$3.apply(MapLike.scala:245)
      	at scala.collection.MapLike$MappedValues$$anonfun$foreach$3.apply(MapLike.scala:245)
      	at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
      	at scala.collection.MapLike$MappedValues$$anonfun$foreach$3.apply(MapLike.scala:245)
      	at scala.collection.MapLike$MappedValues$$anonfun$foreach$3.apply(MapLike.scala:245)
      	at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
      	at scala.collection.MapLike$MappedValues$$anonfun$foreach$3.apply(MapLike.scala:245)
      	at scala.collection.MapLike$MappedValues$$anonfun$foreach$3.apply(MapLike.scala:245)
      	at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
      	at scala.collection.MapLike$MappedValues$$anonfun$foreach$3.apply(MapLike.scala:245)
      	at scala.collection.MapLike$MappedValues$$anonfun$foreach$3.apply(MapLike.scala:245)
      	at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
      	at scala.collection.MapLike$MappedValues$$anonfun$foreach$3.apply(MapLike.scala:245)
      	at scala.collection.MapLike$MappedValues$$anonfun$foreach$3.apply(MapLike.scala:245)
      	at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
      [...]
      16/11/29 15:49:11 INFO yarn.YarnAllocator: Will request 44708 executor containers, each with 1 cores and 6758 MB memory including 614 MB overhead
      

      If you notice in the error above, YARN is trying to request 24576 executor containers, whereas the available cores are 1719. The Driver is requesting for 52902 executor(s), which too high.

      This exception should be fixed

        Attachments

          Activity

            People

            • Assignee:
              vanzin Marcelo Masiero Vanzin
              Reporter:
              neerjakhattar Neerja Khattar
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: