Details
-
Question
-
Status: Resolved
-
Major
-
Resolution: Invalid
-
3.0.0
-
None
-
None
-
spark3.0 + Hadoop3 + yarn cluster
Description
hi, I want to run mang executors(for example, 2) in a server with only one GPU card,I tested spark3.0 in yarn cluster mode with the following config:
spark-shell --conf spark.executor.resource.gpu.amount=0.5
Then ,I find the following errors:
20/06/18 16:24:46 [main] ERROR SparkContext: Error initializing SparkContext.
java.lang.NumberFormatException: For input string: "0.5"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Integer.parseInt(Integer.java:580)
at java.lang.Integer.parseInt(Integer.java:615)
at scala.collection.immutable.StringLike.toInt(StringLike.scala:304)
at scala.collection.immutable.StringLike.toInt$(StringLike.scala:304)
at scala.collection.immutable.StringOps.toInt(StringOps.scala:33)
at org.apache.spark.resource.ResourceUtils$.parseResourceRequest(ResourceUtils.scala:142)
at org.apache.spark.resource.ResourceUtils$.$anonfun$parseAllResourceRequests$1(ResourceUtils.scala:159)
at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238)
at scala.collection.mutable.ArraySeq.foreach(ArraySeq.scala:75)
at scala.collection.TraversableLike.map(TraversableLike.scala:238)
at scala.collection.TraversableLike.map$(TraversableLike.scala:231)
at scala.collection.AbstractTraversable.map(Traversable.scala:108)
at org.apache.spark.resource.ResourceUtils$.parseAllResourceRequests(ResourceUtils.scala:159)
at org.apache.spark.SparkContext$.checkResourcesPerTask$1(SparkContext.scala:2773)
at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2921)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:528)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2555)
at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$1(SparkSession.scala:931)
at scala.Option.getOrElse(Option.scala:189)
。。。。
My question:for spark3.0 Can one gpu card support many executors ?
Thank you!