[HIVE-9040] Spark Memory can be formatted string [Spark Branch]

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.1.0
• Component/s: Spark
• Assignee: Jimmy Xiang
• Reporter: Brock Noland
• Created: 08/Dec/14 17:41
• Updated: 29/May/15 02:32
• Resolved: 09/Dec/14 00:48

Description

Here: https://github.com/apache/hive/blob/spark/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/session/SparkSessionImpl.java#L72

we call getInt on spark.executor.memory but this is a formatted string, example here: http://spark.apache.org/docs/1.0.1/configuration.html as such, I get:

2014-12-08 03:04:48,114 WARN  [HiveServer2-Handler-Pool: Thread-34]: spark.SetSparkReducerParallelism (SetSparkReducerParallelism.java:process(141)) - Failed to create spark client.
java.lang.NumberFormatException: For input string: "23000m"
        at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
        at java.lang.Integer.parseInt(Integer.java:492)
        at java.lang.Integer.parseInt(Integer.java:527)
        at scala.collection.immutable.StringLike$class.toInt(StringLike.scala:229)
        at scala.collection.immutable.StringOps.toInt(StringOps.scala:31)
        at org.apache.spark.SparkConf$$anonfun$getInt$2.apply(SparkConf.scala:184)
        at org.apache.spark.SparkConf$$anonfun$getInt$2.apply(SparkConf.scala:184)
        at scala.Option.map(Option.scala:145)
        at org.apache.spark.SparkConf.getInt(SparkConf.scala:184)
        at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.getMemoryAndCores(SparkSessionImpl.java:72)