Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-7292 Hive on Spark
  3. HIVE-9040

Spark Memory can be formatted string [Spark Branch]

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.1.0
    • Component/s: Spark
    • Labels:
      None

      Description

      Here: https://github.com/apache/hive/blob/spark/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/session/SparkSessionImpl.java#L72

      we call getInt on spark.executor.memory but this is a formatted string, example here: http://spark.apache.org/docs/1.0.1/configuration.html as such, I get:

      2014-12-08 03:04:48,114 WARN  [HiveServer2-Handler-Pool: Thread-34]: spark.SetSparkReducerParallelism (SetSparkReducerParallelism.java:process(141)) - Failed to create spark client.
      java.lang.NumberFormatException: For input string: "23000m"
              at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
              at java.lang.Integer.parseInt(Integer.java:492)
              at java.lang.Integer.parseInt(Integer.java:527)
              at scala.collection.immutable.StringLike$class.toInt(StringLike.scala:229)
              at scala.collection.immutable.StringOps.toInt(StringOps.scala:31)
              at org.apache.spark.SparkConf$$anonfun$getInt$2.apply(SparkConf.scala:184)
              at org.apache.spark.SparkConf$$anonfun$getInt$2.apply(SparkConf.scala:184)
              at scala.Option.map(Option.scala:145)
              at org.apache.spark.SparkConf.getInt(SparkConf.scala:184)
              at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.getMemoryAndCores(SparkSessionImpl.java:72)
      

        Attachments

        1. HIVE-9040.3-spark.patch
          6 kB
          Jimmy Xiang
        2. HIVE-9040.2-spark.patch
          1 kB
          Jimmy Xiang
        3. HIVE-9040.1-spark.patch
          1 kB
          Jimmy Xiang

          Activity

            People

            • Assignee:
              jxiang Jimmy Xiang
              Reporter:
              brocknoland Brock Noland
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: