Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-7292 Hive on Spark
  3. HIVE-9882

Add jar/file doesn't work with yarn-cluster mode [Spark Branch]

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: spark-branch
    • Fix Version/s: 1.2.0
    • Component/s: Hive, Spark
    • Labels:
      None

      Description

      It seems current fix for HIVE-9425 only uploads the Jar/Files to HDFS, however, they are not accessible by the Driver/Executor.
      I found below in the AM log:

      15/02/26 15:10:36 INFO Configuration.deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
      15/02/26 15:10:36 INFO client.SparkClientUtilities: Added jar[file:/data/hadoop-devel/data/nm/usercache/user/appcache/application_1424933948132_0002/container_1424933948132_0002_01_000001/hdfs:/localhost:8020/tmp/hive/user/47040bca-1da4-49b6-b2c7-69be9bc92855/hive-exec-1.2.0-SNAPSHOT.jar] to classpath.
      15/02/26 15:10:36 INFO client.SparkClientUtilities: Added jar[file:/data/hadoop-devel/data/nm/usercache/user/appcache/application_1424933948132_0002/container_1424933948132_0002_01_000001/hdfs:/localhost:8020/tmp/hive/user/47040bca-1da4-49b6-b2c7-69be9bc92855/opennlp-maxent-3.0.3.jar] to classpath.
      15/02/26 15:10:36 INFO client.SparkClientUtilities: Added jar[file:/data/hadoop-devel/data/nm/usercache/user/appcache/application_1424933948132_0002/container_1424933948132_0002_01_000001/hdfs:/localhost:8020/tmp/hive/user/47040bca-1da4-49b6-b2c7-69be9bc92855/bigbenchqueriesmr.jar] to classpath.
      15/02/26 15:10:36 INFO client.SparkClientUtilities: Added jar[file:/data/hadoop-devel/data/nm/usercache/user/appcache/application_1424933948132_0002/container_1424933948132_0002_01_000001/hdfs:/localhost:8020/tmp/hive/user/47040bca-1da4-49b6-b2c7-69be9bc92855/opennlp-tools-1.5.3.jar] to classpath.
      15/02/26 15:10:36 INFO client.SparkClientUtilities: Added jar[file:/data/hadoop-devel/data/nm/usercache/user/appcache/application_1424933948132_0002/container_1424933948132_0002_01_000001/hdfs:/localhost:8020/tmp/hive/user/47040bca-1da4-49b6-b2c7-69be9bc92855/jcl-over-slf4j-1.7.5.jar] to classpath.
      15/02/26 15:10:36 INFO client.RemoteDriver: Failed to run job 6886df05-f430-456c-a0ff-c7621db712d6
      org.apache.hive.com.esotericsoftware.kryo.KryoException: Unable to find class: de.bankmark.bigbench.queries.q10.SentimentUDF 
      

      As above shows, the file path which was attempted to add to Classpath is invalid, so actually all uploaded Jars/Files are still not available for Driver/Executor.

        Attachments

        1. HIVE-9882.1-spark.patch
          9 kB
          Rui Li
        2. HIVE-9882.1-spark.patch
          9 kB
          Rui Li

          Issue Links

            Activity

              People

              • Assignee:
                lirui Rui Li
                Reporter:
                zhos Xiaomin Zhang
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: