Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-12229

Custom script in query cannot be executed in yarn-cluster mode [Spark Branch].

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.1.0
    • Fix Version/s: spark-branch, 1.3.0, 2.0.0
    • Component/s: Spark
    • Labels:
      None

      Description

      Added one python script in the query and the python script cannot be found during execution in yarn-cluster mode.

      15/10/21 21:10:55 INFO exec.ScriptOperator: Executing [/usr/bin/python, q2-sessionize.py, 3600]
      15/10/21 21:10:55 INFO exec.ScriptOperator: tablename=null
      15/10/21 21:10:55 INFO exec.ScriptOperator: partname=null
      15/10/21 21:10:55 INFO exec.ScriptOperator: alias=null
      15/10/21 21:10:55 INFO spark.SparkRecordHandler: processing 10 rows: used memory = 324896224
      15/10/21 21:10:55 INFO exec.ScriptOperator: ErrorStreamProcessor calling reporter.progress()
      /usr/bin/python: can't open file 'q2-sessionize.py': [Errno 2] No such file or directory
      15/10/21 21:10:55 INFO exec.ScriptOperator: StreamThread OutputProcessor done
      15/10/21 21:10:55 INFO exec.ScriptOperator: StreamThread ErrorProcessor done
      15/10/21 21:10:55 INFO spark.SparkRecordHandler: processing 100 rows: used memory = 325619920
      15/10/21 21:10:55 ERROR exec.ScriptOperator: Error in writing to script: Stream closed
      15/10/21 21:10:55 INFO exec.ScriptOperator: The script did not consume all input data. This is considered as an error.
      15/10/21 21:10:55 INFO exec.ScriptOperator: set hive.exec.script.allow.partial.consumption=true; to ignore it.
      15/10/21 21:10:55 ERROR spark.SparkReduceRecordHandler: Fatal error: org.apache.hadoop.hive.ql.metadata.HiveException: Error while processing row (tag=0) {"key":{"reducesinkkey0":2,"reducesinkkey1":3316240655},"value":{"_col0":5529}}
      org.apache.hadoop.hive.ql.metadata.HiveException: Error while processing row (tag=0) {"key":{"reducesinkkey0":2,"reducesinkkey1":3316240655},"value":{"_col0":5529}}
              at org.apache.hadoop.hive.ql.exec.spark.SparkReduceRecordHandler.processKeyValues(SparkReduceRecordHandler.java:340)
              at org.apache.hadoop.hive.ql.exec.spark.SparkReduceRecordHandler.processRow(SparkReduceRecordHandler.java:289)
              at org.apache.hadoop.hive.ql.exec.spark.HiveReduceFunctionResultList.processNextRecord(HiveReduceFunctionResultList.java:49)
              at org.apache.hadoop.hive.ql.exec.spark.HiveReduceFunctionResultList.processNextRecord(HiveReduceFunctionResultList.java:28)
              at org.apache.hadoop.hive.ql.exec.spark.HiveBaseFunctionResultList$ResultIterator.hasNext(HiveBaseFunctionResultList.java:95)
              at scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:41)
              at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.insertAll(BypassMergeSortShuffleWriter.java:99)
              at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:73)
              at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
              at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
              at org.apache.spark.scheduler.Task.run(Task.scala:88)
              at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
              at java.lang.Thread.run(Thread.java:745)
      Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: [Error 20001]: An error occurred while reading or writing to your custom script. It may have crashed with an error.
              at org.apache.hadoop.hive.ql.exec.ScriptOperator.processOp(ScriptOperator.java:453)
              at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
              at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
              at org.apache.hadoop.hive.ql.exec.spark.SparkReduceRecordHandler.processKeyValues(SparkReduceRecordHandler.java:331)
              ... 14 more
      

        Attachments

        1. HIVE-12229.3-spark.patch
          11 kB
          Rui Li
        2. HIVE-12229.3-spark.patch
          11 kB
          Rui Li
        3. HIVE-12229.2-spark.patch
          4 kB
          Rui Li
        4. HIVE-12229.1-spark.patch
          2 kB
          Rui Li

          Activity

            People

            • Assignee:
              lirui Rui Li
              Reporter:
              Lifeng Wang Lifeng Wang
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: