Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-17718

Hive on Spark Debugging Improvements

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • Spark
    • None

    Description

      There are multiple places where it is hard to debug HoS - e.g. the HoS Remote Driver and Client, the Spark RDD graph, etc.

      Attachments

        Issue Links

          is related to

          Improvement - An improvement or enhancement to an existing feature or task. HIVE-18533 Add option to use InProcessLauncher to submit spark jobs

          • Major - Major loss of function.
          • Closed
          1.
          		 HS2 Logs print unnecessary stack trace when HoS query is cancelled Sub-task Closed Sahil Takiar
          2.
          		 Explicitly check if the HoS Remote Driver has been lost in the RemoteSparkJobMonitor Sub-task Closed Sahil Takiar
          3.
          		 Make org.apache.hive.spark.client.rpc logging HoS specific and other logging cleanup Sub-task Closed Sahil Takiar
          4.
          		 Use sessionId for HoS Remote Driver Client id Sub-task Closed Sahil Takiar
          5.
          		 Improving logging with HoS executors spend lots of time in GC Sub-task Closed Sahil Takiar
          6.
          		 Improve logging when HoS application is killed Sub-task Closed Sahil Takiar
          7.
          		 Improve Spark Debug RDD Graph Sub-task Closed Sahil Takiar
          8.
          		 Explain plan should show if a Map/Reduce Work is being cached Sub-task Open liyunzhang
          9.
          		 Display Session Id and Query Name / Id in Spark UI Sub-task Closed Sahil Takiar
          10.
          		 Reduce logging in HiveSparkClientFactory and RemoteHiveSparkClient Sub-task Closed Sahil Takiar
          11.
          		 Print out Spark Web UI URL to the console log Sub-task Closed Sahil Takiar
          12.
          		 Custom Hive on Spark Tab in Spark Web UI Sub-task Open Unassigned
          13.
          		 Add explain plan to Hive on Spark Web UI Sub-task Closed Sahil Takiar
          14.
          		 Expose additional Spark metrics Sub-task Closed Sahil Takiar
          15.
          		 Print Spark metrics on console Sub-task Closed Sahil Takiar
          16.
          		 Logged Spark Job Id contains a UUID instead of the actual id Sub-task Closed Sahil Takiar
          17.
          		 Printed state in RemoteSparkJobMonitor is ambiguous Sub-task Closed Sahil Takiar
          18.
          		 ErrorMsg.SPARK_JOB_MONITOR_TIMEOUT isn't formatted correctly Sub-task Closed Sahil Takiar
          19.
          		 SparkClientImpl usage of SessionState.LogHelper doesn't respect isSilent value Sub-task Closed Sahil Takiar
          20.
          		 Race condition in RemoteSparkJobMonitor Sub-task Open Sahil Takiar
          21.
          		 Integrate with Spark OutputMetrics Sub-task Closed Sahil Takiar
          22.
          		 Hive logs in Spark Executor and Driver should show thread-id. Sub-task Open Unassigned
          23.
          		 SparkClientImpl swallows exception messages from the RemoteDriver Sub-task Closed Sahil Takiar
          24.
          		 Race condition during shutdown of RemoteDriver, error messages aren't always sent Sub-task Closed Aihua Xu
          25.
          		 RemoteSparkJobMonitor failures are logged twice Sub-task Closed Bharath Krishna
          26.
          		 Differentiate errors that are thrown by Spark tasks Sub-task Closed Sahil Takiar
          27.
          		 Improve SparkTask OOM Error Parsing Logic Sub-task Open Unassigned
          28.
          		 Better client logging when a HoS session can't be opened Sub-task Closed Aihua Xu
          29.
          		 SparkClientImpl doesn't error out if spark-submit fails Sub-task Closed Sahil Takiar
          30.
          		 SparkClientImpl should react to errors sent from the RemoteDriver Sub-task Open Unassigned
          31.
          		 Better console logging for lifecycle of a Spark job Sub-task Open Sahil Takiar
          32.
          		 Fix Spark config warnings Sub-task Closed Bharath Krishna
          33.
          		 Improve Spark session id logging Sub-task Closed Sahil Takiar
          34.
          		 Add units to displayed Spark metrics Sub-task Open Unassigned
          35.
          		 RemoteSparkJobStatus#getSparkJobInfo treats all exceptions as timeout errors Sub-task Closed Aihua Xu
          36.
          		 Add extended query string to Spark job description Sub-task Closed Sahil Takiar
          37.
          		 Add HoS support to progress bar on Beeline client Sub-task Closed Bharath Krishna
          38.
          		 Organize Spark metrics into multiple groups Sub-task Open Unassigned
          39.
          		 Allow disabling of long running job on Hive On Spark On YARN Sub-task Closed Sahil Takiar
          40.
          		 Create Docker env for running HoS locally Sub-task Open Aihua Xu
          41.
          		 SparkJobMonitor getReport doesn't print stage progress in order Sub-task Closed Bharath Krishna
          42.
          		 Spark task logs print PLAN PATH excessive number of times Sub-task Closed Bharath Krishna
          43.
          		 SparkClientImpl shouldn't name redirector thread "RemoteDriver" Sub-task Closed Bharath Krishna
          44.
          		 RemoteSparkJobStatus#getSparkStageProgress inefficient implementation Sub-task Closed Bharath Krishna
          45.
          		 Race condition when timeout task is invoked during SASL negotation Sub-task In Progress Aihua Xu
          46.
          		 RpcServer cancelTask log message is incorrect Sub-task Closed Bharath Krishna
          47.
          		 Log message when spark-submit has completed Sub-task Closed Bharath Krishna
          48.
          		 hive.spark.log.dir isn't honored for TestSparkCliDriver Sub-task Open Unassigned
          49.
          		 NPE in SparkTask#printConsoleMetrics Sub-task Open Unassigned
          50.
          		 Show the number of rows inserted when execution engine is Spark Sub-task Closed Bharath Krishna
          51.
          		 Print root cause exception's toString() rather than getMessage() Sub-task Closed Aihua Xu
          52.
          		 Propagate ExecutionExceptions from the driver thread to the client Sub-task Open Unassigned
          53.
          		 Re-add HIVE-19787: Log message when spark-submit has completed Sub-task Open Sahil Takiar
          54.
          		 Typo in MetricsCollection for OutputMetrics Sub-task Patch Available Adesh Kumar Rao
          55.
          		 Improve logging when HoS Driver is killed due to exceeding memory limits Sub-task Open Unassigned
          56.
          		 JobResultSerializer uses wrong registration id in KyroMessageCodec Sub-task Open Sahil Takiar
          57.
          		 SparkSubmitSparkClient#launchDriver should parse exceptions, not just errors Sub-task Closed Bharath Krishna
          58.
          		 Improve record and memory usage logging in SparkRecordHandler Sub-task Closed Bharath Krishna
          59.
          		 Remove 30m min value for hive.spark.session.timeout Sub-task Patch Available Unassigned
          60.
          		 SparkSession should be able to close a session while it is being opened Sub-task Open Antal Sinkovits
          61.
          		 Parse Spark error blacklist errors Sub-task Open Unassigned

          Activity

            People

              Unassigned Unassigned
              stakiar Sahil Takiar
              Votes:
              1 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated: