Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-17718

Hive on Spark Debugging Improvements

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • Spark
    • None

    Description

      There are multiple places where it is hard to debug HoS - e.g. the HoS Remote Driver and Client, the Spark RDD graph, etc.

      Attachments

        Issue Links

          1.
          HS2 Logs print unnecessary stack trace when HoS query is cancelled Sub-task Closed Sahil Takiar
          2.
          Explicitly check if the HoS Remote Driver has been lost in the RemoteSparkJobMonitor Sub-task Closed Sahil Takiar
          3.
          Make org.apache.hive.spark.client.rpc logging HoS specific and other logging cleanup Sub-task Resolved Sahil Takiar
          4.
          Use sessionId for HoS Remote Driver Client id Sub-task Closed Sahil Takiar
          5.
          Improving logging with HoS executors spend lots of time in GC Sub-task Closed Sahil Takiar
          6.
          Improve logging when HoS application is killed Sub-task Closed Sahil Takiar
          7.
          Improve Spark Debug RDD Graph Sub-task Closed Sahil Takiar
          8.
          Explain plan should show if a Map/Reduce Work is being cached Sub-task Open liyunzhang
          9.
          Display Session Id and Query Name / Id in Spark UI Sub-task Closed Sahil Takiar
          10.
          Reduce logging in HiveSparkClientFactory and RemoteHiveSparkClient Sub-task Closed Sahil Takiar
          11.
          Print out Spark Web UI URL to the console log Sub-task Closed Sahil Takiar
          12.
          Custom Hive on Spark Tab in Spark Web UI Sub-task Open Unassigned
          13.
          Add explain plan to Hive on Spark Web UI Sub-task Resolved Sahil Takiar
          14.
          Expose additional Spark metrics Sub-task Closed Sahil Takiar
          15.
          Print Spark metrics on console Sub-task Resolved Sahil Takiar
          16.
          Logged Spark Job Id contains a UUID instead of the actual id Sub-task Closed Sahil Takiar
          17.
          Printed state in RemoteSparkJobMonitor is ambiguous Sub-task Closed Sahil Takiar
          18.
          ErrorMsg.SPARK_JOB_MONITOR_TIMEOUT isn't formatted correctly Sub-task Closed Sahil Takiar
          19.
          SparkClientImpl usage of SessionState.LogHelper doesn't respect isSilent value Sub-task Closed Sahil Takiar
          20.
          Race condition in RemoteSparkJobMonitor Sub-task Open Sahil Takiar
          21.
          Integrate with Spark OutputMetrics Sub-task Resolved Sahil Takiar
          22.
          Hive logs in Spark Executor and Driver should show thread-id. Sub-task Open Unassigned
          23.
          SparkClientImpl swallows exception messages from the RemoteDriver Sub-task Closed Sahil Takiar
          24.
          Race condition during shutdown of RemoteDriver, error messages aren't always sent Sub-task Resolved Aihua Xu
          25.
          RemoteSparkJobMonitor failures are logged twice Sub-task Closed Bharath Krishna
          26.
          Differentiate errors that are thrown by Spark tasks Sub-task Resolved Sahil Takiar
          27.
          Improve SparkTask OOM Error Parsing Logic Sub-task Open Unassigned
          28.
          Better client logging when a HoS session can't be opened Sub-task Resolved Aihua Xu
          29.
          SparkClientImpl doesn't error out if spark-submit fails Sub-task Resolved Sahil Takiar
          30.
          SparkClientImpl should react to errors sent from the RemoteDriver Sub-task Open Unassigned
          31.
          Better console logging for lifecycle of a Spark job Sub-task Open Sahil Takiar
          32.
          Fix Spark config warnings Sub-task Resolved Bharath Krishna
          33.
          Improve Spark session id logging Sub-task Resolved Sahil Takiar
          34.
          Add units to displayed Spark metrics Sub-task Open Unassigned
          35.
          RemoteSparkJobStatus#getSparkJobInfo treats all exceptions as timeout errors Sub-task Resolved Aihua Xu
          36.
          Add extended query string to Spark job description Sub-task Resolved Sahil Takiar
          37.
          Add HoS support to progress bar on Beeline client Sub-task Resolved Bharath Krishna
          38.
          Organize Spark metrics into multiple groups Sub-task Open Unassigned
          39.
          Allow disabling of long running job on Hive On Spark On YARN Sub-task Resolved Sahil Takiar
          40.
          Create Docker env for running HoS locally Sub-task Open Aihua Xu
          41.
          SparkJobMonitor getReport doesn't print stage progress in order Sub-task Resolved Bharath Krishna
          42.
          Spark task logs print PLAN PATH excessive number of times Sub-task Resolved Bharath Krishna
          43.
          SparkClientImpl shouldn't name redirector thread "RemoteDriver" Sub-task Resolved Bharath Krishna
          44.
          RemoteSparkJobStatus#getSparkStageProgress inefficient implementation Sub-task Resolved Bharath Krishna
          45.
          Race condition when timeout task is invoked during SASL negotation Sub-task In Progress Aihua Xu
          46.
          RpcServer cancelTask log message is incorrect Sub-task Resolved Bharath Krishna
          47.
          Log message when spark-submit has completed Sub-task Resolved Bharath Krishna
          48.
          hive.spark.log.dir isn't honored for TestSparkCliDriver Sub-task Open Unassigned
          49.
          NPE in SparkTask#printConsoleMetrics Sub-task Open Unassigned
          50.
          Show the number of rows inserted when execution engine is Spark Sub-task Resolved Bharath Krishna
          51.
          Print root cause exception's toString() rather than getMessage() Sub-task Resolved Aihua Xu
          52.
          Propagate ExecutionExceptions from the driver thread to the client Sub-task Open Unassigned
          53.
          Re-add HIVE-19787: Log message when spark-submit has completed Sub-task Open Sahil Takiar
          54.
          Typo in MetricsCollection for OutputMetrics Sub-task Patch Available Adesh Kumar Rao
          55.
          Improve logging when HoS Driver is killed due to exceeding memory limits Sub-task Open Unassigned
          56.
          JobResultSerializer uses wrong registration id in KyroMessageCodec Sub-task Open Sahil Takiar
          57.
          SparkSubmitSparkClient#launchDriver should parse exceptions, not just errors Sub-task Resolved Bharath Krishna
          58.
          Improve record and memory usage logging in SparkRecordHandler Sub-task Resolved Bharath Krishna
          59.
          Remove 30m min value for hive.spark.session.timeout Sub-task Patch Available Unassigned
          60.
          SparkSession should be able to close a session while it is being opened Sub-task Open Antal Sinkovits
          61.
          Parse Spark error blacklist errors Sub-task Open Unassigned

          Activity

            People

              Unassigned Unassigned
              stakiar Sahil Takiar
              Votes:
              1 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated: