[HIVE-17718] Hive on Spark Debugging Improvements

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Spark
• Target Version/s: 3.0.0
• Assignee: Unassigned Assign to me
• Reporter: Sahil Takiar
• Created: 06/Oct/17 01:57
• Updated: 18/Dec/18 06:41

Description

There are multiple places where it is hard to debug HoS - e.g. the HoS Remote Driver and Client, the Spark RDD graph, etc.

Issue Links

• is related to: HIVE-18533 Add option to use InProcessLauncher to submit spark jobs

Sub-Tasks

• Explain plan should show if a Map/Reduce Work is being cached: Open
• Custom Hive on Spark Tab in Spark Web UI: Open
• Race condition in RemoteSparkJobMonitor: Open
• Hive logs in Spark Executor and Driver should show thread-id.: Open
• Improve SparkTask OOM Error Parsing Logic: Open
• SparkClientImpl should react to errors sent from the RemoteDriver: Open
• Better console logging for lifecycle of a Spark job: Open
• Add units to displayed Spark metrics: Open
• Organize Spark metrics into multiple groups: Open
• Create Docker env for running HoS locally: Open
• Race condition when timeout task is invoked during SASL negotation: In Progress
• hive.spark.log.dir isn't honored for TestSparkCliDriver: Open
• NPE in SparkTask#printConsoleMetrics: Open
• Propagate ExecutionExceptions from the driver thread to the client: Open
• Re-add HIVE-19787: Log message when spark-submit has completed: Open
• Typo in MetricsCollection for OutputMetrics: Patch Available
• Improve logging when HoS Driver is killed due to exceeding memory limits: Open
• JobResultSerializer uses wrong registration id in KyroMessageCodec: Open
• Remove 30m min value for hive.spark.session.timeout: Patch Available
• SparkSession should be able to close a session while it is being opened: Open
• Parse Spark error blacklist errors: Open