[HIVE-7292] Hive on Spark

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Done
• Fix Version/s: 1.1.0
• Component/s: Spark
• Labels: Spark-M1 Spark-M2 Spark-M3 Spark-M4 Spark-M5
• Assignee: Xuefu Zhang Assign to me
• Reporter: Xuefu Zhang
• Created: 25/Jun/14 19:56
• Updated: 19/Oct/21 12:21
• Resolved: 11/Sep/17 21:43

Description

Spark as an open-source data analytics cluster computing framework has gained significant momentum recently. Many Hive users already have Spark installed as their computing backbone. To take advantages of Hive, they still need to have either MapReduce or Tez on their cluster. This initiative will provide user a new alternative so that those user can consolidate their backend.

Secondly, providing such an alternative further increases Hive's adoption as it exposes Spark users to a viable, feature-rich de facto standard SQL tools on Hadoop.

Finally, allowing Hive to run on Spark also has performance benefits. Hive queries, especially those involving multiple reducer stages, will run faster, thus improving user experience as Tez does.

This is an umbrella JIRA which will cover many coming subtask. Design doc will be attached here shortly, and will be on the wiki as well. Feedback from the community is greatly appreciated!

Issue Links

• contains: HIVE-7717 Add .q tests coverage for "union all" [Spark Branch]
• contains: HIVE-7767 hive.optimize.union.remove does not work properly [Spark Branch]
• contains: HIVE-8242 Investigate test failures when hive.multigroupby.singlereducer and hive.optimize.multigroupby.common.distincts are set to false [Spark Branch]
• contains: HIVE-7745 NullPointerException when turn on hive.optimize.union.remove, hive.merge.mapfiles and hive.merge.mapredfiles [Spark Branch]
• contains: HIVE-8216 auto_smb_mapjoin_14.q failed test with exception. [Spark Branch]
• contains: HIVE-8233 multi-table insertion doesn't work with ForwardOperator [Spark Branch]
• contains: HIVE-8496 Re-enable statistics [Spark Branch]
• contains: HIVE-8542 Enable groupby_map_ppr.q and groupby_map_ppr_multi_distinct.q [Spark Branch]
• contains: HIVE-8545 Exception when casting Text to BytesWritable [Spark Branch]
• contains: HIVE-8208 Multi-table insertion optimization #1: don't always break operator tree. [Spark Branch]
• contains: HIVE-8215 Multi-table insertion optimization #3: use 1+1 tasks instead of 1+N tasks [Spark Branch]
• contains: HIVE-8249 Refactoring SparkPlan and SparkPlanGenerator [Spark Branch]
• contains: HIVE-8209 Multi-table insertion optimization #2: use separate context [Spark Branch]
• contains: HIVE-8463 Add numPartitions info to SparkEdgeProperty [Spark Branch]
• contains: HIVE-8207 Add .q tests for multi-table insertion [Spark Branch]
• contains: HIVE-8430 Enable parquet_join.q [Spark Branch]
• contains: HIVE-8431 Enable smb_mapjoin_11.q and smb_mapjoin_12.q [Spark Branch]
• contains: HIVE-8533 Enable all q-tests for multi-insertion [Spark Branch]
• contains: HIVE-7437 Check if servlet-api and jetty module in Spark library are an issue for hive-spark integration [Spark Branch]
• contains: HIVE-7939 Refactoring GraphTran to make it conform to SparkTran interface. [Spark Branch]
• depends upon: SPARK-2633 enhance spark listener API to gather more spark job information
• depends upon: SPARK-2636 Expose job ID in JobWaiter API
• depends upon: SPARK-2895 Support mapPartitionsWithContext in Spark Java API
• depends upon: SPARK-2421 Spark should treat writable as serializable for keys
• depends upon: SPARK-4290 Provide an equivalent functionality of distributed cache as MR does
• depends upon: SPARK-2420 Dependency changes for compatibility with Hive
• incorporates: HIVE-7773 Union all query finished with errors [Spark Branch]
• incorporates: HIVE-7525 Research to find out if it's possible to submit Spark jobs concurrently using shared SparkContext [Spark Branch]
• incorporates: HIVE-7728 Enable q-tests for TABLESAMPLE feature [Spark Branch]
• incorporates: HIVE-7729 Enable q-tests for ANALYZE TABLE feature [Spark Branch]
• incorporates: HIVE-7731 Incorrect result returned when a map work has multiple downstream reduce works [Spark Branch]
• incorporates: HIVE-7775 enable sample8.q.[Spark Branch]
• incorporates: HIVE-7776 enable sample10.q.[Spark Branch]
• incorporates: HIVE-7810 Insert overwrite table query has strange behavior when set hive.optimize.union.remove=true [Spark Branch]
• incorporates: HIVE-7870 Insert overwrite table query does not generate correct task plan [Spark Branch]
• incorporates: HIVE-8054 Disable hive.optimize.union.remove when hive.execution.engine=spark [Spark Branch]
• incorporates: HIVE-8055 Code cleanup after HIVE-8054 [Spark Branch]
• incorporates: HIVE-7613 Research optimization of auto convert join to map join [Spark branch]
• incorporates: HIVE-7746 Cleanup SparkClient and make refreshLocalResources method synchronized [Spark Branch]
• incorporates: HIVE-7372 Select query gives unpredictable incorrect result when parallelism is greater than 1 [Spark Branch]
• incorporates: HIVE-7387 Guava version conflict between hadoop and spark [Spark-Branch]
• incorporates: HIVE-7431 When run on spark cluster, some spark tasks may fail
• incorporates: HIVE-7467 When querying HBase table, task fails with exception: java.lang.IllegalAccessError: com/google/protobuf/HBaseZeroCopyByteString
• incorporates: HIVE-7489 Change ql/pom.xml to fix mvn project setup [Spark Branch]
• incorporates: HIVE-7530 Go thru the common code to find references to HIVE_EXECUCTION_ENGINE to make sure conditions works with Spark [Spark Branch]
• incorporates: HIVE-7540 NotSerializableException encountered when using sortByKey transformation
• incorporates: HIVE-7556 Fix code style, license header, tabs, etc. [Spark Branch]
• incorporates: HIVE-7624 Reduce operator initialization failed when running multiple MR query on spark
• incorporates: HIVE-7626 Add jar through CLI did not loaded by Spark executor[Spark Branck]
• incorporates: HIVE-7627 FSStatsPublisher does fit into Spark multi-thread task mode[Spark Branch]
• incorporates: HIVE-7642 Set hive input format by configuration.[Spark Branch]
• incorporates: HIVE-7643 ExecMapper static states lead to unpredictable query result.[Spark Branch]
• incorporates: HIVE-7747 Submitting a query to Spark from HiveServer2 fails [Spark Branch]
• incorporates: HIVE-7761 Failed to analyze stats with CounterStatsAggregator [SparkBranch]
• incorporates: HIVE-7763 Failed to query TABLESAMPLE on empty bucket table [Spark Branch]
• incorporates: HIVE-7780 Query with OVER clause return duplicate results[Spark Branch]
• incorporates: HIVE-7795 Enable ptf.q and ptf_streaming.q.[Spark Branch]
• incorporates: HIVE-7799 TRANSFORM failed in transform_ppr1.q[Spark Branch]
• incorporates: HIVE-7909 Fix sample8.q automatic test failure[Spark Branch]
• incorporates: HIVE-7916 Snappy-java error when running hive query on spark [Spark Branch]
• incorporates: HIVE-7956 When inserting into a bucketed table, all data goes to a single bucket [Spark Branch]
• incorporates: HIVE-8118 Support work that have multiple child works to work around SPARK-3622 [Spark Branch]
• incorporates: HIVE-8300 Missing guava lib causes IllegalStateException when deserializing a task [Spark Branch]
• incorporates: HIVE-8426 paralle.q assert failed.[Spark Branch]
• incorporates: HIVE-8548 Integrate with remote Spark context after HIVE-8528 [Spark Branch]
• incorporates: HIVE-7551 expand spark accumulator to support hive counter [Spark Branch]
• incorporates: HIVE-7552 Collect spark job statistic through spark metrics [Spark Branch]
• incorporates: HIVE-8699 Enable support for common map join [Spark Branch]
• incorporates: HIVE-7516 Add capacity control over queries running on Spark cluster [Spark Branch]
• incorporates: HIVE-7564 Remove some redundant code plus a bit of cleanup in SparkClient [Spark Branch]
• incorporates: HIVE-7659 Unnecessary sort in query plan [Spark Branch]
• incorporates: HIVE-7707 Optimize SparkMapRecordHandler implementation
• incorporates: HIVE-7726 Refactor to reuse common logic between SparkMapRecordHandler and ExecMapper.
• incorporates: HIVE-7727 Refactor to reuse common logic between SparkReduceRecordHandler and ExecReducer
• incorporates: HIVE-8029 Remove reducers number configure in SparkTask [Spark Branch]
• incorporates: HIVE-8219 Multi-Insert optimization, don't sink the source into a file [Spark Branch]
• incorporates: HIVE-8220 Refactor multi-insert code such that plan splitting and task generation are modular and reusable [Spark Branch]
• incorporates: HIVE-7772 Add tests for order/sort/distribute/cluster by query [Spark Branch]
• incorporates: HIVE-8098 The spark golden file for union_remove_25 is different from MR version [Spark Branch]
• incorporates: HIVE-7370 Initial ground work for Hive on Spark [Spark branch]
• incorporates: HIVE-7371 Identify a minimum set of JARs needed to ship to Spark cluster [Spark Branch]
• incorporates: HIVE-7526 Research to use groupby transformation to replace Hive existing partitionByKey and SparkCollector combination
• incorporates: HIVE-7567 support automatic calculating reduce task number [Spark Branch]
• incorporates: HIVE-8024 Find out whether it's possible to remove UnionOperator from original operator tree [Spark Branch]
• incorporates: HIVE-8043 Support merging small files [Spark Branch]
• incorporates: HIVE-8274 Refactoring SparkPlan and SparkPlanGeneration [Spark Branch]
• incorporates: HIVE-8537 Update to use the stable TaskContext API [Spark Branch]
• incorporates: HIVE-7893 Find a way to get a job identifier when submitting a spark job [Spark Branch]
• incorporates: HIVE-8073 Go thru all operator plan optimizations and disable those that are not suitable for Spark [Spark Branch]
• is blocked by: SPARK-2243 Support multiple SparkContexts in the same JVM
• is related to: HIVE-9611 Allow SPARK_HOME as well as spark.home to define sparks location
• is related to: HIVE-9134 Uber JIRA to track HOS performance work
• is related to: HIVE-9367 CombineFileInputFormatShim#getDirIndices is expensive
• is related to: SPARK-2741 Publish version of spark assembly which does not contain Hive
• relates to: HIVE-7607 Spark "Explain" should give useful information on dependencies [Spark Branch]
• requires: HIVE-7958 SparkWork generated by SparkCompiler may require multiple Spark jobs to run
• requires: SPARK-2688 Need a way to run multiple data pipeline concurrently
• requires: HIVE-7391 Refactoring TezWork/TezEdgeProperty for code reuse
• links to: https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark

Sub-Tasks

• Refactoring: make Hive reduce side data processing reusable [Spark Branch]: Reopened
• Refactoring: make Hive map side data processing reusable [Spark Branch]: Resolved
• Create SparkWork [Spark Branch]: Resolved
• Create SparkTask [Spark Branch]: Resolved
• Create SparkCompiler [Spark Branch]: Resolved
• Create SparkClient, interface to Spark cluster [Spark Branch]: Resolved
• Create RDD translator, translating Hive Tables into Spark RDDs [Spark Branch]: Resolved
• Create SparkShuffler, shuffling data between map-side data processing and reduce-side processing [Spark Branch]: Resolved
• Create SparkPlan, DAG representation of a Spark job [Spark Branch]: Resolved
• Create MapFunction [Spark Branch]: Resolved
• Create ReduceFunction [Spark Branch]: Resolved
• Create SparkPlanGenerator [Spark Branch]: Resolved
• Create a MiniSparkCluster and set up a testing framework [Spark Branch]: Resolved
• Research into reduce-side join [Spark Branch]: Resolved
• Spark 1.0.1 is released, stop using SNAPSHOT [Spark Branch]: Resolved
• Exclude hadoop 1 from spark dep [Spark Branch]: Resolved
• Load Spark configuration into Hive driver [Spark Branch]: Resolved
• Counters, statistics, and metrics [Spark Branch]: Resolved
• Spark job monitoring and error reporting [Spark Branch]: Resolved
• Implement pre-commit testing [Spark Branch]: Resolved
• Enhance SparkCollector [Spark Branch]: Resolved
• Enhance HiveReduceFunction's row clustering [Spark Branch]: Resolved
• Support Hive's multi-table insert query with Spark [Spark Branch]: Resolved
• Support order by and sort by on Spark [Spark Branch]: Resolved
• Support cluster by and distributed by [Spark Branch]: Resolved
• Support union all on Spark [Spark Branch]: Resolved
• StarterProject: Move configuration from SparkClient to HiveConf [Spark Branch]: Open
• StarterProject: Fix exception handling in POC code [Spark Branch]: Resolved
• StarterProject: Move from assert to Guava Preconditions.* in Hive on Spark [Spark Branch]: Resolved
• Make sure multi-MR queries work [Spark Branch]: Resolved
• Support dynamic partitioning [Spark Branch]: Resolved
• Instantiate SparkClient per user session [Spark Branch]: Resolved
• Support analyze table [Spark Branch]: Resolved
• Find solution for closures containing writables [Spark Branch]: Resolved
• Support Hive TABLESAMPLE [Spark Branch]: Resolved
• Create TestSparkCliDriver to run test in spark local mode [Spark Branch]: Resolved
• Update to Spark 1.2 [Spark Branch]: Resolved
• Implement native HiveMapFunction [Spark Branch]: Resolved
• Implement native HiveReduceFunction [Spark Branch]: Resolved
• Start running .q file tests on spark [Spark Branch]: Resolved
• Fix qtest-spark pom.xml reference to test properties [Spark Branch]: Resolved
• Create SparkReporter [Spark Branch]: Resolved
• Incorrect result returned when a map work has multiple downstream reduce works [Spark Branch]: Resolved
• TestSparkCliDriver should not use includeQueryFiles [Spark Branch]: Resolved
• Add .q tests coverage for "union all" [Spark Branch]: Resolved
• Enable q-tests for TABLESAMPLE feature [Spark Branch]: Resolved
• Research to find out if it's possible to submit Spark jobs concurrently using shared SparkContext [Spark Branch]: Resolved
• Enable q-tests for ANALYZE TABLE feature [Spark Branch]: Resolved
• Add qfile_regex to qtest-spark pom [Spark Branch]: Resolved
• Enable timestamp.* tests [Spark Branch]: Resolved
• Enable avro* tests [Spark Branch]: Resolved
• PTest2 separates test files with spaces while QTestGen uses commas [Spark Branch]: Resolved
• Cleanup Reduce operator code [Spark Branch]: Resolved
• hive.optimize.union.remove does not work properly [Spark Branch]: Resolved
• Integrate with Spark executor scaling [Spark Branch]: Resolved
• Research optimization of auto convert join to map join [Spark branch]: Resolved
• Support windowing and analytic functions [Spark Branch]: Resolved
• Enable windowing and analytic function qtests [Spark Branch]: Resolved
• Union all query finished with errors [Spark Branch]: Resolved
• Enable tests on Spark branch (1) [Sparch Branch]: Resolved
• Enable tests on Spark branch (2) [Sparch Branch]: Resolved
• Enable tests on Spark branch (3) [Sparch Branch]: Resolved
• Enable tests on Spark branch (4) [Sparch Branch]: Resolved
• Enable map-join tests which Tez executes [Spark Branch]: Resolved
• CounterStatsAggregator throws a class cast exception: Resolved
• union_null.q is not deterministic: Closed
• StarterProject: enable groupby4.q [Spark Branch]: Resolved
• Research commented out unset in Utiltities [Spark Branch]: Resolved
• Update union_null results now that it's deterministic [Spark Branch]: Resolved
• Refresh SparkContext when spark configuration changes [Spark Branch]: Resolved
• Enable reduce-side join tests (1) [Spark Branch]: Resolved
• Merge from trunk (1) [Spark Branch]: Resolved
• Re-order spark.query.files in sorted order [Spark Branch]: Resolved
• Build long running HS2 test framework: Closed
• Insert overwrite table query has strange behavior when set hive.optimize.union.remove=true [Spark Branch]: Resolved
• Re-enable lazy HiveBaseFunctionResultList [Spark Branch]: Resolved
• Enable qtest load_dyn_part1.q [Spark Branch]: Resolved
• orc_analyze.q fails due to random mapred.task.id in FileSinkOperator [Spark Branch]: Resolved
• optimize_nullscan.q fails due to differences in explain plan [Spark Branch]: Resolved
• Support multiple concurrent users: Resolved
• Support subquery [Spark Branch]: Resolved
• enable Qtest scriptfile1.q [Spark Branch]: Resolved
• enable sample8.q.[Spark Branch]: Resolved
• enable sample10.q.[Spark Branch]: Resolved
• Insert overwrite table query does not generate correct task plan [Spark Branch]: Resolved
• Research Hive dependency on MR distributed cache[Spark Branch]: Open
• Merge from trunk (2) [Spark Branch]: Resolved
• Investigate query failures (1): Resolved
• Investigate query failures (2): Resolved
• Investigate query failures (3): Resolved
• Investigate query failures (4): Resolved
• Merge from trunk (3) [Spark Branch]: Resolved
• Use HiveKey instead of BytesWritable as key type of the pair RDD [Spark Branch]: Resolved
• Fix TestSparkCliDriver => optimize_nullscan.q: Resolved
• Merge trunk into spark 9/12/2014: Resolved
• Enable vectorization for spark [spark branch]: Resolved
• Code cleanup after HIVE-8054 [Spark Branch]: Resolved
• Disable hive.optimize.union.remove when hive.execution.engine=spark [Spark Branch]: Resolved
• Remove obsolete code from SparkWork [Spark Branch]: Resolved
• Refactor the GraphTran code by moving union handling logic to UnionTran [Spark Branch]: Resolved
• Support SMB Join for Hive on Spark [Spark Branch]: Resolved
• Merge from trunk to spark 9/20/14: Resolved
• clone SparkWork for join optimization: Resolved
• GroupByShuffler.java missing apache license header [Spark Branch]: Resolved
• Merge from trunk to spark 9/29/14: Resolved
• Enable windowing.q for spark [Spark Branch]: Resolved
• Merge trunk into spark 10/4/2015 [Spark Branch]: Resolved
• Fix fs_default_name2.q on spark [Spark Branch]: Resolved
• Investigate flaky test parallel.q: Resolved
• TPCDS query #7 fails with IndexOutOfBoundsException [Spark Branch]: Resolved
• Research Bucket Map Join [Spark Branch]: Resolved
• Research on skewed join [Spark Branch]: Resolved
• Make reduce side join work for all join queries [Spark Branch]: Resolved
• Turn on all join .q tests [Spark Branch]: Resolved
• Print Spark job progress format info on the console[Spark Branch]: Resolved
• Support Hive Counter to collect spark job metric[Spark Branch]: Resolved
• Update timestamp in status console [Spark Branch]: Resolved
• TPC-DS Query 96 parallelism is not set correcly: Resolved
• Merge trunk into spark 10/17/14 [Spark Branch]: Resolved
• UT: add TestSparkMinimrCliDriver to run UTs that use HDFS: Open
• UT: fix bucket_num_reducers test: Open
• UTs: create missing output files for some tests under clientpositive/spark: Open
• UT: add test flag in hive-site.xml for spark tests: Resolved
• UT: fix rcfile_bigdata test [Spark Branch]: Resolved
• UT: fix bucketsort_insert tests - related to SMBMapJoinOperator: Resolved
• UT: fix list_bucket_dml_2 test [Spark Branch]: Resolved
• Update async action in SparkClient as Spark add new Java action API[Spark Branch]: Resolved
• Add remote Spark client to Hive [Spark Branch]: Resolved
• Enable collect table statistics based on SparkCounter[Spark Branch]: Resolved
• HivePairFlatMapFunction.java missing license header [Spark Branch]: Resolved
• Add InterfaceAudience annotations to spark-client [Spark Branch]: Resolved
• convert joinOp to MapJoinOp and generate MapWorks only [Spark Branch]: Resolved
• Implement bucket map join optimization [Spark Branch]: Resolved
• Convert SMBJoin to MapJoin [Spark Branch]: Resolved
• Support hints of SMBJoin [Spark Branch]: Resolved
• Reduce Side Join with single reducer [Spark Branch]: Resolved
• Enable parallelism in Reduce Side Join [Spark Branch]: Resolved
• Increase level of parallelism in reduce phase [Spark Branch]: Resolved
• Combine Hive Operator statistic and Spark Metric to an uniformed query statistic.[Spark Branch]: Resolved
• Result differences after merge [Spark Branch]: Resolved
• Fix tests after merge [Spark Branch]: Resolved
• Enable table statistic collection on counter for CTAS query[Spark Branch]: Resolved
• spark-client build failed sometimes.[Spark Branch]: Resolved
• Collect Spark TaskMetrics and build job statistic[Spark Branch]: Resolved
• Null Pointer Exception when counter is used for stats during inserting overwrite partitioned tables [Spark Branch]: Resolved
• numRows and rawDataSize are not collected by the Spark stats [Spark Branch]: Resolved
• Investigate test failures related to HIVE-8545 [Spark Branch]: Resolved
• Fix hadoop-1 build [Spark Branch]: Resolved
• Merge from trunk 11/6/14 [SPARK BRANCH]: Resolved
• Should only register used counters in SparkCounters[Spark Branch]: Resolved
• insert1.q and ppd_join4.q hangs with hadoop-1 [Spark Branch]: Resolved
• Create some tests that use Spark counter for stats collection [Spark Branch]: Resolved
• UT: update hive-site.xml for spark UTs to add hive_admin_user to admin role: Resolved
• UT: fix partition test case [Spark Branch]: Resolved
• UT: fix udf_context_aware: Resolved
• UT: fix hook_context_cs test case: Open
• Switch precommit test from local to local-cluster [Spark Branch]: Resolved
• Print prettier Spark work graph after HIVE-8793 [Spark Branch]: Resolved
• Release RDD cache when Hive query is done [Spark Branch]: Resolved
• Choose a persisent policy for RDD caching [Spark Branch]: Resolved
• Hive/Spark/Yarn integration [Spark Branch]: Resolved
• Update new spark progress API for local submitted job monitoring [Spark Branch]: Resolved
• Visualize generated Spark plan [Spark Branch]: Closed
• Downgrade guava version to be consistent with Hive and the rest of Hadoop [Spark Branch]: Open
• Fix test TestHiveKVResultCache [Spark Branch]: Resolved
• Use MEMORY_AND_DISK for RDD caching [Spark Branch]: Resolved
• Merge from trunk to spark [Spark Branch]: Resolved
• downgrade guava version for spark branch from 14.0.1 to 11.0.2.[Spark Branch]: Resolved
• Servlet classes signer information does not match [Spark branch]: Resolved
• IOContext problem with multiple MapWorks cloned for multi-insert [Spark Branch]: Resolved
• Remove unnecessary dependency collection task [Spark Branch]: Resolved
• Make sure Spark + HS2 work [Spark Branch]: Resolved
• Merge from trunk Nov 28 2014: Resolved
• Find thread leak in RSC Tests [Spark Branch]: Resolved
• Logging is not configured in spark-submit sub-process: Resolved
• SparkCounter display name is not set correctly[Spark Branch]: Resolved
• Clean up temp files of RSC [Spark Branch]: Open
• Avoid using SPARK_JAVA_OPTS [Spark Branch]: Resolved
• Re-enable remaining tests after HIVE-8970 [Spark Branch]: Resolved
• Enable ppd_join4 [Spark Branch]: Resolved
• Replace akka for remote spark client RPC [Spark Branch]: Resolved
• Spark Memory can be formatted string [Spark Branch]: Resolved
• Support multiple mapjoin operators in one work [Spark Branch]: Resolved
• HiveException: Conflict on row inspector for {table}: Resolved
• Choosing right preference between map join and bucket map join [Spark Branch]: Open
• Add additional logging to SetSparkReducerParallelism [Spark Branch]: Resolved
• Remove wrappers for SparkJobInfo and SparkStageInfo [Spark Branch]: Resolved
• NPE in RemoteSparkJobStatus.getSparkStatistics [Spark Branch]: Resolved
• Generate better plan for queries containing both union and multi-insert [Spark Branch]: Resolved
• Allow RPC Configuration [Spark Branch]: Resolved
• Hive should not submit second SparkTask while previous one has failed.[Spark Branch]: Resolved
• Hive hangs while failed to get executorCount[Spark Branch]: Resolved
• Skip child tasks if parent task failed [Spark Branch]: Resolved
• Bucket mapjoin should use the new alias in posToAliasMap [Spark Branch]: Resolved
• Investigate IOContext object initialization problem [Spark Branch]: Resolved
• Spark Client RPC should have larger default max message size [Spark Branch]: Resolved
• Spark counter serialization error in spark.log [Spark Branch]: Resolved
• Error when cleaning up in spark.log [Spark Branch]: Open
• TimeoutException when trying get executor count from RSC [Spark Branch]: Resolved
• Check cross product for conditional task [Spark Branch]: Resolved
• infer_bucket_sort_convert_join.q and mapjoin_hook.q failed.[Spark Branch]: Resolved
• bucket_map_join_spark4.q failed due to NPE.[Spark Branch]: Resolved
• Support backup task for join related optimization [Spark Branch]: Patch Available
• windowing.q failed when mapred.reduce.tasks is set to larger than one: Resolved
• Add unit test for multi sessions.[Spark Branch]: Resolved
• Enable beeline query progress information for Spark job[Spark Branch]: Resolved
• RSC stdout is logged twice [Spark Branch]: Resolved
• Clean up GenSparkProcContext.clonedReduceSinks and related code [Spark Branch]: Closed
• authorization_admin_almighty1.q fails with result diff [Spark Branch]: Resolved
• Merge from trunk to spark 12/26/2014 [Spark Branch]: Resolved
• UT: set hive.support.concurrency to true for spark UTs: Open
• UT: udf_in_file fails with filenotfoundexception [Spark Branch]: Resolved
• Create a separate API for remote Spark Context RPC other than job submission [Spark Branch]: Resolved
• Add listeners on JobHandle so job status change can be notified to the client [Spark Branch]: Resolved
• TimeOutException when using RSC with beeline [Spark Branch]: Resolved
• One-pass SMB Optimizations [Spark Branch]: Resolved
• Choose Kryo as the serializer for pTest [Spark Branch]: Resolved
• Test windowing.q is failing [Spark Branch]: Resolved
• Add more log information for debug RSC[Spark Branch]: Resolved
• Spark branch compile failed on hadoop-1[Spark Branch]: Resolved
• Research on build mini HoS cluster on YARN for unit test[Spark Branch]: Resolved
• Remove authorization_admin_almighty1 from spark tests [Spark Branch]: Resolved
• Investigate differences for auto join tests in explain after merge from trunk [Spark Branch]: Resolved
• Followup for HIVE-9125, update ppd_join4.q.out for Spark [Spark Branch]: Resolved
• Remove tabs from spark code [Spark Branch]: Resolved
• SetSparkReducerParallelism is likely to set too small number of reducers [Spark Branch]: Resolved
• Merge trunk to spark 1/5/2015 [Spark Branch]: Resolved
• Merge from spark to trunk January 2015: Resolved
• Explain query should share the same Spark application with regular queries [Spark Branch]: Resolved
• Ensure custom UDF works with Spark [Spark Branch]: Resolved
• Code cleanup [Spark Branch]: Resolved
• TODO cleanup task1.[Spark Branch]: Resolved
• Cleanup code for getting spark job progress and metrics: Open
• Improve replication factor of small table file given big table partitions [Spark branch]: Open
• Set default miniClusterType back to none in QTestUtil.[Spark branch]: Resolved
• Let Context.isLocalOnlyExecutionMode() return false if execution engine is Spark [Spark Branch]: Resolved
• thrift.transport.TTransportException [Spark Branch]: Open
• Cleanup Modified Files [Spark Branch]: Resolved
• Merge from trunk to spark 1/8/2015: Resolved
• BaseProtocol.Error failed to deserialization due to NPE.[Spark Branch]: Resolved
• Address review items on HIVE-9257 [Spark Branch]: Resolved
• Optimize split grouping for CombineHiveInputFormat [Spark Branch]: Resolved
• Address review of HIVE-9257 (ii) [Spark Branch]: Resolved
• Fix windowing.q for Spark on trunk: Resolved
• Merge from spark to trunk (follow-up of HIVE-9257): Resolved
• SparkJobMonitor timeout as sortByKey would launch extra Spark job before original job get submitted [Spark Branch]: Resolved
• Fix tests with some versions of Spark + Snappy [Spark Branch]: Resolved
• add num-executors / executor-cores / executor-memory option support for hive on spark in Yarn mode [Spark Branch]: Resolved
• Shutting down cli takes quite some time [Spark Branch]: Resolved
• Make WAIT_SUBMISSION_TIMEOUT configuable and check timeout in SparkJobMonitor level.[Spark Branch]: Resolved
• Avoid ser/de loggers as logging framework can be incompatible on driver and workers: Resolved
• ClassNotFoundException occurs during hive query case execution with UDF defined [Spark Branch]: Resolved
• Add jar/file doesn't work with yarn-cluster mode [Spark Branch]: Resolved
• Merge trunk to spark 1/21/2015: Resolved
• Move more hive.spark.* configurations to HiveConf [Spark Branch]: Resolved
• LocalSparkJobStatus may return failed job as successful [Spark Branch]: Resolved
• Push YARN configuration to Spark while deply Spark on YARN [Spark Branch]: Resolved
• MapJoin task shouldn't start if HashTableSink task failed [Spark Branch]: Resolved
• No error thrown when global limit optimization failed to find enough number of rows [Spark Branch]: Resolved
• Make Remote Spark Context secure [Spark Branch]: Resolved
• Failed job may not throw exceptions [Spark Branch]: Resolved
• Enable CBO related tests [Spark Branch]: Closed
• UNION ALL query failed with ArrayIndexOutOfBoundsException [Spark Branch]: Resolved
• Hive reported exception because that hive's derby version conflict with spark's derby version [Spark Branch]: Patch Available
• Enable infer_bucket_sort_dyn_part.q for TestMiniSparkOnYarnCliDriver test. [Spark Branch]: Open
• SparkSessionImpl calcualte wrong cores number in TestSparkCliDriver [Spark Branch]: Open
• Merge trunk to Spark branch 2/2/2015 [Spark Branch]: Resolved
• SHUFFLE_SORT should only be used for order by query [Spark Branch]: Closed
• Revert changes in two test configuration files accidently brought in by HIVE-9552 [Spark Branch]: Resolved
• Enable more unit tests for UNION ALL [Spark Branch]: Closed
• Lazy computing in HiveBaseFunctionResultList may hurt performance [Spark Branch]: Resolved
• 'Error while trying to create table container' occurs during hive query case execution when hive.optimize.skewjoin set to 'true' [Spark Branch]: Closed
• Improve some qtests: Closed
• Support Impersonation [Spark Branch]: Closed
• Address RB comments for HIVE-9425 [Spark Branch]: Closed
• Hive on Spark is not as aggressive as MR on map join [Spark Branch]: Resolved
• Merge trunk to Spark branch 2/15/2015 [Spark Branch]: Closed
• Upgrade to spark 1.3 [Spark Branch]: Closed
• Print yarn application id to console [Spark Branch]: Closed
• Utilize spark.kryo.classesToRegister [Spark Branch]: Closed
• java.lang.NoSuchMethodError occurs during hive query execution which has 'ADD FILE XXXX.jar' sentence: Resolved
• Merge trunk to Spark branch 02/27/2015 [Spark Branch]: Closed
• Load spark-defaults.conf from classpath [Spark Branch]: Closed
• Querying parquet tables fails with IllegalStateException [Spark Branch]: Resolved
• Print spark job id in history file [spark branch]: Closed
• Add jar/file doesn't work with yarn-cluster mode [Spark Branch]: Closed
• Merge trunk to Spark branch 3/6/2015 [Spark Branch]: Closed
• New Beeline queries will hang If Beeline terminates in-properly [Spark Branch]: Closed
• Avoid Utilities.getMapRedWork for spark [Spark Branch]: Closed
• RSC has memory leak while execute multi queries.[Spark Branch]: Closed
• HiveInputFormat implementations getsplits may lead to memory leak.[Spark Branch]: Open
• Log the information of cached RDD [Spark Branch]: Resolved
• Provide more informative stage description in Spark Web UI [Spark Branch]: Open
• Improve common join performance [Spark Branch]: Open
• Merge trunk to Spark branch 03/27/2015 [Spark Branch]: Closed
• Fix test failures after HIVE-10130 [Spark Branch]: Closed
• Merge Spark branch to master 7/30/2015: Closed
• Implement Hybrid Hybrid Grace Hash Join for Spark Branch [Spark Branch]: Open
• Hive on Spark job configuration needs to be logged [Spark Branch]: Closed
• ParseException issue (Failed to recognize predicate 'user') [Spark Branch]: Resolved
• Merge trunk to spark 4/14/2015 [Spark Branch]: Resolved
• Fix test failures after last merge from trunk [Spark Branch]: Open
• Merge spark to trunk 4/15/2015: Closed
• Cancel connection when remote Spark driver process has failed [Spark Branch]: Closed
• Enable parallel order by for spark [Spark Branch]: Closed
• Hive query should fail when it fails to initialize a session in SetSparkReducerParallelism [Spark Branch]: Closed
• NPE in SparkUtilities::isDedicatedCluster [Spark Branch]: Closed
• Dynamic RDD caching optimization for HoS.[Spark Branch]: Closed
• Combine equivalent Works for HoS[Spark Branch]: Closed
• Followup for HIVE-10550, check performance w.r.t. persistence level [Spark Branch]: Open
• Make HIVE-10001 work with Spark [Spark Branch]: Open
• Make HIVE-10568 work with Spark [Spark Branch]: Closed
• Merge master to Spark branch 7/29/2015 [Spark Branch]: Resolved
• Merge master to Spark branch 6/7/2015 [Spark Branch]: Resolved
• HoS can't control number of map tasks for runtime skew join [Spark Branch]: Closed
• Upgrade Spark dependency to 1.4 [Spark Branch]: Closed
• Hive not able to pass Hive's Kerberos credential to spark-submit process [Spark Branch]: Resolved
• Enable more tests for grouping by skewed data [Spark Branch]: Resolved
• Add more tests for HIVE-10844[Spark Branch]: Closed
• Remote Spark client doesn't use Kerberos keytab to authenticate [Spark Branch]: Closed
• Merge master to Spark branch 6/20/2015 [Spark Branch]: Resolved
• Support multi edge between nodes in SparkPlan[Spark Branch]: Closed
• Investigate intermitten failure of join28.q for Spark: Resolved
• Add support for running negative q-tests [Spark Branch]: Closed
• HashTableSinkOperator doesn't support vectorization [Spark Branch]: Closed
• Support hive.explain.user for Spark: Closed
• Query fails when there isn't a comparator for an operator [Spark Branch]: Closed
• Enable native vectorized map join for spark [Spark Branch]: Closed
• Research on recent failed qtests[Spark Branch]: Resolved
• Combine equavilent leaf works in SparkWork[Spark Branch]: Open
• Optimization around job submission and adding jars [Spark Branch]: Resolved
• Print "Execution completed successfully" as part of spark job info [Spark Branch]: Closed
• Prewarm Hive on Spark containers [Spark Branch]: Closed
• Merge master to Spark branch 9/16/2015 [Spark Branch]: Resolved
• HiveException (Failed to close AbstractFileMergeOperator) occurs during loading data to ORC file, when hive.merge.sparkfiles is set to true. [Spark Branch]: Resolved
• Merge file doesn't work for ORC table when running on Spark. [Spark Branch]: Closed
• Fix test failures after HIVE-11844 [Spark Branch]: Closed
• Merge master to Spark branch 10/28/2015 [Spark Branch]: Closed
• Merge master into spark 11/17/2015 [Spark Branch]: Resolved
• [Spark Branch] ClassNotFoundException occurs during query case with group by and UDF defined: Open
• NullPointerException thrown by Executors causes job can't be finished: Open