Description
For the time being, at least, we've decided to build the Spark client (see SPARK-3215) inside Hive. This task tracks merging the ongoing work into the Spark branch.
Attachments
Attachments
- HIVE-8528.1-spark.patch
- 105 kB
- Xuefu Zhang
- HIVE-8528.1-spark-client.patch
- 105 kB
- Marcelo Masiero Vanzin
- HIVE-8528.2-spark.patch
- 105 kB
- Xuefu Zhang
- HIVE-8528.2-spark.patch
- 105 kB
- Marcelo Masiero Vanzin
- HIVE-8528.3-spark.patch
- 101 kB
- Marcelo Masiero Vanzin
Issue Links
- is depended upon by
-
HIVE-8548 Integrate with remote Spark context after HIVE-8528 [Spark Branch]
- Resolved
- is related to
-
HIVE-8574 Enhance metrics gathering in Spark Client [Spark Branch]
- Resolved
- relates to
-
SPARK-3215 Add remote interface for SparkContext
- Resolved
Activity
Load the same patch but with different name to trigger the test run. For information only, patch needs to be postfixed with spark.patch in order to make this happen.
Overall: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12676014/HIVE-8528.1-spark.patch
ERROR: -1 due to 13 failed/errored test(s), 6782 tests executed
Failed tests:
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_tez_smb_1 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vectorization_12 org.apache.hadoop.hive.ql.security.TestExtendedAcls.testAlterPartition org.apache.hadoop.hive.ql.security.TestExtendedAcls.testInsert org.apache.hadoop.hive.ql.security.TestExtendedAcls.testLoad org.apache.hadoop.hive.ql.security.TestExtendedAcls.testLoadLocal org.apache.hadoop.hive.ql.security.TestExtendedAcls.testStaticPartition org.apache.hadoop.hive.ql.security.TestFolderPermissions.testAlterPartition org.apache.hadoop.hive.ql.security.TestFolderPermissions.testInsert org.apache.hadoop.hive.ql.security.TestFolderPermissions.testLoad org.apache.hadoop.hive.ql.security.TestFolderPermissions.testLoadLocal org.apache.hadoop.hive.ql.security.TestFolderPermissions.testStaticPartition
Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/244/testReport
Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/244/console
Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-244/
Messages:
Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 13 tests failed
This message is automatically generated.
ATTACHMENT ID: 12676014 - PreCommit-HIVE-SPARK-Build
vanzin, I'm wondering if the test failures are caused by the junit version change introduced in the patch. Could you take a look please? I'm not sure if it's necessary to upgrade junit. Please comment.
I upgraded because my tests use assertNotEquals which was added in 4.11. I'll revert that change and change the test to see if it fixes the issues.
Ah, I'll also have to update the code to match changes in the Spark API, so it will take a little longer...
Overall: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12676117/HIVE-8528.2-spark.patch
Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/246/testReport
Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/246/console
Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-246/
Messages:
**** This message was trimmed, see log for full details **** As a result, alternative(s) 6 were disabled for that input warning(200): IdentifiersParser.g:179:5: Decision can match input such as "KW_ARRAY" using multiple alternatives: 2, 6 As a result, alternative(s) 6 were disabled for that input warning(200): IdentifiersParser.g:179:5: Decision can match input such as "KW_UNIONTYPE" using multiple alternatives: 5, 6 As a result, alternative(s) 6 were disabled for that input warning(200): IdentifiersParser.g:261:5: Decision can match input such as "KW_TRUE" using multiple alternatives: 3, 8 As a result, alternative(s) 8 were disabled for that input warning(200): IdentifiersParser.g:261:5: Decision can match input such as "KW_NULL" using multiple alternatives: 1, 8 As a result, alternative(s) 8 were disabled for that input warning(200): IdentifiersParser.g:261:5: Decision can match input such as "KW_FALSE" using multiple alternatives: 3, 8 As a result, alternative(s) 8 were disabled for that input warning(200): IdentifiersParser.g:261:5: Decision can match input such as "KW_DATE StringLiteral" using multiple alternatives: 2, 3 As a result, alternative(s) 3 were disabled for that input warning(200): IdentifiersParser.g:393:5: Decision can match input such as "KW_BETWEEN KW_MAP LPAREN" using multiple alternatives: 8, 9 As a result, alternative(s) 9 were disabled for that input warning(200): IdentifiersParser.g:393:5: Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_INSERT KW_INTO" using multiple alternatives: 2, 9 As a result, alternative(s) 9 were disabled for that input warning(200): IdentifiersParser.g:393:5: Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_UNION KW_ALL" using multiple alternatives: 2, 9 As a result, alternative(s) 9 were disabled for that input warning(200): IdentifiersParser.g:393:5: Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_SORT KW_BY" using multiple alternatives: 2, 9 As a result, alternative(s) 9 were disabled for that input warning(200): IdentifiersParser.g:393:5: Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_CLUSTER KW_BY" using multiple alternatives: 2, 9 As a result, alternative(s) 9 were disabled for that input warning(200): IdentifiersParser.g:393:5: Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_LATERAL KW_VIEW" using multiple alternatives: 2, 9 As a result, alternative(s) 9 were disabled for that input warning(200): IdentifiersParser.g:393:5: Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_MAP LPAREN" using multiple alternatives: 2, 9 As a result, alternative(s) 9 were disabled for that input warning(200): IdentifiersParser.g:393:5: Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_DISTRIBUTE KW_BY" using multiple alternatives: 2, 9 As a result, alternative(s) 9 were disabled for that input warning(200): IdentifiersParser.g:393:5: Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_ORDER KW_BY" using multiple alternatives: 2, 9 As a result, alternative(s) 9 were disabled for that input warning(200): IdentifiersParser.g:393:5: Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_GROUP KW_BY" using multiple alternatives: 2, 9 As a result, alternative(s) 9 were disabled for that input warning(200): IdentifiersParser.g:393:5: Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_INSERT KW_OVERWRITE" using multiple alternatives: 2, 9 As a result, alternative(s) 9 were disabled for that input warning(200): IdentifiersParser.g:518:5: Decision can match input such as "{AMPERSAND..BITWISEXOR, DIV..DIVIDE, EQUAL..EQUAL_NS, GREATERTHAN..GREATERTHANOREQUALTO, KW_AND, KW_ARRAY, KW_BETWEEN..KW_BOOLEAN, KW_CASE, KW_DOUBLE, KW_FLOAT, KW_IF, KW_IN, KW_INT, KW_LIKE, KW_MAP, KW_NOT, KW_OR, KW_REGEXP, KW_RLIKE, KW_SMALLINT, KW_STRING..KW_STRUCT, KW_TINYINT, KW_UNIONTYPE, KW_WHEN, LESSTHAN..LESSTHANOREQUALTO, MINUS..NOTEQUAL, PLUS, STAR, TILDE}" using multiple alternatives: 1, 3 As a result, alternative(s) 3 were disabled for that input [INFO] [INFO] --- maven-remote-resources-plugin:1.5:process (default) @ hive-exec --- [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ hive-exec --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 2 resources [INFO] Copying 3 resources [INFO] [INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ hive-exec --- [INFO] Executing tasks main: [INFO] Executed tasks [INFO] [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hive-exec --- [INFO] Compiling 1967 source files to /data/hive-ptest/working/apache-svn-spark-source/ql/target/classes [INFO] ------------------------------------------------------------- [WARNING] COMPILATION WARNING : [INFO] ------------------------------------------------------------- [WARNING] /data/hive-ptest/working/apache-svn-spark-source/ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java: Some input files use or override a deprecated API. [WARNING] /data/hive-ptest/working/apache-svn-spark-source/ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java: Recompile with -Xlint:deprecation for details. [WARNING] /data/hive-ptest/working/apache-svn-spark-source/ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java: Some input files use unchecked or unsafe operations. [WARNING] /data/hive-ptest/working/apache-svn-spark-source/ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java: Recompile with -Xlint:unchecked for details. [INFO] 4 warnings [INFO] ------------------------------------------------------------- [INFO] ------------------------------------------------------------- [ERROR] COMPILATION ERROR : [INFO] ------------------------------------------------------------- [ERROR] /data/hive-ptest/working/apache-svn-spark-source/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkClient.java:[190,50] incompatible types required: org.apache.spark.FutureAction found: org.apache.spark.api.java.JavaFutureAction<java.lang.Void> [ERROR] /data/hive-ptest/working/apache-svn-spark-source/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HivePairFlatMapFunction.java:[46,53] cannot find symbol symbol: method getStageId() location: class org.apache.spark.TaskContext [ERROR] /data/hive-ptest/working/apache-svn-spark-source/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HivePairFlatMapFunction.java:[57,70] cannot find symbol symbol: method getPartitionId() location: class org.apache.spark.TaskContext [ERROR] /data/hive-ptest/working/apache-svn-spark-source/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HivePairFlatMapFunction.java:[63,62] cannot find symbol symbol: method getPartitionId() location: class org.apache.spark.TaskContext [ERROR] /data/hive-ptest/working/apache-svn-spark-source/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkUtilities.java:[30,56] cannot find symbol symbol: method getPartitionId() location: variable taskContext of type org.apache.spark.TaskContext [ERROR] /data/hive-ptest/working/apache-svn-spark-source/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkUtilities.java:[35,47] cannot find symbol symbol: method getPartitionId() location: variable taskContext of type org.apache.spark.TaskContext [ERROR] /data/hive-ptest/working/apache-svn-spark-source/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkUtilities.java:[35,77] cannot find symbol symbol: method getAttemptId() location: variable taskContext of type org.apache.spark.TaskContext [INFO] 7 errors [INFO] ------------------------------------------------------------- [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Hive .............................................. SUCCESS [2.860s] [INFO] Hive Shims Common ................................. SUCCESS [2.529s] [INFO] Hive Shims 0.20 ................................... SUCCESS [0.913s] [INFO] Hive Shims Secure Common .......................... SUCCESS [1.574s] [INFO] Hive Shims 0.20S .................................. SUCCESS [0.819s] [INFO] Hive Shims 0.23 ................................... SUCCESS [2.144s] [INFO] Hive Shims ........................................ SUCCESS [0.321s] [INFO] Hive Common ....................................... SUCCESS [4.172s] [INFO] Hive Serde ........................................ SUCCESS [3.852s] [INFO] Hive Metastore .................................... SUCCESS [12.316s] [INFO] Hive Ant Utilities ................................ SUCCESS [0.538s] [INFO] Hive Query Language ............................... FAILURE [19.009s] [INFO] Hive Service ...................................... SKIPPED [INFO] Hive Accumulo Handler ............................. SKIPPED [INFO] Hive JDBC ......................................... SKIPPED [INFO] Hive Beeline ...................................... SKIPPED [INFO] Hive CLI .......................................... SKIPPED [INFO] Hive Contrib ...................................... SKIPPED [INFO] Hive HBase Handler ................................ SKIPPED [INFO] Hive HCatalog ..................................... SKIPPED [INFO] Hive HCatalog Core ................................ SKIPPED [INFO] Hive HCatalog Pig Adapter ......................... SKIPPED [INFO] Hive HCatalog Server Extensions ................... SKIPPED [INFO] Hive HCatalog Webhcat Java Client ................. SKIPPED [INFO] Hive HCatalog Webhcat ............................. SKIPPED [INFO] Hive HCatalog Streaming ........................... SKIPPED [INFO] Hive HWI .......................................... SKIPPED [INFO] Hive ODBC ......................................... SKIPPED [INFO] Hive Shims Aggregator ............................. SKIPPED [INFO] Spark Remote Client ............................... SKIPPED [INFO] Hive TestUtils .................................... SKIPPED [INFO] Hive Packaging .................................... SKIPPED [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 51.985s [INFO] Finished at: Tue Oct 21 13:15:36 EDT 2014 [INFO] Final Memory: 90M/1067M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project hive-exec: Compilation failure: Compilation failure: [ERROR] /data/hive-ptest/working/apache-svn-spark-source/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkClient.java:[190,50] incompatible types [ERROR] required: org.apache.spark.FutureAction [ERROR] found: org.apache.spark.api.java.JavaFutureAction<java.lang.Void> [ERROR] /data/hive-ptest/working/apache-svn-spark-source/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HivePairFlatMapFunction.java:[46,53] cannot find symbol [ERROR] symbol: method getStageId() [ERROR] location: class org.apache.spark.TaskContext [ERROR] /data/hive-ptest/working/apache-svn-spark-source/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HivePairFlatMapFunction.java:[57,70] cannot find symbol [ERROR] symbol: method getPartitionId() [ERROR] location: class org.apache.spark.TaskContext [ERROR] /data/hive-ptest/working/apache-svn-spark-source/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HivePairFlatMapFunction.java:[63,62] cannot find symbol [ERROR] symbol: method getPartitionId() [ERROR] location: class org.apache.spark.TaskContext [ERROR] /data/hive-ptest/working/apache-svn-spark-source/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkUtilities.java:[30,56] cannot find symbol [ERROR] symbol: method getPartitionId() [ERROR] location: variable taskContext of type org.apache.spark.TaskContext [ERROR] /data/hive-ptest/working/apache-svn-spark-source/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkUtilities.java:[35,47] cannot find symbol [ERROR] symbol: method getPartitionId() [ERROR] location: variable taskContext of type org.apache.spark.TaskContext [ERROR] /data/hive-ptest/working/apache-svn-spark-source/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkUtilities.java:[35,77] cannot find symbol [ERROR] symbol: method getAttemptId() [ERROR] location: variable taskContext of type org.apache.spark.TaskContext [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <goals> -rf :hive-exec '
This message is automatically generated.
ATTACHMENT ID: 12676117 - PreCommit-HIVE-SPARK-Build
Hmmm, seems the rest of the Spark-related code in Hive needs to be updated to match the recent changes in Spark...
Overall: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12676126/HIVE-8528.2-spark.patch
ERROR: -1 due to 2 failed/errored test(s), 6782 tests executed
Failed tests:
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_tez_smb_1
Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/247/testReport
Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/247/console
Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-247/
Messages:
Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed
This message is automatically generated.
ATTACHMENT ID: 12676126 - PreCommit-HIVE-SPARK-Build
vanzin, it seems tests passed (remaining failure are known). Awesome! Could you please create a review board entry for easier code review? Thanks.
With my limitted knowledge in Spark, akka, etc. the patch looks good. I only have a few comments about code styling, logging, etc.
chengxiang li and lirui, could you please take a look at the patch. Especially, chengxiang li will do the integration, so please make sure it meets our need.
For any found deficiencies in functionality, we can create followup JIRAs to track.
Forgive my ignorance, just some high level questions:
- Why would we want this remote spark client?
- With the change, it seems instead of creating a SparkContext locally, user now has to create a SparkClient, which will create the SparkContext in separate process (within RemoteDriver). Would this remote SparkContext be shared somehow?
- Do we plan to make this the default mode for hive on spark?
Hi ruili, SPARK-3215 has the relevent info and a document. It should help answer some if not all your questions. Please let us know if you have additional questions after reading those. Thanks.
xuefuz - thanks for pointing me to the detailed info. I guess the main use case for hive is in hive server or when the client has limited resources?
It's mainly for HiveSever2, where there are concurrent client sessions, each having its SparkContext object. SparkContext is heavy, putting memory pressure on HiveServer2, proportionally to the number of active client sessions. With remote SparkContext, the pressure is transfered to remote process.
I'm +1 on this patch on RB(seems not updated here yet), while with one addition that we need a flow control for Metrics, otherwise it would lead to a memory leak finally, althrough I think it could be tracked in an independent ticket.
Yeah, the metrics code in general is a little hacky and sort of ugly to use. I need to spend more time thinking about it.
Thanks to chengxiang li for the review. vanzin, would you mind creating a JIRA to track the metrics flow control issue? Thanks.
Overall: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12676621/HIVE-8528.3-spark.patch
ERROR: -1 due to 2 failed/errored test(s), 6782 tests executed
Failed tests:
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_tez_smb_1
Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/255/testReport
Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/255/console
Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-255/
Messages:
Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed
This message is automatically generated.
ATTACHMENT ID: 12676621 - PreCommit-HIVE-SPARK-Build
Patch committed to Spark branch. Thanks to Marcelo for the contribution.
Does this need any documentation, or is it already covered in the PDF attached to SPARK-3215?
Hi Lefty, what kind of documentation are you looking for? This is, at the moment, targeted at internal Hive use only, so having nice end-user documentation is not currently a goal. (In fact, I should probably go and add those annotations to the classes.)
Hi leftylev, As explained by Marcelo, it's our goal to hide this completely from the user, and thus user doc doesn't need to cover it. Thanks for asking though.
Well, if it needs any setup or configuration you could put that in the "Hive on Spark: Getting Started" doc. Usage notes too. If it will need documentation later on, you can add the label TODOC-SPARK. (Oops, I added a TODOC label by accident while checking the name of the Spark label. Gone now.) A release note could hold some doc notes, but it won't get published from the branch. Of course these comments can hold doc notes too.
I just want to make sure docs don't get forgotten, but maybe this doesn't need any docs. Thanks.
Actually, Left, that's a good point, this might need some end-user documentation since the recommended setup is to have a full Spark installation available on the HS2 node. I don't know if the plan is to somehow package that with HS2 or leave it as a configuration step.
vanzin, I thought spark installation on HS2 host was optional. Let me know if this has changed.
It is optional, but I don't really think we should encourage that. A full install should be the recommended setup.
Doc note: szehon added a "Remote Spark Drive" section to Configuration Properties with a nice overview. (Thanks, Szehon.)
Hi vanzin, thank for working on this. For patch to trigger test, it needs to be named as something like
HIVE-8528.1-spark.patch. Thanks.