[BIGTOP-3333] Fix Hive build after upgrading Spark to 2.4.5

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.5.0
• Component/s: build, hive
• Assignee: Kengo Seki
• Reporter: Kengo Seki
• Created: 03/Apr/20 02:03
• Updated: 27/Apr/20 23:43
• Resolved: 27/Apr/20 23:43

Description

After BIGTOP-3165, building Hive came to fail as follows, due to the API change of Spark:

$ ./gradlew hive-pkg

(snip)

[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR : 
[INFO] -------------------------------------------------------------
[ERROR] /home/sekikn/repos/bigtop/output/hive/hive-2.3.6/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/LocalHiveSparkClient.java:[87,26] cannot find symbol
  symbol:   method addListener(org.apache.hadoop.hive.ql.exec.spark.status.impl.JobMetricsListener)
  location: class org.apache.spark.scheduler.LiveListenerBus
[INFO] 1 error
[INFO] -------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Hive 2.3.6:
[INFO] 
[INFO] Hive ............................................... SUCCESS [  2.115 s]
[INFO] Hive Shims Common .................................. SUCCESS [  3.889 s]
[INFO] Hive Shims 0.23 .................................... SUCCESS [  2.686 s]
[INFO] Hive Shims Scheduler ............................... SUCCESS [  0.976 s]
[INFO] Hive Shims ......................................... SUCCESS [  0.793 s]
[INFO] Hive Common ........................................ SUCCESS [  5.762 s]
[INFO] Hive Service RPC ................................... SUCCESS [  2.480 s]
[INFO] Hive Serde ......................................... SUCCESS [  4.136 s]
[INFO] Hive Metastore ..................................... SUCCESS [ 16.040 s]
[INFO] Hive Vector-Code-Gen Utilities ..................... SUCCESS [  0.306 s]
[INFO] Hive Llap Common ................................... SUCCESS [  3.953 s]
[INFO] Hive Llap Client ................................... SUCCESS [  2.334 s]
[INFO] Hive Llap Tez ...................................... SUCCESS [  2.591 s]
[INFO] Spark Remote Client ................................ SUCCESS [  3.028 s]
[INFO] Hive Query Language ................................ FAILURE [ 17.783 s]
[INFO] Hive Llap Server ................................... SKIPPED
[INFO] Hive Service ....................................... SKIPPED
[INFO] Hive Accumulo Handler .............................. SKIPPED
[INFO] Hive JDBC .......................................... SKIPPED
[INFO] Hive Beeline ....................................... SKIPPED
[INFO] Hive CLI ........................................... SKIPPED
[INFO] Hive Contrib ....................................... SKIPPED
[INFO] Hive Druid Handler ................................. SKIPPED
[INFO] Hive HBase Handler ................................. SKIPPED
[INFO] Hive JDBC Handler .................................. SKIPPED
[INFO] Hive HCatalog ...................................... SKIPPED
[INFO] Hive HCatalog Core ................................. SKIPPED
[INFO] Hive HCatalog Pig Adapter .......................... SKIPPED
[INFO] Hive HCatalog Server Extensions .................... SKIPPED
[INFO] Hive HCatalog Webhcat Java Client .................. SKIPPED
[INFO] Hive HCatalog Webhcat .............................. SKIPPED
[INFO] Hive HCatalog Streaming ............................ SKIPPED
[INFO] Hive HPL/SQL ....................................... SKIPPED
[INFO] Hive Llap External Client .......................... SKIPPED
[INFO] Hive Shims Aggregator .............................. SKIPPED
[INFO] Hive TestUtils ..................................... SKIPPED
[INFO] Hive Packaging ..................................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  01:09 min
[INFO] Finished at: 2020-04-03T10:55:51+09:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project hive-exec: Compilation failure
[ERROR] /home/sekikn/repos/bigtop/output/hive/hive-2.3.6/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/LocalHiveSparkClient.java:[87,26] cannot find symbol
[ERROR]   symbol:   method addListener(org.apache.hadoop.hive.ql.exec.spark.status.impl.JobMetricsListener)
[ERROR]   location: class org.apache.spark.scheduler.LiveListenerBus
[ERROR] 
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :hive-exec
debian/rules:32: recipe for target 'override_dh_auto_build' failed
make[1]: *** [override_dh_auto_build] Error 1
make[1]: Leaving directory '/home/sekikn/repos/bigtop/output/hive/hive-2.3.6'
debian/rules:29: recipe for target 'build' failed
make: *** [build] Error 2
dpkg-buildpackage: error: debian/rules build subprocess returned exit status 2
debuild: fatal error at line 1152:
dpkg-buildpackage -rfakeroot -us -uc -ui -b
 failed

> Task :hive-deb FAILED

FAILURE: Build failed with an exception.

Issue Links

• is depended upon by: BIGTOP-3347 Fix failures of Hive ProxyFileSystem against Hadoop 2.10.0
• relates to: HIVE-18436 Upgrade to Spark 2.3.0
• links to: GitHub Pull Request #622