Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-3333

Fix Hive build after upgrading Spark to 2.4.5

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.5.0
    • Component/s: build, hive
    • Labels:
      None

      Description

      After BIGTOP-3165, building Hive came to fail as follows, due to the API change of Spark:

      $ ./gradlew hive-pkg
      
      (snip)
      
      [INFO] -------------------------------------------------------------
      [ERROR] COMPILATION ERROR : 
      [INFO] -------------------------------------------------------------
      [ERROR] /home/sekikn/repos/bigtop/output/hive/hive-2.3.6/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/LocalHiveSparkClient.java:[87,26] cannot find symbol
        symbol:   method addListener(org.apache.hadoop.hive.ql.exec.spark.status.impl.JobMetricsListener)
        location: class org.apache.spark.scheduler.LiveListenerBus
      [INFO] 1 error
      [INFO] -------------------------------------------------------------
      [INFO] ------------------------------------------------------------------------
      [INFO] Reactor Summary for Hive 2.3.6:
      [INFO] 
      [INFO] Hive ............................................... SUCCESS [  2.115 s]
      [INFO] Hive Shims Common .................................. SUCCESS [  3.889 s]
      [INFO] Hive Shims 0.23 .................................... SUCCESS [  2.686 s]
      [INFO] Hive Shims Scheduler ............................... SUCCESS [  0.976 s]
      [INFO] Hive Shims ......................................... SUCCESS [  0.793 s]
      [INFO] Hive Common ........................................ SUCCESS [  5.762 s]
      [INFO] Hive Service RPC ................................... SUCCESS [  2.480 s]
      [INFO] Hive Serde ......................................... SUCCESS [  4.136 s]
      [INFO] Hive Metastore ..................................... SUCCESS [ 16.040 s]
      [INFO] Hive Vector-Code-Gen Utilities ..................... SUCCESS [  0.306 s]
      [INFO] Hive Llap Common ................................... SUCCESS [  3.953 s]
      [INFO] Hive Llap Client ................................... SUCCESS [  2.334 s]
      [INFO] Hive Llap Tez ...................................... SUCCESS [  2.591 s]
      [INFO] Spark Remote Client ................................ SUCCESS [  3.028 s]
      [INFO] Hive Query Language ................................ FAILURE [ 17.783 s]
      [INFO] Hive Llap Server ................................... SKIPPED
      [INFO] Hive Service ....................................... SKIPPED
      [INFO] Hive Accumulo Handler .............................. SKIPPED
      [INFO] Hive JDBC .......................................... SKIPPED
      [INFO] Hive Beeline ....................................... SKIPPED
      [INFO] Hive CLI ........................................... SKIPPED
      [INFO] Hive Contrib ....................................... SKIPPED
      [INFO] Hive Druid Handler ................................. SKIPPED
      [INFO] Hive HBase Handler ................................. SKIPPED
      [INFO] Hive JDBC Handler .................................. SKIPPED
      [INFO] Hive HCatalog ...................................... SKIPPED
      [INFO] Hive HCatalog Core ................................. SKIPPED
      [INFO] Hive HCatalog Pig Adapter .......................... SKIPPED
      [INFO] Hive HCatalog Server Extensions .................... SKIPPED
      [INFO] Hive HCatalog Webhcat Java Client .................. SKIPPED
      [INFO] Hive HCatalog Webhcat .............................. SKIPPED
      [INFO] Hive HCatalog Streaming ............................ SKIPPED
      [INFO] Hive HPL/SQL ....................................... SKIPPED
      [INFO] Hive Llap External Client .......................... SKIPPED
      [INFO] Hive Shims Aggregator .............................. SKIPPED
      [INFO] Hive TestUtils ..................................... SKIPPED
      [INFO] Hive Packaging ..................................... SKIPPED
      [INFO] ------------------------------------------------------------------------
      [INFO] BUILD FAILURE
      [INFO] ------------------------------------------------------------------------
      [INFO] Total time:  01:09 min
      [INFO] Finished at: 2020-04-03T10:55:51+09:00
      [INFO] ------------------------------------------------------------------------
      [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project hive-exec: Compilation failure
      [ERROR] /home/sekikn/repos/bigtop/output/hive/hive-2.3.6/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/LocalHiveSparkClient.java:[87,26] cannot find symbol
      [ERROR]   symbol:   method addListener(org.apache.hadoop.hive.ql.exec.spark.status.impl.JobMetricsListener)
      [ERROR]   location: class org.apache.spark.scheduler.LiveListenerBus
      [ERROR] 
      [ERROR] -> [Help 1]
      [ERROR] 
      [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
      [ERROR] Re-run Maven using the -X switch to enable full debug logging.
      [ERROR] 
      [ERROR] For more information about the errors and possible solutions, please read the following articles:
      [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
      [ERROR] 
      [ERROR] After correcting the problems, you can resume the build with the command
      [ERROR]   mvn <args> -rf :hive-exec
      debian/rules:32: recipe for target 'override_dh_auto_build' failed
      make[1]: *** [override_dh_auto_build] Error 1
      make[1]: Leaving directory '/home/sekikn/repos/bigtop/output/hive/hive-2.3.6'
      debian/rules:29: recipe for target 'build' failed
      make: *** [build] Error 2
      dpkg-buildpackage: error: debian/rules build subprocess returned exit status 2
      debuild: fatal error at line 1152:
      dpkg-buildpackage -rfakeroot -us -uc -ui -b
       failed
      
      > Task :hive-deb FAILED
      
      FAILURE: Build failed with an exception.
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                sekikn Kengo Seki
                Reporter:
                sekikn Kengo Seki
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 10m
                  10m