Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 1.1.0
    • Fix Version/s: 1.1.0
    • Component/s: build
    • Labels:
      None

      Description

      Since BIGTOP-2104 was committed, the hive failed to build with the following error:
      [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project spark-client: Compilation failure
      [ERROR] /ws/build/hive/rpm/BUILD/apache-hive-1.2.1-src/spark-client/src/main/java/org/apache/hive/spark/client/RemoteDriver.java:[441,11] org.apache.hive.spark.client.RemoteDriver.ClientListener is not abstract and does not override abstract method onBlockUpdated(org.apache.spark.scheduler.SparkListenerBlockUpdated) in org.apache.spark.scheduler.SparkListener

      1. BIGTOP-2114.patch
        9 kB
        Sergey Soldatov
      2. BIGTOP-2114.2.patch
        9 kB
        Sergey Soldatov

        Issue Links

          Activity

          Hide
          cos Konstantin Boudnik added a comment -

          As Sergey Soldatov pointed out HIVE-11473 is right in the alley. Shall we just incorporate the patch into our Hive build process until the upstream is updated?

          Show
          cos Konstantin Boudnik added a comment - As Sergey Soldatov pointed out HIVE-11473 is right in the alley. Shall we just incorporate the patch into our Hive build process until the upstream is updated?
          Hide
          sergey.soldatov Sergey Soldatov added a comment -

          incorporated HIVE-11473

          Show
          sergey.soldatov Sergey Soldatov added a comment - incorporated HIVE-11473
          Hide
          sergey.soldatov Sergey Soldatov added a comment -

          Fix for hive based on HIVE-11473

          Show
          sergey.soldatov Sergey Soldatov added a comment - Fix for hive based on HIVE-11473
          Hide
          cos Konstantin Boudnik added a comment -

          We are building with 1.5.1 of Spark, so the following

          +-    <spark.version>1.3.1</spark.version>
          ++    <spark.version>1.5.0</spark.version>
          

          needs to be changed accordingly. Looks great other than that.

          Show
          cos Konstantin Boudnik added a comment - We are building with 1.5.1 of Spark, so the following +- <spark.version>1.3.1</spark.version> ++ <spark.version>1.5.0</spark.version> needs to be changed accordingly. Looks great other than that.
          Hide
          sergey.soldatov Sergey Soldatov added a comment -

          Updated Spark version to 1.5.1

          Show
          sergey.soldatov Sergey Soldatov added a comment - Updated Spark version to 1.5.1
          Hide
          cos Konstantin Boudnik added a comment - - edited

          The HIVE patch has some whitespaces in it, but if anything I don't care because the patch file will be gone in the next bump of Hive.

          Looks all right. I will commit it shortly!

          Show
          cos Konstantin Boudnik added a comment - - edited The HIVE patch has some whitespaces in it, but if anything I don't care because the patch file will be gone in the next bump of Hive. Looks all right. I will commit it shortly!
          Hide
          tomzeng Tom Zeng added a comment -

          Hey guys, HIVE-11473 is already in Hive 1.2.1, and with that build still breaks due to new changes in Spark 1.5.x. The problem we are experiencing is HIVE-11795, and I created a patch for it already. The tests are not passing there due to possibily the build server do have the latest Spark dependencies.

          Show
          tomzeng Tom Zeng added a comment - Hey guys, HIVE-11473 is already in Hive 1.2.1, and with that build still breaks due to new changes in Spark 1.5.x. The problem we are experiencing is HIVE-11795 , and I created a patch for it already. The tests are not passing there due to possibily the build server do have the latest Spark dependencies.
          Hide
          cos Konstantin Boudnik added a comment -

          Committed and pushed to the master. Thank you Sergey!

          Show
          cos Konstantin Boudnik added a comment - Committed and pushed to the master. Thank you Sergey!
          Hide
          sergey.soldatov Sergey Soldatov added a comment -

          It could be there since the bug was created after the 1.2.1 release

          Show
          sergey.soldatov Sergey Soldatov added a comment - It could be there since the bug was created after the 1.2.1 release
          Hide
          cos Konstantin Boudnik added a comment -

          Sorry, I've missed this comment while commiting the patch. Tests aren't passing as in on the cluster or else?

          Show
          cos Konstantin Boudnik added a comment - Sorry, I've missed this comment while commiting the patch. Tests aren't passing as in on the cluster or else?
          Hide
          tomzeng Tom Zeng added a comment -

          Hive QA was not passing for my patch for HIVE-11795. I tested the patch locally on latest Hive and Hive 1.2.1 both worked, without it, Hive master and 1.2.1 builds will fail

          Show
          tomzeng Tom Zeng added a comment - Hive QA was not passing for my patch for HIVE-11795 . I tested the patch locally on latest Hive and Hive 1.2.1 both worked, without it, Hive master and 1.2.1 builds will fail
          Hide
          tomzeng Tom Zeng added a comment - - edited

          You can try pull down Hive and run mvn clean install -Phadoop-2,dist -DskipTests -Dhadoop-23.version=2.7.1 -Dspark.version=1.5.0 or mvn clean install -Phadoop-2,dist -DskipTests -Dhadoop-23.version=2.7.1 -Dspark.version=1.5.1

          Show
          tomzeng Tom Zeng added a comment - - edited You can try pull down Hive and run mvn clean install -Phadoop-2,dist -DskipTests -Dhadoop-23.version=2.7.1 -Dspark.version=1.5.0 or mvn clean install -Phadoop-2,dist -DskipTests -Dhadoop-23.version=2.7.1 -Dspark.version=1.5.1
          Hide
          cos Konstantin Boudnik added a comment -

          Let's see how our CI build turns out before fixing anything: in clean docker the build is green with this patch in place.

          Show
          cos Konstantin Boudnik added a comment - Let's see how our CI build turns out before fixing anything: in clean docker the build is green with this patch in place.
          Hide
          sergey.soldatov Sergey Soldatov added a comment -

          This patch modifies JobMetricsListener and RemoteDriver in addition to pom.xml, so it supposed to fix the compilation issue you have experienced.

          Show
          sergey.soldatov Sergey Soldatov added a comment - This patch modifies JobMetricsListener and RemoteDriver in addition to pom.xml, so it supposed to fix the compilation issue you have experienced.
          Hide
          tomzeng Tom Zeng added a comment -

          I think Hive 1.2.1 as well as master (Hive 2.0) has the HIVE-11473 patch already. Since that patch a new method got added to Spark 1.5.0 and 1.5.1 which break Hive build again. The HIVE-11795 patch added the new method.

          Show
          tomzeng Tom Zeng added a comment - I think Hive 1.2.1 as well as master (Hive 2.0) has the HIVE-11473 patch already. Since that patch a new method got added to Spark 1.5.0 and 1.5.1 which break Hive build again. The HIVE-11795 patch added the new method.
          Hide
          sergey.soldatov Sergey Soldatov added a comment -

          Please check the modification that was made in the HIVE-11473:
          +- private class ClientListener implements SparkListener {
          ++ private class ClientListener extends JavaSparkListener {

          This method is implemented in JavaSparkListener, so no problems with the build.

          Show
          sergey.soldatov Sergey Soldatov added a comment - Please check the modification that was made in the HIVE-11473 : +- private class ClientListener implements SparkListener { ++ private class ClientListener extends JavaSparkListener { This method is implemented in JavaSparkListener, so no problems with the build.
          Hide
          tomzeng Tom Zeng added a comment -

          If you look at the error "does not override abstract method onBlockUpdated(org.apache.spark.scheduler.SparkListenerBlockUpdated)", the HIVE-11473 patch does not include fix for that.

          Show
          tomzeng Tom Zeng added a comment - If you look at the error "does not override abstract method onBlockUpdated(org.apache.spark.scheduler.SparkListenerBlockUpdated)", the HIVE-11473 patch does not include fix for that.
          Hide
          evans_ye Evans Ye added a comment -

          I just ran a manual build for Hive only on our CI:
          http://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-packages-manual-component-test/
          Looks like the builds are all good now.
          I didn't take time to look deeper so I just post the result here in case you need it.

          Show
          evans_ye Evans Ye added a comment - I just ran a manual build for Hive only on our CI: http://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-packages-manual-component-test/ Looks like the builds are all good now. I didn't take time to look deeper so I just post the result here in case you need it.
          Hide
          cos Konstantin Boudnik added a comment -

          Great stuff, thanks Evans Ye! It's super useful to be able to quickly run a build in the official environment!

          Show
          cos Konstantin Boudnik added a comment - Great stuff, thanks Evans Ye ! It's super useful to be able to quickly run a build in the official environment!
          Hide
          sergey.soldatov Sergey Soldatov added a comment -

          It doesn't need to implement that since now it extends the class that has the implementation instead of implementing the interface where this method was introduced.

          Show
          sergey.soldatov Sergey Soldatov added a comment - It doesn't need to implement that since now it extends the class that has the implementation instead of implementing the interface where this method was introduced.
          Hide
          tomzeng Tom Zeng added a comment -

          Ah I see, yes that's better than the patch I made for HIVE-11795. Great thanks.

          Show
          tomzeng Tom Zeng added a comment - Ah I see, yes that's better than the patch I made for HIVE-11795 . Great thanks.

            People

            • Assignee:
              sergey.soldatov Sergey Soldatov
              Reporter:
              sergey.soldatov Sergey Soldatov
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development