Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: 1.0.0
    • Component/s: build
    • Labels:
      None

      Description

      [INFO] -------------------------------------------------------------
      [ERROR] COMPILATION ERROR : 
      [INFO] -------------------------------------------------------------
      [ERROR] /home/olaf/bigtop/output/hive/hive-1.1.0/spark-client/src/main/java/org/apache/hive/spark/client/RemoteDriver.java:[423,11] org.apache.hive.spark.client.RemoteDriver.ClientListener is not abstract and does not override abstract method onExecutorRemoved(org.apache.spark.scheduler.SparkListenerExecutorRemoved) in org.apache.spark.scheduler.SparkListener
      [INFO] 1 error
      [INFO] -------------------------------------------------------------
      [INFO] ------------------------------------------------------------------------
      [INFO] Reactor Summary:
      [INFO] 
      [INFO] Hive .............................................. SUCCESS [3.328s]
      [INFO] Hive Shims Common ................................. SUCCESS [4.690s]
      [INFO] Hive Shims 0.20S .................................. SUCCESS [0.950s]
      [INFO] Hive Shims 0.23 ................................... SUCCESS [3.070s]
      [INFO] Hive Shims Scheduler .............................. SUCCESS [0.589s]
      [INFO] Hive Shims ........................................ SUCCESS [0.553s]
      [INFO] Hive Common ....................................... SUCCESS [3.379s]
      [INFO] Hive Serde ........................................ SUCCESS [5.979s]
      [INFO] Hive Metastore .................................... SUCCESS [12.397s]
      [INFO] Hive Ant Utilities ................................ SUCCESS [0.430s]
      [INFO] Spark Remote Client ............................... FAILURE [5.161s]
      [INFO] Hive Query Language ............................... SKIPPED
      [INFO] Hive Service ...................................... SKIPPED
      [INFO] Hive Accumulo Handler ............................. SKIPPED
      [INFO] Hive JDBC ......................................... SKIPPED
      [INFO] Hive Beeline ...................................... SKIPPED
      [INFO] Hive CLI .......................................... SKIPPED
      [INFO] Hive Contrib ...................................... SKIPPED
      [INFO] Hive HBase Handler ................................ SKIPPED
      [INFO] Hive HCatalog ..................................... SKIPPED
      [INFO] Hive HCatalog Core ................................ SKIPPED
      [INFO] Hive HCatalog Pig Adapter ......................... SKIPPED
      [INFO] Hive HCatalog Server Extensions ................... SKIPPED
      [INFO] Hive HCatalog Webhcat Java Client ................. SKIPPED
      [INFO] Hive HCatalog Webhcat ............................. SKIPPED
      [INFO] Hive HCatalog Streaming ........................... SKIPPED
      [INFO] Hive HWI .......................................... SKIPPED
      [INFO] Hive ODBC ......................................... SKIPPED
      [INFO] Hive Shims Aggregator ............................. SKIPPED
      [INFO] Hive TestUtils .................................... SKIPPED
      [INFO] Hive Packaging .................................... SKIPPED
      [INFO] ------------------------------------------------------------------------
      [INFO] BUILD FAILURE
      [INFO] ------------------------------------------------------------------------
      [INFO] Total time: 41.394s
      [INFO] Finished at: Fri Mar 27 10:28:06 CET 2015
      [INFO] Final Memory: 79M/1364M
      [INFO] ------------------------------------------------------------------------
      [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project spark-client: Compilation failure
      [ERROR] /home/olaf/bigtop/output/hive/hive-1.1.0/spark-client/src/main/java/org/apache/hive/spark/client/RemoteDriver.java:[423,11] org.apache.hive.spark.client.RemoteDriver.ClientListener is not abstract and does not override abstract method onExecutorRemoved(org.apache.spark.scheduler.SparkListenerExecutorRemoved) in org.apache.spark.scheduler.SparkListener
      [ERROR] -> [Help 1]
      [ERROR] 
      [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
      [ERROR] Re-run Maven using the -X switch to enable full debug logging.
      [ERROR] 
      [ERROR] For more information about the errors and possible solutions, please read the following articles:
      [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
      [ERROR] 
      [ERROR] After correcting the problems, you can resume the build with the command
      [ERROR]   mvn <goals> -rf :spark-client
      

        Issue Links

          Activity

          Hide
          oflebbe Olaf Flebbe added a comment -

          If you reset SPARK version to 1.2.0 Hive compiles

          Show
          oflebbe Olaf Flebbe added a comment - If you reset SPARK version to 1.2.0 Hive compiles
          Hide
          cos Konstantin Boudnik added a comment -

          Shall we revert the 1.3.0 commit then?

          Show
          cos Konstantin Boudnik added a comment - Shall we revert the 1.3.0 commit then?
          Hide
          jonathak Jonathan Kelly added a comment -

          Bummer, I wasn't even aware that the Hive project depends upon the Spark project. It does probably make sense to revert the Spark 1.3.0 upgrade commit if you want to get the build back to a stable state (especially since the commit is so small). It would be really nice to get Spark 1.3.0 into the Bigtop 1.0 release, but it's understandable if it needs to wait due to this failure.

          Show
          jonathak Jonathan Kelly added a comment - Bummer, I wasn't even aware that the Hive project depends upon the Spark project. It does probably make sense to revert the Spark 1.3.0 upgrade commit if you want to get the build back to a stable state (especially since the commit is so small). It would be really nice to get Spark 1.3.0 into the Bigtop 1.0 release, but it's understandable if it needs to wait due to this failure.
          Hide
          jayunit100 jay vyas added a comment -

          imo spark 1.3 is way more important than hive. are we sure we want to revert ?????

          Show
          jayunit100 jay vyas added a comment - imo spark 1.3 is way more important than hive. are we sure we want to revert ?????
          Hide
          jonathak Jonathan Kelly added a comment -

          I agree, but I meant just for the sake of getting a stable build for now. Besides, if we leave the Spark 1.3 upgrade, Hive will refuse to build, and Bigtop 1.0 can't be released like that, right? Can't really upgrade Spark without fixing Hive, unless Hive in Bigtop 1.0 gets bumped back to 1.0. So it's probably ideal both to keep the upgrade and to fix the Hive failure, as long as both can be done by the Bigtop 1.0 release, right?

          BTW, the reason my team and I hadn't noticed this failure in Hive is that we currently have the Hive version set to 1.0 rather than 1.1 in our bigtop.mk. All Spark functionality (aka "Hive on Spark") was added in 1.1.0. HIVE-9726 supposedly fixed the upgrade to Spark 1.3, so I'm trying to figure out why this failure is occurring. The fix version still just says "spark-branch", so it's unclear to me whether or not that fix was actually merged into Hive's 1.1 branch.

          Show
          jonathak Jonathan Kelly added a comment - I agree, but I meant just for the sake of getting a stable build for now. Besides, if we leave the Spark 1.3 upgrade, Hive will refuse to build, and Bigtop 1.0 can't be released like that, right? Can't really upgrade Spark without fixing Hive, unless Hive in Bigtop 1.0 gets bumped back to 1.0. So it's probably ideal both to keep the upgrade and to fix the Hive failure, as long as both can be done by the Bigtop 1.0 release, right? BTW, the reason my team and I hadn't noticed this failure in Hive is that we currently have the Hive version set to 1.0 rather than 1.1 in our bigtop.mk. All Spark functionality (aka "Hive on Spark") was added in 1.1.0. HIVE-9726 supposedly fixed the upgrade to Spark 1.3, so I'm trying to figure out why this failure is occurring. The fix version still just says "spark-branch", so it's unclear to me whether or not that fix was actually merged into Hive's 1.1 branch.
          Show
          jonathak Jonathan Kelly added a comment - Looks like the Hive upgrade to Spark 1.3 ( https://github.com/apache/hive/commit/88996a87f7a784f1385b238081f5ed9ab13dcb93 ) is not yet in https://github.com/apache/hive/tree/branch-1.1 , just in the https://github.com/apache/hive/tree/spark branch.
          Hide
          jayunit100 jay vyas added a comment -

          Thanks Jonathan Kelly !!!
          so, if thats the case, lets outline possible solutions ........ and choose a path forward in this jira.

          Show
          jayunit100 jay vyas added a comment - Thanks Jonathan Kelly !!! so, if thats the case, lets outline possible solutions ........ and choose a path forward in this jira.
          Hide
          cos Konstantin Boudnik added a comment -

          What Jonathan Kelly said: even if 1.3 is very important the only other option to fix the build is get rid of Hive, as I don't think anyone wants to patch it in our release. I'd suggest to revert the commit and re-open the original ticket to keep the track of the upgrade later...

          Show
          cos Konstantin Boudnik added a comment - What Jonathan Kelly said: even if 1.3 is very important the only other option to fix the build is get rid of Hive, as I don't think anyone wants to patch it in our release. I'd suggest to revert the commit and re-open the original ticket to keep the track of the upgrade later...
          Hide
          oflebbe Olaf Flebbe added a comment -

          I vote for cherry-picking the changes needed for Spark-1.3. Shall I propose a patch?

          Show
          oflebbe Olaf Flebbe added a comment - I vote for cherry-picking the changes needed for Spark-1.3. Shall I propose a patch?
          Hide
          oflebbe Olaf Flebbe added a comment -

          We can revert back to HIVE 1.0.0, too , if this fixes the situation

          Show
          oflebbe Olaf Flebbe added a comment - We can revert back to HIVE 1.0.0, too , if this fixes the situation
          Hide
          cos Konstantin Boudnik added a comment -

          Cherry picking for Hive? We really never did this and I am very hesitant to start, honestly.

          Show
          cos Konstantin Boudnik added a comment - Cherry picking for Hive? We really never did this and I am very hesitant to start, honestly.
          Hide
          jonathak Jonathan Kelly added a comment -

          Yeah the cherry-picking would need to be in the Hive project, not in Bigtop. It would be better to have the Hive community merge from the spark branch to the 1.1 branch again, as long as it's ready.

          Show
          jonathak Jonathan Kelly added a comment - Yeah the cherry-picking would need to be in the Hive project, not in Bigtop. It would be better to have the Hive community merge from the spark branch to the 1.1 branch again, as long as it's ready.
          Hide
          jayunit100 jay vyas added a comment -

          can we just build hive w/o spark support?

          Show
          jayunit100 jay vyas added a comment - can we just build hive w/o spark support?
          Hide
          jonathak Jonathan Kelly added a comment -

          Yes, I was wondering the same. Looks like from BIGTOP-1701 that the Hive 1.0->1.1 upgrade was trivial, and so was the Spark 1.2->1.3 upgrade, and yet they weren't compatible with each other because Hive on Spark's upgrade to 1.2->1.3 apparently didn't make it into the Hive 1.1 release. So then I guess returning to ~jayunit100's question of what's more important, Spark 1.3 is probably more important than Hive 1.1, right?

          Show
          jonathak Jonathan Kelly added a comment - Yes, I was wondering the same. Looks like from BIGTOP-1701 that the Hive 1.0->1.1 upgrade was trivial, and so was the Spark 1.2->1.3 upgrade, and yet they weren't compatible with each other because Hive on Spark's upgrade to 1.2->1.3 apparently didn't make it into the Hive 1.1 release. So then I guess returning to ~jayunit100's question of what's more important, Spark 1.3 is probably more important than Hive 1.1, right?
          Hide
          jonathak Jonathan Kelly added a comment -

          I apparently do not know how to reference other users. Don't you just use a tilde followed by a username? Maybe it didn't like the 's.

          Show
          jonathak Jonathan Kelly added a comment - I apparently do not know how to reference other users. Don't you just use a tilde followed by a username? Maybe it didn't like the 's.
          Hide
          cos Konstantin Boudnik added a comment -

          you also can use ampersand and then type user's name - JIRA will look it up for you.

          Show
          cos Konstantin Boudnik added a comment - you also can use ampersand and then type user's name - JIRA will look it up for you.
          Hide
          cos Konstantin Boudnik added a comment - - edited

          The beauty of Bigtop - I bet no one has tried this combination before: hence the problem was sitting there completely hidden.

          Show
          cos Konstantin Boudnik added a comment - - edited The beauty of Bigtop - I bet no one has tried this combination before: hence the problem was sitting there completely hidden.
          Hide
          oflebbe Olaf Flebbe added a comment -

          Yep. Use Hive-1.0.0 as a work around for the short term and watch upstream... Please review patch

          Show
          oflebbe Olaf Flebbe added a comment - Yep. Use Hive-1.0.0 as a work around for the short term and watch upstream... Please review patch
          Hide
          cos Konstantin Boudnik added a comment -

          Looks legit.
          +1

          Show
          cos Konstantin Boudnik added a comment - Looks legit. +1
          Hide
          oflebbe Olaf Flebbe added a comment -

          pushed. Fixed for now.

          Show
          oflebbe Olaf Flebbe added a comment - pushed. Fixed for now.
          Hide
          cos Konstantin Boudnik added a comment -

          Looks like this has broke pig build as it now it can not find Hive 1.1.0 shims.... Damn, what a mess these things are ;(

          Show
          cos Konstantin Boudnik added a comment - Looks like this has broke pig build as it now it can not find Hive 1.1.0 shims.... Damn, what a mess these things are ;(
          Hide
          jonathak Jonathan Kelly added a comment - - edited

          Dang, really? That's odd though, since Pig was upgraded to 0.14 months ago (see BIGTOP-1594), long before Hive was upgraded to 1.1.0. Do you have an error message or stack trace?

          Show
          jonathak Jonathan Kelly added a comment - - edited Dang, really? That's odd though, since Pig was upgraded to 0.14 months ago (see BIGTOP-1594 ), long before Hive was upgraded to 1.1.0. Do you have an error message or stack trace?
          Hide
          cos Konstantin Boudnik added a comment -

          The comment was wrong - I thought I remove it - please disregard it - it works, actually...

          Show
          cos Konstantin Boudnik added a comment - The comment was wrong - I thought I remove it - please disregard it - it works, actually...
          Hide
          jonathak Jonathan Kelly added a comment -

          Oh, I see. Haha. I saw in my email that you deleted a comment right after you posted that one, but it was a different one that you deleted: "Hey guys, if the recent upgrade to HIVE 1.1.0 introduced this problem, we can revert backy to 1.0.0. No problem. Looking into it"

          Show
          jonathak Jonathan Kelly added a comment - Oh, I see. Haha. I saw in my email that you deleted a comment right after you posted that one, but it was a different one that you deleted: "Hey guys, if the recent upgrade to HIVE 1.1.0 introduced this problem, we can revert backy to 1.0.0. No problem. Looking into it"
          Hide
          cos Konstantin Boudnik added a comment -

          Ah crap. I should stop working past 2 am ;(

          Show
          cos Konstantin Boudnik added a comment - Ah crap. I should stop working past 2 am ;(
          Hide
          jayunit100 jay vyas added a comment -

          is this resolved then?

          Show
          jayunit100 jay vyas added a comment - is this resolved then?
          Hide
          cos Konstantin Boudnik added a comment -

          It is, and the ticket is clearly marked as such. Sorry about the confusion.

          Show
          cos Konstantin Boudnik added a comment - It is, and the ticket is clearly marked as such. Sorry about the confusion.

            People

            • Assignee:
              oflebbe Olaf Flebbe
              Reporter:
              oflebbe Olaf Flebbe
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development