Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-1420

Update Phoenix version to 4.1.0 in the BOM

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: 0.8.0
    • Component/s: None
    • Labels:
      None

      Description

      When building BigTop with Phoenix 4.0, application fail to run because Phoenix contains an hadoop1 xml definition in the hadoop2 jar. This is fixed on Phoenix 4.0.1, however, BigTop doesn't compile with this version and need some manual intervention.

      So to have a fully running BigTop Phoenix rpm running with Hadoop 2.x.x, we need to update the build scripts to run with Phoenx 4.0.1 or above.

      1. BIGTOP-1420.patch
        2 kB
        Andrew Purtell
      2. BIGTOP-1420.patch
        1 kB
        Andrew Purtell

        Issue Links

          Activity

          Hide
          jmspaggi Jean-Marc Spaggiari added a comment -

          To get Phoenix into BigTop build we had to do those steps.

          1) Modify bigtop.ml with those variables:

          PHOENIX_BASE_VERSION=4.0.1
          PHOENIX_PKG_VERSION=4.0.1

          Also, since 4.0.1 is not yet available on the apache mirror, we had to download he source from GitHub, rename the directory to phoenix-4.0.1-src, packed it with tar –czf phoenix-4.0.1-src.tar.gz phoenix-4.0.1-src and put the file into the BigTop dl directory.

          After that make phoenix-rpm worked.

          I think the only thing missing to get it compiled correctly is the src file into the Apache site.

          -Dan

          Show
          jmspaggi Jean-Marc Spaggiari added a comment - To get Phoenix into BigTop build we had to do those steps. 1) Modify bigtop.ml with those variables: PHOENIX_BASE_VERSION=4.0.1 PHOENIX_PKG_VERSION=4.0.1 Also, since 4.0.1 is not yet available on the apache mirror, we had to download he source from GitHub, rename the directory to phoenix-4.0.1-src, packed it with tar –czf phoenix-4.0.1-src.tar.gz phoenix-4.0.1-src and put the file into the BigTop dl directory. After that make phoenix-rpm worked. I think the only thing missing to get it compiled correctly is the src file into the Apache site. -Dan
          Hide
          apurtell Andrew Purtell added a comment -

          So this issue is only about updating the BOM to specify Phoenix 4.0.1. Renamed. Waiting for Phoenix 4.0.1 release.

          Show
          apurtell Andrew Purtell added a comment - So this issue is only about updating the BOM to specify Phoenix 4.0.1. Renamed. Waiting for Phoenix 4.0.1 release.
          Hide
          jamestaylor James Taylor added a comment -

          We just released Phoenix 4.1.0 which has many bug fixes and improvements over 4.0: https://blogs.apache.org/phoenix/entry/announcing_phoenix_3_1_and. How about if Bigtop links to that one?

          Show
          jamestaylor James Taylor added a comment - We just released Phoenix 4.1.0 which has many bug fixes and improvements over 4.0: https://blogs.apache.org/phoenix/entry/announcing_phoenix_3_1_and . How about if Bigtop links to that one?
          Hide
          jmspaggi Jean-Marc Spaggiari added a comment -

          I can for sure give it a try!

          I will update the build script, test, and report...

          Show
          jmspaggi Jean-Marc Spaggiari added a comment - I can for sure give it a try! I will update the build script, test, and report...
          Hide
          apurtell Andrew Purtell added a comment -

          Update description: 4.1.0

          Show
          apurtell Andrew Purtell added a comment - Update description: 4.1.0
          Hide
          jmspaggi Jean-Marc Spaggiari added a comment -

          Getting a error with trying to build with 4.1.0.

          + /bin/tar -xvvf /root/bigtop-0.8/bigtop/build/phoenix/rpm/SOURCES/phoenix-4.1.0-src.tar.gz
          /bin/tar: This does not look like a tar archive
          

          So I did have to manually download the source to the dl folder in addition to setting the version in bigtop.mk.

          I think 4.1.0 has to be pushed somewhere to be available in the build?

          Show
          jmspaggi Jean-Marc Spaggiari added a comment - Getting a error with trying to build with 4.1.0. + /bin/tar -xvvf /root/bigtop-0.8/bigtop/build/phoenix/rpm/SOURCES/phoenix-4.1.0-src.tar.gz /bin/tar: This does not look like a tar archive So I did have to manually download the source to the dl folder in addition to setting the version in bigtop.mk. I think 4.1.0 has to be pushed somewhere to be available in the build?
          Hide
          rvs Roman Shaposhnik added a comment -

          Jean-Marc Spaggiari quick question: are we done with this JIRA or is there anything else that needs to be done?

          Show
          rvs Roman Shaposhnik added a comment - Jean-Marc Spaggiari quick question: are we done with this JIRA or is there anything else that needs to be done?
          Hide
          jmspaggi Jean-Marc Spaggiari added a comment -

          BigTop still can not compile with Phoenix 4.1.0. So I think we can not close this JIRA. Maybe title is not perfect?

          Basically, there is 2 things.

          1) Change 4.1.0 as the default value.
          2) Make it works

          Show
          jmspaggi Jean-Marc Spaggiari added a comment - BigTop still can not compile with Phoenix 4.1.0. So I think we can not close this JIRA. Maybe title is not perfect? Basically, there is 2 things. 1) Change 4.1.0 as the default value. 2) Make it works
          Hide
          jamestaylor James Taylor added a comment -

          Apache Phoenix 4.1.0 is available in maven - does that help? Andrew Purtell?

          Show
          jamestaylor James Taylor added a comment - Apache Phoenix 4.1.0 is available in maven - does that help? Andrew Purtell ?
          Hide
          jmspaggi Jean-Marc Spaggiari added a comment -

          Will retry and report here. Tried yesterday morning but failed.

          Show
          jmspaggi Jean-Marc Spaggiari added a comment - Will retry and report here. Tried yesterday morning but failed.
          Hide
          jamestaylor James Taylor added a comment -

          Jean-Marc Spaggiari - one thing that may or may not be an issue - our maven build will use hadoop1 by default. To build for hadoop2 you need to specify -Dhadoop.profile=2

          Show
          jamestaylor James Taylor added a comment - Jean-Marc Spaggiari - one thing that may or may not be an issue - our maven build will use hadoop1 by default. To build for hadoop2 you need to specify -Dhadoop.profile=2
          Hide
          jmspaggi Jean-Marc Spaggiari added a comment -

          IIRC last BigTop version already has this parameter setup. I think it even need to be -Dhadoop-two.profile=2 or someting like that. will double check...

          Show
          jmspaggi Jean-Marc Spaggiari added a comment - IIRC last BigTop version already has this parameter setup. I think it even need to be -Dhadoop-two.profile=2 or someting like that. will double check...
          Hide
          jmspaggi Jean-Marc Spaggiari added a comment -

          Ok. another thing we need to updatd to be able to build with Phoenix 4.1.0:

          PHOENIX_DOWNLOAD_PATH=/incubator/phoenix/phoenix-$(PHOENIX_BASE_VERSION)/src/
          

          Should be:

          PHOENIX_DOWNLOAD_PATH=/phoenix/phoenix-$(PHOENIX_BASE_VERSION)/src/
          
          Show
          jmspaggi Jean-Marc Spaggiari added a comment - Ok. another thing we need to updatd to be able to build with Phoenix 4.1.0: PHOENIX_DOWNLOAD_PATH=/incubator/phoenix/phoenix-$(PHOENIX_BASE_VERSION)/src/ Should be: PHOENIX_DOWNLOAD_PATH=/phoenix/phoenix-$(PHOENIX_BASE_VERSION)/src/
          Hide
          plinnell Peter Linnell added a comment -

          +1 for the patch, but does anything else need to be done ?

          Show
          plinnell Peter Linnell added a comment - +1 for the patch, but does anything else need to be done ?
          Hide
          apurtell Andrew Purtell added a comment -

          Something I just realized... Why is Phoenix not in the top level POM?

          Show
          apurtell Andrew Purtell added a comment - Something I just realized... Why is Phoenix not in the top level POM?
          Hide
          apurtell Andrew Purtell added a comment -

          Updated patch adds Phoenix core, flume, and pig modules under the Maven dependency management section.

          Show
          apurtell Andrew Purtell added a comment - Updated patch adds Phoenix core, flume, and pig modules under the Maven dependency management section.
          Hide
          cos Konstantin Boudnik added a comment -

          good catch.
          +1

          Show
          cos Konstantin Boudnik added a comment - good catch. +1
          Hide
          jamestaylor James Taylor added a comment -

          Thanks for fixing this up, Andrew Purtell.

          Show
          jamestaylor James Taylor added a comment - Thanks for fixing this up, Andrew Purtell .
          Hide
          apurtell Andrew Purtell added a comment -

          Now that BIGTOP-1432 has been committed this is ready to go in. I tested the DEB build:

          $ gradle phoenix-deb
          [...]
          BUILD SUCCESSFUL
          
          Total time: 2 mins 45.403 secs
          
          Show
          apurtell Andrew Purtell added a comment - Now that BIGTOP-1432 has been committed this is ready to go in. I tested the DEB build: $ gradle phoenix-deb [...] BUILD SUCCESSFUL Total time: 2 mins 45.403 secs
          Hide
          cos Konstantin Boudnik added a comment -

          Please do!

          Show
          cos Konstantin Boudnik added a comment - Please do!
          Hide
          jmspaggi Jean-Marc Spaggiari added a comment -

          Passed for me too with BIGTOP-1420, BIGTOP-1421 and BIGTOP-1432 applied on Debian FR. Good work! Thanks! I wil retry with CentOS tomorrow...

          Show
          jmspaggi Jean-Marc Spaggiari added a comment - Passed for me too with BIGTOP-1420 , BIGTOP-1421 and BIGTOP-1432 applied on Debian FR. Good work! Thanks! I wil retry with CentOS tomorrow...
          Hide
          apurtell Andrew Purtell added a comment -

          Pushed to master and 0.8

          Show
          apurtell Andrew Purtell added a comment - Pushed to master and 0.8
          Hide
          cos Konstantin Boudnik added a comment -

          Actually, 0.8 was an exact replica of the master, so you could've just merge it. And now they have diverged. No worries - I will hard-reset it for now.

          Past 0.8 timeframe I will restart the conversation about moving away from SVN branching model (which overarching on every single project in the Hadoop ecosystem) towards proper git branching model. I was offering it to HBase community, but it didn't got enough ppl excited about it. But I think Bigtop will benefit greatly!

          Show
          cos Konstantin Boudnik added a comment - Actually, 0.8 was an exact replica of the master, so you could've just merge it. And now they have diverged. No worries - I will hard-reset it for now. Past 0.8 timeframe I will restart the conversation about moving away from SVN branching model (which overarching on every single project in the Hadoop ecosystem) towards proper git branching model. I was offering it to HBase community, but it didn't got enough ppl excited about it. But I think Bigtop will benefit greatly!
          Hide
          apurtell Andrew Purtell added a comment - - edited

          I cherry-picked from master back to branch. That's not a ff merge but I believe it's tracked such that filtering with 'git log' will do the right thing. Anyway yeah that is our git practice over in hbase and I shall remember the rules are different here, I can ff if git will do it.

          Show
          apurtell Andrew Purtell added a comment - - edited I cherry-picked from master back to branch. That's not a ff merge but I believe it's tracked such that filtering with 'git log' will do the right thing. Anyway yeah that is our git practice over in hbase and I shall remember the rules are different here, I can ff if git will do it.
          Hide
          cos Konstantin Boudnik added a comment -

          cherry-pick generates new SHA hence there's no way to track where the change has came from. Anyway - I've hard-reset branch-0.8 against the master, so all should be fine now.

          Show
          cos Konstantin Boudnik added a comment - cherry-pick generates new SHA hence there's no way to track where the change has came from. Anyway - I've hard-reset branch-0.8 against the master, so all should be fine now.
          Hide
          cos Konstantin Boudnik added a comment -

          It seems that Phoenix build has a way deeper problems with the dependencies... I am looking at the release build and see a log of this:

          06:23:52  [INFO] ------------------------------------------------------------------------
          06:23:52  [INFO] Building Phoenix - Flume 4.1.0
          06:23:52  [INFO] ------------------------------------------------------------------------
          06:23:52  Downloading: https://repository.apache.org/content/repositories/releases/org/apache/hbase/hbase-server/0.98.4-hadoop2/hbase-server-0.98.4-hadoop2.pom
          
          06:23:53  4/27 KB    
          8/27 KB   
          

          So, clearly Phoenix build doesn't give a damn about HBase version passed via system property. For the scope of 0.8.0 - I am willing to let it go, but it needs to be fixed in the upstream.

          Show
          cos Konstantin Boudnik added a comment - It seems that Phoenix build has a way deeper problems with the dependencies... I am looking at the release build and see a log of this: 06:23:52 [INFO] ------------------------------------------------------------------------ 06:23:52 [INFO] Building Phoenix - Flume 4.1.0 06:23:52 [INFO] ------------------------------------------------------------------------ 06:23:52 Downloading: https://repository.apache.org/content/repositories/releases/org/apache/hbase/hbase-server/0.98.4-hadoop2/hbase-server-0.98.4-hadoop2.pom 06:23:53 4/27 KB 8/27 KB So, clearly Phoenix build doesn't give a damn about HBase version passed via system property. For the scope of 0.8.0 - I am willing to let it go, but it needs to be fixed in the upstream.
          Hide
          apurtell Andrew Purtell added a comment -

          The flume and pig modules came after early changes we made to the Phoenix POM for bigtop builds. Some issues there looks like. Will open an issue for the Phoenix project. The core module, which is the part hosted in HBase, does work as expected.

          Show
          apurtell Andrew Purtell added a comment - The flume and pig modules came after early changes we made to the Phoenix POM for bigtop builds. Some issues there looks like. Will open an issue for the Phoenix project. The core module, which is the part hosted in HBase, does work as expected.
          Hide
          cos Konstantin Boudnik added a comment -

          I see. Thanks for the clarification. In the meanwhile - looks like we are getting outselves an RC!
          http://bigtop01.cloudera.org:8080/view/Releases/job/Bigtop-0.8.0/4/

          Show
          cos Konstantin Boudnik added a comment - I see. Thanks for the clarification. In the meanwhile - looks like we are getting outselves an RC! http://bigtop01.cloudera.org:8080/view/Releases/job/Bigtop-0.8.0/4/

            People

            • Assignee:
              apurtell Andrew Purtell
              Reporter:
              jmspaggi Jean-Marc Spaggiari
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development