Bigtop
  1. Bigtop
  2. BIGTOP-1241

BIGTOP should update protobuf to 2.5 from 2.4.x

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.7.0
    • Fix Version/s: 0.8.0
    • Component/s: build
    • Labels:
      None

      Description

      Currently, Bigtop toolchain still deploy protobuf 2.4.x, later Hadoop 2.2.0 version, hadoop updated protobuf to 2.5 from 2.4.x https://issues.apache.org/jira/i#browse/HADOOP-9845
      so BIGTOP need update protobuf to 2.5 to avoid compile error

      1. BIGTOP-1241.1.patch
        2 kB
        Wenwu Peng
      2. BIGTOP-1241.2.patch
        3 kB
        Wenwu Peng

        Issue Links

          Activity

          Hide
          Konstantin Boudnik added a comment -

          Committed to the master. Thanks Wenwu!

          Show
          Konstantin Boudnik added a comment - Committed to the master. Thanks Wenwu!
          Hide
          Konstantin Boudnik added a comment -

          Patch looks good. +1 and committing shortly. Thanks Wenwu!
          I think we can improve later on by using PPA instead of the direct download.

          Show
          Konstantin Boudnik added a comment - Patch looks good. +1 and committing shortly. Thanks Wenwu! I think we can improve later on by using PPA instead of the direct download.
          Hide
          Wenwu Peng added a comment - - edited

          Can we just use it via Puppet?

          Thanks for your comments, the BIGTOP-1241.2.patch address this comment

          Show
          Wenwu Peng added a comment - - edited Can we just use it via Puppet? Thanks for your comments, the BIGTOP-1241 .2.patch address this comment
          Hide
          Konstantin Boudnik added a comment -

          Pinging Patrick Linehan and Wenwu Peng to see if this can be wrapped up by addressing the concerns above? Thanks!

          Show
          Konstantin Boudnik added a comment - Pinging Patrick Linehan and Wenwu Peng to see if this can be wrapped up by addressing the concerns above? Thanks!
          Hide
          Martin Bukatovic added a comment -

          Konstantin Boudnik ok, thanks for the direction, I have created the BIGTOP-1289

          Show
          Martin Bukatovic added a comment - Konstantin Boudnik ok, thanks for the direction, I have created the BIGTOP-1289
          Hide
          Konstantin Boudnik added a comment -

          Martin Bukatovic, thanks for catching this - completely fell out of my attention span, which is notoriously short

          I think the protobuf is coming our as a transitive dependency of Hadoop libs. I believe we need to update hadoop.version in the top level pom to 2.3.0. Also, other dependencies - like pig, hbase, etc. - have to be brough in-line with the stack BOM. Could you please do it as a separate JIRA?

          Show
          Konstantin Boudnik added a comment - Martin Bukatovic , thanks for catching this - completely fell out of my attention span, which is notoriously short I think the protobuf is coming our as a transitive dependency of Hadoop libs. I believe we need to update hadoop.version in the top level pom to 2.3.0. Also, other dependencies - like pig, hbase, etc. - have to be brough in-line with the stack BOM. Could you please do it as a separate JIRA?
          Hide
          Martin Bukatovic added a comment -

          This should be resolved for bigtop tests as well. Maven downloads protobuf-java-2.4.0a.jar into local maven repository and adds it into classpath of the tests, but it's not directly specified in any test pom file, so I can't fixt it directly. Any ideas how this could be fixed properly? Also: would you like to discuss this issue here or should I create separate JIRA (since this is rather testing issue)?

          Show
          Martin Bukatovic added a comment - This should be resolved for bigtop tests as well. Maven downloads protobuf-java-2.4.0a.jar into local maven repository and adds it into classpath of the tests, but it's not directly specified in any test pom file, so I can't fixt it directly. Any ideas how this could be fixed properly? Also: would you like to discuss this issue here or should I create separate JIRA (since this is rather testing issue)?
          Hide
          Konstantin Boudnik added a comment - - edited

          Just searched around a bit: looks like there's a binary package for the stuff https://launchpad.net/ubuntu/+source/protobuf

          Can we just use it via Puppet?

          Show
          Konstantin Boudnik added a comment - - edited Just searched around a bit: looks like there's a binary package for the stuff https://launchpad.net/ubuntu/+source/protobuf Can we just use it via Puppet?
          Hide
          Konstantin Boudnik added a comment - - edited

          Peter Linnell, any chance the prebuilt dpkg can be done, so then we don't need to build the binaries every time?

          Show
          Konstantin Boudnik added a comment - - edited Peter Linnell , any chance the prebuilt dpkg can be done, so then we don't need to build the binaries every time?
          Hide
          Konstantin Boudnik added a comment -

          The patch looks good - thanks! One small comment - the formatting here

          -      notify {"Ubuntu provides protobuf in repo":}
          +               exec {'/usr/bin/make install':
          +          cwd         => '/usr/local/protobuf-2.5.0',
          

          Another small point: the installed version of protoc is unable to find dynamic libs in /usr/local/lib. Do you think it worth updating the templates for {{ /etc/profile.d/bigtop.sh}}??

          Show
          Konstantin Boudnik added a comment - The patch looks good - thanks! One small comment - the formatting here - notify {"Ubuntu provides protobuf in repo":} + exec {'/usr/bin/make install': + cwd => '/usr/local/protobuf-2.5.0', Another small point: the installed version of protoc is unable to find dynamic libs in /usr/local/lib . Do you think it worth updating the templates for {{ /etc/profile.d/bigtop.sh}}??
          Hide
          Wenwu Peng added a comment -

          provide a patch to download/build/install process into the bigtop_toolchain in Ubuntu

          Show
          Wenwu Peng added a comment - provide a patch to download/build/install process into the bigtop_toolchain in Ubuntu
          Hide
          Konstantin Boudnik added a comment -

          I was looking on 12.04 - its standard PPAs don't have the 2.5 so far.
          If you can provide a package for Ubuntu - it'd be great!

          Show
          Konstantin Boudnik added a comment - I was looking on 12.04 - its standard PPAs don't have the 2.5 so far. If you can provide a package for Ubuntu - it'd be great!
          Hide
          Peter Linnell added a comment -

          Maybe I can get it built as a deb for Ubuntu. Which versions of Ubuntu are missing 2.5 ?

          Show
          Peter Linnell added a comment - Maybe I can get it built as a deb for Ubuntu. Which versions of Ubuntu are missing 2.5 ?
          Hide
          Konstantin Boudnik added a comment -

          Also, it seems that Ubuntu doesn't have a package for 2.5 version yet. Shall we implement the download/build/install process into the bigtop_toolchain's recipe?

          Show
          Konstantin Boudnik added a comment - Also, it seems that Ubuntu doesn't have a package for 2.5 version yet. Shall we implement the download/build/install process into the bigtop_toolchain's recipe?
          Hide
          Konstantin Boudnik added a comment -

          I have ran latest bigtop_toolchain on a centos host and got protobuf 2.5 as the result. I think we can close this now

          Show
          Konstantin Boudnik added a comment - I have ran latest bigtop_toolchain on a centos host and got protobuf 2.5 as the result. I think we can close this now
          Hide
          Konstantin Boudnik added a comment -

          Thanks a bunch, Peter!

          Show
          Konstantin Boudnik added a comment - Thanks a bunch, Peter!
          Hide
          Peter Linnell added a comment -

          Cos, Exactly. That said, I've not tested these packages against a Bigtop build run, so I would test it on a separate build job on bigtop jenkins before switching over the toolchain.

          I'll try to punch out 2.5 debs by Monday or Tuesday latest.

          Show
          Peter Linnell added a comment - Cos, Exactly. That said, I've not tested these packages against a Bigtop build run, so I would test it on a separate build job on bigtop jenkins before switching over the toolchain. I'll try to punch out 2.5 debs by Monday or Tuesday latest.
          Hide
          Konstantin Boudnik added a comment -

          Peter, I am looking into SUSE repo and 2.5 bits are there. So, I believe it all it takes to make out bigtop_toolchain to pick it up.

          Show
          Konstantin Boudnik added a comment - Peter, I am looking into SUSE repo and 2.5 bits are there. So, I believe it all it takes to make out bigtop_toolchain to pick it up.
          Hide
          Peter Linnell added a comment -

          When all the distros have 2.5.x compiled in my repo and the docs are updated, I will resolve this.

          Show
          Peter Linnell added a comment - When all the distros have 2.5.x compiled in my repo and the docs are updated, I will resolve this.
          Hide
          Wenwu Peng added a comment -

          is there any repositories have protobuf 2.5 RPMs?
          can't get the protobuf 2.5 from http://download.opensuse.org/repositories

          Show
          Wenwu Peng added a comment - is there any repositories have protobuf 2.5 RPMs? can't get the protobuf 2.5 from http://download.opensuse.org/repositories

            People

            • Assignee:
              Wenwu Peng
              Reporter:
              Wenwu Peng
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development