Bigtop
  1. Bigtop
  2. BIGTOP-289

hadoop-0.23: need to package native libraries for compression

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.3.0
    • Fix Version/s: 0.4.0
    • Component/s: rpm
    • Labels:
      None

      Description

      I installed hadoop from the hadoop-0.23 branch build.
      There does not appear to be a hadoop-native package.
      I'm not finding the native libraries for compression, although the snappy codec is in the hadoop-common.jar:

      [root@localhost hadoop]# jar -tf hadoop-common.jar | grep -i snappy
      org/apache/hadoop/io/compress/snappy/
      org/apache/hadoop/io/compress/SnappyCodec.class
      org/apache/hadoop/io/compress/snappy/LoadSnappy.class
      org/apache/hadoop/io/compress/snappy/SnappyDecompressor.class
      org/apache/hadoop/io/compress/snappy/SnappyCompressor.class
      org/apache/hadoop/io/compress/snappy/package-info.class

      1. BIGTOP-289.patch.txt
        2 kB
        Roman Shaposhnik
      2. libsnappy.package.tar.gz
        2 kB
        Roman Shaposhnik

        Activity

        Wing Yew Poon created issue -
        Hide
        Roman Shaposhnik added a comment -

        AFAIK, the only library that is missing is snappy. Bigtop relies on OS installed version of snappy, since it seems that modern distros are starting to package it natively.

        Please let me know if anything else is missing. If its just snappy – I'm inclined to close this as will not fix.

        Show
        Roman Shaposhnik added a comment - AFAIK, the only library that is missing is snappy. Bigtop relies on OS installed version of snappy, since it seems that modern distros are starting to package it natively. Please let me know if anything else is missing. If its just snappy – I'm inclined to close this as will not fix.
        Roman Shaposhnik made changes -
        Field Original Value New Value
        Assignee Roman Shaposhnik [ rvs ]
        Hide
        Tom White added a comment -

        I think we need to build Hadoop's native package, which includes Snappy (see the Snappy build instructions in http://svn.apache.org/repos/asf/hadoop/common/trunk/BUILDING.txt).

        Show
        Tom White added a comment - I think we need to build Hadoop's native package, which includes Snappy (see the Snappy build instructions in http://svn.apache.org/repos/asf/hadoop/common/trunk/BUILDING.txt ).
        Hide
        Bruno Mahé added a comment -

        We could build snappy where it's not available, but reuse the native package where available (provided it's a compatible version).

        Show
        Bruno Mahé added a comment - We could build snappy where it's not available, but reuse the native package where available (provided it's a compatible version).
        Hide
        Roman Shaposhnik added a comment -

        @Tom, as Bruno pointed out, we are better off treating libsnappy as its own package. On newer Linux distros that package is provided by the distribution itself. On older – it is not. The question then becomes what Bigtop to do. We can, theoretically, introduce a package for snappy, but that will be somewhat orthogonal to Bigtop's charter of packaging only Apache stuff.

        Show
        Roman Shaposhnik added a comment - @Tom, as Bruno pointed out, we are better off treating libsnappy as its own package. On newer Linux distros that package is provided by the distribution itself. On older – it is not. The question then becomes what Bigtop to do. We can, theoretically, introduce a package for snappy, but that will be somewhat orthogonal to Bigtop's charter of packaging only Apache stuff.
        Roman Shaposhnik made changes -
        Attachment BIGTOP-289.patch.txt [ 12506549 ]
        Roman Shaposhnik made changes -
        Attachment libsnappy.package.tar.gz [ 12506550 ]
        Hide
        Roman Shaposhnik added a comment -

        I'm attaching a patch plus a skeletal package files for libsnappy. This is what we need to build packages for all the supported platforms on something like OpenSUSE build service. Perhaps, we can put them into a bigtop-contrib/libsnappy subdirectory for those who would like to build snappy themselves.

        Thougths?

        Show
        Roman Shaposhnik added a comment - I'm attaching a patch plus a skeletal package files for libsnappy. This is what we need to build packages for all the supported platforms on something like OpenSUSE build service. Perhaps, we can put them into a bigtop-contrib/libsnappy subdirectory for those who would like to build snappy themselves. Thougths?
        Roman Shaposhnik made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Bruno Mahé added a comment -
        • One issue:
          +if [ -z "$BUNDLE_SNAPPY" ] ; then
          + [ -f /usr/lib/libsnappy.so ] && BUNDLE_SNAPPY="-Dsnappy.prefix=x -Dbundle.snappy=true -Dsnappy.lib=/usr/lib"
          + [ -f /usr/lib64/libsnappy.so ] && BUNDLE_SNAPPY="-Dsnappy.prefix=x -Dbundle.snappy=true -Dsnappy.lib=/usr/lib64"
          +fi
          +
          I would reverse the order. So in 64bit platforms, the 64bit library would take precedence over the 32bit one
        • We will need to kill that bundled library in install_hadoop.sh to avoid confusion and complex troubleshooting
        Show
        Bruno Mahé added a comment - One issue: +if [ -z "$BUNDLE_SNAPPY" ] ; then + [ -f /usr/lib/libsnappy.so ] && BUNDLE_SNAPPY="-Dsnappy.prefix=x -Dbundle.snappy=true -Dsnappy.lib=/usr/lib" + [ -f /usr/lib64/libsnappy.so ] && BUNDLE_SNAPPY="-Dsnappy.prefix=x -Dbundle.snappy=true -Dsnappy.lib=/usr/lib64" +fi + I would reverse the order. So in 64bit platforms, the 64bit library would take precedence over the 32bit one We will need to kill that bundled library in install_hadoop.sh to avoid confusion and complex troubleshooting
        Hide
        Bruno Mahé added a comment -

        nevermind my previous comment. it's late

        +1

        Show
        Bruno Mahé added a comment - nevermind my previous comment. it's late +1
        Hide
        Roman Shaposhnik added a comment -

        I've created an OBS Build project for Bigtop based on the SPEC files for snappy:
        https://build.opensuse.org/project/show?project=home%3Abigtop

        Show
        Roman Shaposhnik added a comment - I've created an OBS Build project for Bigtop based on the SPEC files for snappy: https://build.opensuse.org/project/show?project=home%3Abigtop
        Roman Shaposhnik made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Roman Shaposhnik made changes -
        Fix Version/s 0.4.0 [ 12318889 ]
        Gavin made changes -
        Workflow no-reopen-closed, patch-avail [ 12644411 ] patch-available, re-open possible [ 12666039 ]
        Roman Shaposhnik made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        5d 2h 36m 1 Roman Shaposhnik 08/Dec/11 01:00
        Patch Available Patch Available Resolved Resolved
        14d 21h 26m 1 Roman Shaposhnik 22/Dec/11 22:27
        Resolved Resolved Closed Closed
        547d 1h 27m 1 Roman Shaposhnik 22/Jun/13 00:55

          People

          • Assignee:
            Roman Shaposhnik
            Reporter:
            Wing Yew Poon
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development