Hadoop Common
  1. Hadoop Common
  2. HADOOP-8133

upgrade commons-daemon.version in hadoop-project/pom.xml from 1.0.3 to 1.0.5 to avoid jsvc / libcap.so linking problem

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Not a Problem
    • Affects Version/s: 0.23.0
    • Fix Version/s: None
    • Component/s: build, native
    • Labels:
      None

      Description

      On several recent Linux distributions (tested Ubuntu 9.04, RHEL 6.2, Amazon Linux AMI release 2011.09), there is no libcap.so.1. With commons-daemon 1.0.3, the current version selected by hadoop-project/pom.xml, this causes a linking error:

      $ wget http://archive.apache.org/dist/commons/daemon/binaries/1.0.3/linux/commons-daemon-1.0.3-bin-linux-x86_64.tar.gz
      $ tar xfz commons-daemon-1.0.3-bin-linux-x86_64.tar.gz
      $ ldd jsvc 
      	linux-vdso.so.1 =>  (0x00007ffff43cc000)
      	libdl.so.2 => /lib64/libdl.so.2 (0x00007f9cbbeee000)
      	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f9cbbcd2000)
      	libcap.so.1 => not found
      	libc.so.6 => /lib64/libc.so.6 (0x00007f9cbb930000)
      	/lib64/ld-linux-x86-64.so.2 (0x00007f9cbc0fb000)
      

      The same problem also occurs with 1.0.4, but 1.0.5 doesn't have the problem:

      $ wget http://archive.apache.org/dist/commons/daemon/binaries/1.0.5/linux/commons-daemon-1.0.5-bin-linux-x86_64.tar.gz
      $ tar xfz commons-daemon-1.0.5-bin-linux-x86_64.tar.gz 
      $ ldd jsvc 
      	linux-vdso.so.1 =>  (0x00007fff157ff000)
      	libdl.so.2 => /lib64/libdl.so.2 (0x00007f9e14778000)
      	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f9e1455c000)
      	libcap.so.2 => /lib64/libcap.so.2 (0x00007f9e14357000)
      	libc.so.6 => /lib64/libc.so.6 (0x00007f9e13fb6000)
      	/lib64/ld-linux-x86-64.so.2 (0x00007f9e14985000)
      	libattr.so.1 => /lib64/libattr.so.1 (0x00007f9e13db2000)
      
      1. HADOOP-8133.patch
        0.5 kB
        Eugene Koontz

        Issue Links

          Activity

          Show
          Eugene Koontz added a comment - See also https://issues.apache.org/jira/secure/IssueNavigator!executeAdvanced.jspa?jqlQuery=%28summary+~+%27libcap%27+OR+summary+~+%27libcap.%27%29+AND+project%3D%27DAEMON%27&runQuery=true&clear=true
          Hide
          Roman Shaposhnik added a comment -

          Personally, I'm -0 on this patch. Two points:

          1. by moving to the one requiring libcap.so.2 you're killing off the distros where that is not available. I'm pretty sure at least CentOS/RHEL5 fall into that category and given how widespread they are in the enterprise I'm not sure you want to do that to them.
          2. I thought the agreement was to KILL jsvc bundling to begin with. See HDFS-2303 for details. This is a much cleaner solution and you have a wide choice of what jsvc to depend upon ranging from bundled with your system (http://pkgs.org/search/?keyword=jsvc) all the way to the one we're curating in Bigtop (bigtop-jsvc package).
          Show
          Roman Shaposhnik added a comment - Personally, I'm -0 on this patch. Two points: by moving to the one requiring libcap.so.2 you're killing off the distros where that is not available. I'm pretty sure at least CentOS/RHEL5 fall into that category and given how widespread they are in the enterprise I'm not sure you want to do that to them. I thought the agreement was to KILL jsvc bundling to begin with. See HDFS-2303 for details. This is a much cleaner solution and you have a wide choice of what jsvc to depend upon ranging from bundled with your system ( http://pkgs.org/search/?keyword=jsvc ) all the way to the one we're curating in Bigtop (bigtop-jsvc package).
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12516855/HADOOP-8133.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/660//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12516855/HADOOP-8133.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/660//console This message is automatically generated.
          Hide
          Eugene Koontz added a comment -

          Hi Roman, I set HADOOP-8133 to "is superceded by" HDFS-2303. Thanks to bringing it to my attention. It does seem like the better solution.

          Show
          Eugene Koontz added a comment - Hi Roman, I set HADOOP-8133 to "is superceded by" HDFS-2303 . Thanks to bringing it to my attention. It does seem like the better solution.
          Hide
          Eugene Koontz added a comment -

          Roman, with regard to your first point about an upgrade breaking older distributions, I think the daemon is able to choose the right version depending on the platform (at least that is the sense I get from reading DAEMON-234). I will look on Amazon for an older CentOS image to test this assumption with.

          Show
          Eugene Koontz added a comment - Roman, with regard to your first point about an upgrade breaking older distributions, I think the daemon is able to choose the right version depending on the platform (at least that is the sense I get from reading DAEMON-234 ). I will look on Amazon for an older CentOS image to test this assumption with.
          Hide
          Eugene Koontz added a comment -

          Hi Roman,
          You are right: commons-daemon 1.0.5 will not run on a libcap.so.1 distribution (tested Centos 5.6):

          
          [root@ip-10-168-186-206 ~]# wget http://archive.apache.org/dist/commons/daemon/binaries/1.0.3/linux/commons-daemon-1.0.3-bin-linux-x86_64.tar.gz
          --2012-03-02 12:59:50--  http://archive.apache.org/dist/commons/daemon/binaries/1.0.3/linux/commons-daemon-1.0.3-bin-linux-x86_64.tar.gz
          Resolving archive.apache.org... 140.211.11.131
          Connecting to archive.apache.org|140.211.11.131|:80... connected.
          HTTP request sent, awaiting response... 200 OK
          Length: 26431 (26K) [application/x-gzip]
          Saving to: `commons-daemon-1.0.3-bin-linux-x86_64.tar.gz'
          
          100%[======================================>] 26,431      --.-K/s   in 0.06s   
          
          2012-03-02 12:59:50 (437 KB/s) - `commons-daemon-1.0.3-bin-linux-x86_64.tar.gz' saved [26431/26431]
          
          [root@ip-10-168-186-206 ~]# tar xfz commons-daemon-1.0.3-bin-linux-x86_64.tar.gz 
          [root@ip-10-168-186-206 ~]# ldd jsvc
          	linux-vdso.so.1 =>  (0x00007fffc4dfd000)
          	libdl.so.2 => /lib64/libdl.so.2 (0x00002b36b7cb3000)
          	libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b36b7eb7000)
          	libcap.so.1 => /lib64/libcap.so.1 (0x00002b36b80d2000)
          	libc.so.6 => /lib64/libc.so.6 (0x00002b36b82d7000)
          	/lib64/ld-linux-x86-64.so.2 (0x00002b36b7a95000)
          [root@ip-10-168-186-206 ~]# wget http://archive.apache.org/dist/commons/daemon/binaries/1.0.5/linux/commons-daemon-1.0.5-bin-linux-x86_64.tar.gz
          --2012-03-02 13:00:18--  http://archive.apache.org/dist/commons/daemon/binaries/1.0.5/linux/commons-daemon-1.0.5-bin-linux-x86_64.tar.gz
          Resolving archive.apache.org... 140.211.11.131
          Connecting to archive.apache.org|140.211.11.131|:80... connected.
          HTTP request sent, awaiting response... 200 OK
          Length: 27631 (27K) [application/x-gzip]
          Saving to: `commons-daemon-1.0.5-bin-linux-x86_64.tar.gz'
          
          100%[======================================>] 27,631      --.-K/s   in 0.07s   
          
          2012-03-02 13:00:18 (399 KB/s) - `commons-daemon-1.0.5-bin-linux-x86_64.tar.gz' saved [27631/27631]
          
          [root@ip-10-168-186-206 ~]# tar xfz commons-daemon-1.0.5-bin-linux-x86_64.tar.gz 
          [root@ip-10-168-186-206 ~]# ldd jsvc	not a dynamic executable
          [root@ip-10-168-186-206 ~]# ./jsvc 
          ./jsvc: error while loading shared libraries: libcap.so.2: cannot open shared object file: No such file or directory
          [root@ip-10-168-186-206 ~]# cat /etc/issue
          CentOS release 5.6 (Final)
          Kernel \r on an \m
          
          [root@ip-10-168-186-206 ~]# uname -a
          Linux ip-10-168-186-206 2.6.18-238.19.1.el5.centos.plusxen #1 SMP Mon Jul 18 11:02:46 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux
          [root@ip-10-168-186-206 ~]# 
          
          Show
          Eugene Koontz added a comment - Hi Roman, You are right: commons-daemon 1.0.5 will not run on a libcap.so.1 distribution (tested Centos 5.6): [root@ip-10-168-186-206 ~]# wget http: //archive.apache.org/dist/commons/daemon/binaries/1.0.3/linux/commons-daemon-1.0.3-bin-linux-x86_64.tar.gz --2012-03-02 12:59:50-- http: //archive.apache.org/dist/commons/daemon/binaries/1.0.3/linux/commons-daemon-1.0.3-bin-linux-x86_64.tar.gz Resolving archive.apache.org... 140.211.11.131 Connecting to archive.apache.org|140.211.11.131|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 26431 (26K) [application/x-gzip] Saving to: `commons-daemon-1.0.3-bin-linux-x86_64.tar.gz' 100%[======================================>] 26,431 --.-K/s in 0.06s 2012-03-02 12:59:50 (437 KB/s) - `commons-daemon-1.0.3-bin-linux-x86_64.tar.gz' saved [26431/26431] [root@ip-10-168-186-206 ~]# tar xfz commons-daemon-1.0.3-bin-linux-x86_64.tar.gz [root@ip-10-168-186-206 ~]# ldd jsvc linux-vdso.so.1 => (0x00007fffc4dfd000) libdl.so.2 => /lib64/libdl.so.2 (0x00002b36b7cb3000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b36b7eb7000) libcap.so.1 => /lib64/libcap.so.1 (0x00002b36b80d2000) libc.so.6 => /lib64/libc.so.6 (0x00002b36b82d7000) /lib64/ld-linux-x86-64.so.2 (0x00002b36b7a95000) [root@ip-10-168-186-206 ~]# wget http: //archive.apache.org/dist/commons/daemon/binaries/1.0.5/linux/commons-daemon-1.0.5-bin-linux-x86_64.tar.gz --2012-03-02 13:00:18-- http: //archive.apache.org/dist/commons/daemon/binaries/1.0.5/linux/commons-daemon-1.0.5-bin-linux-x86_64.tar.gz Resolving archive.apache.org... 140.211.11.131 Connecting to archive.apache.org|140.211.11.131|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 27631 (27K) [application/x-gzip] Saving to: `commons-daemon-1.0.5-bin-linux-x86_64.tar.gz' 100%[======================================>] 27,631 --.-K/s in 0.07s 2012-03-02 13:00:18 (399 KB/s) - `commons-daemon-1.0.5-bin-linux-x86_64.tar.gz' saved [27631/27631] [root@ip-10-168-186-206 ~]# tar xfz commons-daemon-1.0.5-bin-linux-x86_64.tar.gz [root@ip-10-168-186-206 ~]# ldd jsvc not a dynamic executable [root@ip-10-168-186-206 ~]# ./jsvc ./jsvc: error while loading shared libraries: libcap.so.2: cannot open shared object file: No such file or directory [root@ip-10-168-186-206 ~]# cat /etc/issue CentOS release 5.6 (Final) Kernel \r on an \m [root@ip-10-168-186-206 ~]# uname -a Linux ip-10-168-186-206 2.6.18-238.19.1.el5.centos.plusxen #1 SMP Mon Jul 18 11:02:46 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux [root@ip-10-168-186-206 ~]#
          Hide
          Harsh J added a comment -

          Since HDFS-2303 is done, this is no longer an issue. Thanks for the patch effort though, Eugene!

          Show
          Harsh J added a comment - Since HDFS-2303 is done, this is no longer an issue. Thanks for the patch effort though, Eugene!

            People

            • Assignee:
              Unassigned
              Reporter:
              Eugene Koontz
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development