Hadoop Common
  1. Hadoop Common
  2. HADOOP-9164

Print paths of loaded native libraries in NativeLibraryChecker

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 2.0.2-alpha
    • Fix Version/s: 2.1.0-beta
    • Component/s: native
    • Labels:
      None
    1. HADOOP-9164.v1.patch
      14 kB
      Binglin Chang
    2. HADOOP-9164.v2.patch
      13 kB
      Binglin Chang
    3. HADOOP-9164.v3.patch
      13 kB
      Binglin Chang
    4. HADOOP-9164.v4.2.patch
      13 kB
      Binglin Chang
    5. HADOOP-9164.v4.patch
      13 kB
      Binglin Chang
    6. HADOOP-9164.v4.patch
      13 kB
      Binglin Chang
    7. HADOOP-9164.v5.patch
      16 kB
      Binglin Chang
    8. HADOOP-9164.v6.patch
      17 kB
      Binglin Chang

      Issue Links

        Activity

        Hide
        Binglin Chang added a comment -

        Here is the initial patch, I have run NativeLibraryChecker, and here is the output:
        Native library checking:
        hadoop: true hadoop 1.0.0
        zlib: true libz.so.1
        snappy: true libsnappy.so.1
        lz4: true [bundled:revision:43]

        Show
        Binglin Chang added a comment - Here is the initial patch, I have run NativeLibraryChecker, and here is the output: Native library checking: hadoop: true hadoop 1.0.0 zlib: true libz.so.1 snappy: true libsnappy.so.1 lz4: true [bundled:revision:43]
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12562479/HADOOP-9164.v1.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1929//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1929//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12562479/HADOOP-9164.v1.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1929//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1929//console This message is automatically generated.
        Hide
        Colin Patrick McCabe added a comment -

        Hi Binglin,

        Thanks for looking into this-- and also for doing HADOOP-9162. I think these will be helpful.

        I don't think C/C++ library version numbers are the most interesting things to report, though. The reality is, we very seldom increment those numbers, so just knowing that you're using libhadoop-1.0.0 doesn't give you much information (there were never any other version numbers for that library :\ )

        Reporting the full path of the libhadoop.so file would be very useful, however. There is some information about how to get that information here: http://stackoverflow.com/questions/1681060/library-path-when-dynamically-loaded.

        Show
        Colin Patrick McCabe added a comment - Hi Binglin, Thanks for looking into this-- and also for doing HADOOP-9162 . I think these will be helpful. I don't think C/C++ library version numbers are the most interesting things to report, though. The reality is, we very seldom increment those numbers, so just knowing that you're using libhadoop-1.0.0 doesn't give you much information (there were never any other version numbers for that library :\ ) Reporting the full path of the libhadoop.so file would be very useful, however. There is some information about how to get that information here: http://stackoverflow.com/questions/1681060/library-path-when-dynamically-loaded .
        Hide
        Binglin Chang added a comment -

        Thanks for review and tips. Glad to learn a new magic function dladdr
        I'll update the patch soon.

        Show
        Binglin Chang added a comment - Thanks for review and tips. Glad to learn a new magic function dladdr I'll update the patch soon.
        Hide
        Binglin Chang added a comment -

        Here is the output of NativeLibraryChecker when invoked in test:
        Native library checking:
        hadoop: true /home/decster/hadoop-trunk/hadoop-common-project/hadoop-common/target/native/target/usr/local/lib/libhadoop.so.1.0.0
        zlib: true /lib/x86_64-linux-gnu/libz.so.1
        snappy: true /usr/lib/libsnappy.so.1
        lz4: true [bundled:revision:43]

        Show
        Binglin Chang added a comment - Here is the output of NativeLibraryChecker when invoked in test: Native library checking: hadoop: true /home/decster/hadoop-trunk/hadoop-common-project/hadoop-common/target/native/target/usr/local/lib/libhadoop.so.1.0.0 zlib: true /lib/x86_64-linux-gnu/libz.so.1 snappy: true /usr/lib/libsnappy.so.1 lz4: true [bundled:revision:43]
        Hide
        Binglin Chang added a comment -

        And here is the output of NativeLIbraryChecker when invoked using bin/hadoop
        decster@localhost:~/hadoop-trunk/hadoop-common-project/hadoop-common/target/hadoop-common-3.0.0-SNAPSHOT$ bin/hadoop checknative
        12/12/27 21:44:23 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
        Native library checking:
        hadoop: true /home/decster/hadoop-trunk/hadoop-common-project/hadoop-common/target/hadoop-common-3.0.0-SNAPSHOT/lib/native/libhadoop.so.1.0.0
        zlib: true /lib/x86_64-linux-gnu/libz.so.1
        snappy: true /usr/lib/libsnappy.so.1
        lz4: true [bundled:revision:43]

        Show
        Binglin Chang added a comment - And here is the output of NativeLIbraryChecker when invoked using bin/hadoop decster@localhost:~/hadoop-trunk/hadoop-common-project/hadoop-common/target/hadoop-common-3.0.0-SNAPSHOT$ bin/hadoop checknative 12/12/27 21:44:23 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library Native library checking: hadoop: true /home/decster/hadoop-trunk/hadoop-common-project/hadoop-common/target/hadoop-common-3.0.0-SNAPSHOT/lib/native/libhadoop.so.1.0.0 zlib: true /lib/x86_64-linux-gnu/libz.so.1 snappy: true /usr/lib/libsnappy.so.1 lz4: true [bundled:revision:43]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12562556/HADOOP-9164.v2.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common:

        org.apache.hadoop.util.TestNativeCodeLoader

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1930//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1930//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12562556/HADOOP-9164.v2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common: org.apache.hadoop.util.TestNativeCodeLoader +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1930//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1930//console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12562585/HADOOP-9164.v3.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1933//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1933//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12562585/HADOOP-9164.v3.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1933//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1933//console This message is automatically generated.
        Hide
        Allen Wittenauer added a comment -

        I don't think C/C++ library version numbers are the most interesting things to report, though. The reality is, we very seldom increment those numbers, so just knowing that you're using libhadoop-1.0.0 doesn't give you much information (there were never any other version numbers for that library :\ )

        This should probably get fixed as part of this patch.

        Show
        Allen Wittenauer added a comment - I don't think C/C++ library version numbers are the most interesting things to report, though. The reality is, we very seldom increment those numbers, so just knowing that you're using libhadoop-1.0.0 doesn't give you much information (there were never any other version numbers for that library :\ ) This should probably get fixed as part of this patch.
        Hide
        Colin Patrick McCabe added a comment -

        Hi Allen Wittenauer

        Why not open a discussion on hadoop-dev about libhadoop.so version numbers? I don't think this JIRA is the right place to sort that out. This JIRA is simply an enhancement to NativeLibraryChecker.

        Hi Binglin Chang

        Looks good. I'll check this out more tomorrow.

        Show
        Colin Patrick McCabe added a comment - Hi Allen Wittenauer Why not open a discussion on hadoop-dev about libhadoop.so version numbers? I don't think this JIRA is the right place to sort that out. This JIRA is simply an enhancement to NativeLibraryChecker . Hi Binglin Chang Looks good. I'll check this out more tomorrow.
        Hide
        Allen Wittenauer added a comment -

        Why not open a discussion on hadoop-dev about libhadoop.so version numbers?

        I'm not sure what value there is in doing that. The version numbers are completely fictional, confusing, and break with ancient UNIX traditions. Just tying them back to the build version a la OpenSSL would be a major step forward since that at least gives some clue about what the compatibility expectations are.

        I don't think this JIRA is the right place to sort that out.

        As I said, I don't think there is anything controversial here. Fixing it should be relatively trivial (just read the generated java file and stuff it into the appropriate #define's and make vars) and would allow the version of the patch that reports library versions to go forward.

        Show
        Allen Wittenauer added a comment - Why not open a discussion on hadoop-dev about libhadoop.so version numbers? I'm not sure what value there is in doing that. The version numbers are completely fictional, confusing, and break with ancient UNIX traditions. Just tying them back to the build version a la OpenSSL would be a major step forward since that at least gives some clue about what the compatibility expectations are. I don't think this JIRA is the right place to sort that out. As I said, I don't think there is anything controversial here. Fixing it should be relatively trivial (just read the generated java file and stuff it into the appropriate #define's and make vars) and would allow the version of the patch that reports library versions to go forward.
        Hide
        Colin Patrick McCabe added a comment -
          if ((void *)dlsym_snappy_compress != NULL) {
            ret = dladdr(
                (void *)dlsym_snappy_compress,
                &dl_info);
          }
          return (*env)->NewStringUTF(env, ret==0?HADOOP_SNAPPY_LIBRARY:dl_info.dli_fname);
        

        I think this could be simplified a bit by returning a new string based on HADOOP_SNAPPY_LIBRARY inside the if statement. Also, there isn't any need to cast pointers to void*, or explicitly write != NULL. You can simply write this:

          if (dlsym_snappy_compress) {
            ...
          }
        
        Show
        Colin Patrick McCabe added a comment - if ((void *)dlsym_snappy_compress != NULL) { ret = dladdr( (void *)dlsym_snappy_compress, &dl_info); } return (*env)->NewStringUTF(env, ret==0?HADOOP_SNAPPY_LIBRARY:dl_info.dli_fname); I think this could be simplified a bit by returning a new string based on HADOOP_SNAPPY_LIBRARY inside the if statement. Also, there isn't any need to cast pointers to void* , or explicitly write != NULL . You can simply write this: if (dlsym_snappy_compress) { ... }
        Hide
        Binglin Chang added a comment -

        Thanks for review Colin!
        New version addressing previous comments

        Show
        Binglin Chang added a comment - Thanks for review Colin! New version addressing previous comments
        Hide
        Binglin Chang added a comment -

        Seems jenkins have not started, resubmit

        Show
        Binglin Chang added a comment - Seems jenkins have not started, resubmit
        Hide
        Binglin Chang added a comment -

        Jenkins has some issues recently, resubmit

        Show
        Binglin Chang added a comment - Jenkins has some issues recently, resubmit
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12563707/HADOOP-9164.v4.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2008//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2008//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12563707/HADOOP-9164.v4.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2008//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2008//console This message is automatically generated.
        Hide
        Arpit Agarwal added a comment -

        Hi Binglin Chang, thanks for the patch. It looks good overall.

        A few comments.

        NativeLibraryChecker.java:

        • Line 61-63: "null" as the default could be confused with printing a null object. Consider using "Unavailable"?

        SnappyCompressor.c:

        • Line 121: Cast to void* is superfluous.
        • Line 121-123: The clause fits on one line.

        ZlibCompressor.c:

        • Ditto

        NativeCodeLoader.c:

        • Line 42: Formatting - there should be spaces before and after the ternary operands.
        • Line 42: "null" as the default could be confused with printing a null object. Consider using "Unavailable"?

        Lz4Compressor.c:

        • I did not understand the meaning of "[bundled:revision:43]".
        Show
        Arpit Agarwal added a comment - Hi Binglin Chang , thanks for the patch. It looks good overall. A few comments. NativeLibraryChecker.java: Line 61-63: "null" as the default could be confused with printing a null object. Consider using "Unavailable"? SnappyCompressor.c: Line 121: Cast to void* is superfluous. Line 121-123: The clause fits on one line. ZlibCompressor.c: Ditto NativeCodeLoader.c: Line 42: Formatting - there should be spaces before and after the ternary operands. Line 42: "null" as the default could be confused with printing a null object. Consider using "Unavailable"? Lz4Compressor.c: I did not understand the meaning of " [bundled:revision:43] ".
        Hide
        Arpit Agarwal added a comment -

        (Feedback as non-committer)

        Show
        Arpit Agarwal added a comment - (Feedback as non-committer)
        Hide
        Owen O'Malley added a comment -

        I agree with Allen that we should use the Hadoop release number on all of the binary libraries that go with the release. Once that is true, reporting them is a very good idea.

        Show
        Owen O'Malley added a comment - I agree with Allen that we should use the Hadoop release number on all of the binary libraries that go with the release. Once that is true, reporting them is a very good idea.
        Hide
        Colin Patrick McCabe added a comment -

        Hi Owen,

        I tend to agree with you that libhadoop.so should have the same version number as the jars do. It's just more consistent. Why don't we open up a separate JIRA just for that? It sounds like a small change to you or I, but in reality it involves pulling in the bigtop people (who rely on the current naming scheme) and a bunch of other folks.

        On the other hand, printing out the full path to the library in NativeLibraryChecker is just an HDFS-internal issue which we can easily do. There's no reason why the two changes should be coupled, and I feel like the request to do so is blocking progress.

        It's pretty common to have multiple copies of libhadoop.so on the same node. For me personally, knowing the full path to the library actually being used is much more useful than the library version number. Even if we redefine the libhadoop.so version number to be the same as the release number, that just means I will have a probably always be set to 3.0.0-SNAPSHOT or the CDH version number. I could have known that I was using 3.0.0-SNAPSHOT just by looking at the name of the jars in my path.

        Show
        Colin Patrick McCabe added a comment - Hi Owen, I tend to agree with you that libhadoop.so should have the same version number as the jars do. It's just more consistent. Why don't we open up a separate JIRA just for that? It sounds like a small change to you or I, but in reality it involves pulling in the bigtop people (who rely on the current naming scheme) and a bunch of other folks. On the other hand, printing out the full path to the library in NativeLibraryChecker is just an HDFS-internal issue which we can easily do. There's no reason why the two changes should be coupled, and I feel like the request to do so is blocking progress. It's pretty common to have multiple copies of libhadoop.so on the same node. For me personally, knowing the full path to the library actually being used is much more useful than the library version number. Even if we redefine the libhadoop.so version number to be the same as the release number, that just means I will have a probably always be set to 3.0.0-SNAPSHOT or the CDH version number. I could have known that I was using 3.0.0-SNAPSHOT just by looking at the name of the jars in my path.
        Hide
        Sanjay Radia added a comment -

        Colins

        • I believe folks are fine about the full path name. Binglin's has posted a comment above which shows that the command displays full path as you have requested.
        • I am tempted to have this jira set the Hadoop release number on the binaries that go with it.
        Show
        Sanjay Radia added a comment - Colins I believe folks are fine about the full path name. Binglin's has posted a comment above which shows that the command displays full path as you have requested. I am tempted to have this jira set the Hadoop release number on the binaries that go with it.
        Hide
        Colin Patrick McCabe added a comment -

        If everyone agrees that we should set the Hadoop release number on the native libraries, I'm fine with doing it in this JIRA. It would be nice to move beyond version 1.0.0 and 0.0.0 at long last

        Show
        Colin Patrick McCabe added a comment - If everyone agrees that we should set the Hadoop release number on the native libraries, I'm fine with doing it in this JIRA. It would be nice to move beyond version 1.0.0 and 0.0.0 at long last
        Hide
        Sanjay Radia added a comment -

        If everyone agrees that we should set the Hadoop release number on the native libraries ..

        Colin, I think you were the last hold out against that ... lets move forward with that in this jira.

        Show
        Sanjay Radia added a comment - If everyone agrees that we should set the Hadoop release number on the native libraries .. Colin, I think you were the last hold out against that ... lets move forward with that in this jira.
        Hide
        Binglin Chang added a comment -

        One question about set hadoop release number:
        suppose current version is 3.0.0-SNAPSHOT, so the native library should be libhadoop.so.3.0.0-SNAPSHOT or just libhadoop.so.3.0.0?
        I am not sure linux like -SNAPSHOT suffix

        Show
        Binglin Chang added a comment - One question about set hadoop release number: suppose current version is 3.0.0-SNAPSHOT, so the native library should be libhadoop.so.3.0.0-SNAPSHOT or just libhadoop.so.3.0.0? I am not sure linux like -SNAPSHOT suffix
        Hide
        Luke Lu added a comment -

        Release version and so version are different things. The latter is for ABI compatibility, i.e., if we just fixed some bugs and didn't break compatibility in a release, we should not increase the so version, which more similar to our RPC version. The trailing version in lib<name>.so.<version> is supposed to be the so version. If we really want embed a release version in the so file name. We should do something like: lib<name>-<release-version>.so.<so_version>. Also so version should also be set in the ELF header (trivial in cmake) and be consistent with that in file name. For compatibility, a symlink should also be provided.

        Show
        Luke Lu added a comment - Release version and so version are different things. The latter is for ABI compatibility, i.e., if we just fixed some bugs and didn't break compatibility in a release, we should not increase the so version, which more similar to our RPC version. The trailing version in lib<name>.so.<version> is supposed to be the so version. If we really want embed a release version in the so file name. We should do something like: lib<name>-<release-version>.so.<so_version>. Also so version should also be set in the ELF header (trivial in cmake) and be consistent with that in file name. For compatibility, a symlink should also be provided.
        Hide
        Binglin Chang added a comment -

        I did some experiments in CMake, turns out cmake has properties VERSION and SOVERSION:

        SET_TARGET_PROPERTIES(hadoop PROPERTIES
            VERSION "3.0.0-snapshot"
            SOVERSION "1")
        lrwxrwxrwx  1 decster decster    14 2013-07-11 21:05 libhadoop.so -> libhadoop.so.1
        lrwxrwxrwx  1 decster decster    27 2013-07-11 21:05 libhadoop.so.1 -> libhadoop.so.3.0.0-snapshot
        -rwxrwxr-x  1 decster decster  7721 2013-07-11 21:05 libhadoop.so.3.0.0-snapshot
        
        SET_TARGET_PROPERTIES(hadoop PROPERTIES
            VERSION "3.0.0-snapshot"
        )
        lrwxrwxrwx  1 decster decster    27 2013-07-11 22:41 libhadoop.so -> libhadoop.so.3.0.0-snapshot
        -rwxrwxr-x  1 decster decster  7721 2013-07-11 22:41 libhadoop.so.3.0.0-snapshot
        
        SET_TARGET_PROPERTIES(hadoop PROPERTIES
            SOVERSION "1")
        lrwxrwxrwx  1 decster decster    14 2013-07-11 22:43 libhadoop.so -> libhadoop.so.1*
        -rwxrwxr-x  1 decster decster  7721 2013-07-11 22:43 libhadoop.so.1*
        
        

        I am not sure VERSION means "release version",
        if we use lib<name><release-version>.so.<so_version> format, we need to add this to library name, like SET_TARGET_PROPERTIES(hadoop<release-version> PROPERTIES SOVERSION "<so_version>")
        We need some agreement on this, or should I just leave version number change to another JIRA?

        Show
        Binglin Chang added a comment - I did some experiments in CMake, turns out cmake has properties VERSION and SOVERSION: SET_TARGET_PROPERTIES(hadoop PROPERTIES VERSION "3.0.0-snapshot" SOVERSION "1" ) lrwxrwxrwx 1 decster decster 14 2013-07-11 21:05 libhadoop.so -> libhadoop.so.1 lrwxrwxrwx 1 decster decster 27 2013-07-11 21:05 libhadoop.so.1 -> libhadoop.so.3.0.0-snapshot -rwxrwxr-x 1 decster decster 7721 2013-07-11 21:05 libhadoop.so.3.0.0-snapshot SET_TARGET_PROPERTIES(hadoop PROPERTIES VERSION "3.0.0-snapshot" ) lrwxrwxrwx 1 decster decster 27 2013-07-11 22:41 libhadoop.so -> libhadoop.so.3.0.0-snapshot -rwxrwxr-x 1 decster decster 7721 2013-07-11 22:41 libhadoop.so.3.0.0-snapshot SET_TARGET_PROPERTIES(hadoop PROPERTIES SOVERSION "1" ) lrwxrwxrwx 1 decster decster 14 2013-07-11 22:43 libhadoop.so -> libhadoop.so.1* -rwxrwxr-x 1 decster decster 7721 2013-07-11 22:43 libhadoop.so.1* I am not sure VERSION means "release version", if we use lib<name> <release-version>.so.<so_version> format, we need to add this to library name, like SET_TARGET_PROPERTIES(hadoop <release-version> PROPERTIES SOVERSION "<so_version>") We need some agreement on this, or should I just leave version number change to another JIRA?
        Hide
        Luke Lu added a comment -

        There are quite a few different conventions:

        1. http://apr.apache.org/versioning.html
        2. http://tldp.org/HOWTO/Program-Library-HOWTO/shared-libraries.html

        I'm leaning towards the Apache runtime library naming convention even though the trailing version convention is a little unconventional, because it handles include directory naming consistently as well.

        Show
        Luke Lu added a comment - There are quite a few different conventions: http://apr.apache.org/versioning.html http://tldp.org/HOWTO/Program-Library-HOWTO/shared-libraries.html I'm leaning towards the Apache runtime library naming convention even though the trailing version convention is a little unconventional, because it handles include directory naming consistently as well.
        Hide
        Luke Lu added a comment -

        IMO the patch to print the resolved paths of shared libs is already very useful and orthogonal to the naming conventions. I think we should defer the naming/version discussions to another JIRA.

        Show
        Luke Lu added a comment - IMO the patch to print the resolved paths of shared libs is already very useful and orthogonal to the naming conventions. I think we should defer the naming/version discussions to another JIRA.
        Hide
        Binglin Chang added a comment -

        If there is no further comment, I will open a new JIRA to defer version name discussion.
        And here is the new patch addressing @Arpit and @Colin's comments.
        Changes:
        1. add support for bzip2, depending on the config, bzip2 library name shows as "system-native" or actual so path
        2. change [bundled:revision:43] to just revision:43, which means using lz4 revision 43, bundled with libhadoop.so

        Here are local test output:

        hadoop: true /home/decster/hadoop-trunk/hadoop-common-project/hadoop-common/target/native/target/usr/local/lib/libhadoop.so.1.0.0
        zlib:   true /lib/x86_64-linux-gnu/libz.so.1
        snappy: true /usr/lib/libsnappy.so.1
        lz4:    true revision:43
        bzip2:  false system-native
        
        hadoop: true /home/decster/hadoop-trunk/hadoop-common-project/hadoop-common/target/native/target/usr/local/lib/libhadoop.so.1.0.0
        zlib:   true /lib/x86_64-linux-gnu/libz.so.1
        snappy: true /usr/lib/libsnappy.so.1
        lz4:    true revision:43
        bzip2:  true /lib/libbz2.so.1
        
        Show
        Binglin Chang added a comment - If there is no further comment, I will open a new JIRA to defer version name discussion. And here is the new patch addressing @Arpit and @Colin's comments. Changes: 1. add support for bzip2, depending on the config, bzip2 library name shows as "system-native" or actual so path 2. change [bundled:revision:43] to just revision:43, which means using lz4 revision 43, bundled with libhadoop.so Here are local test output: hadoop: true /home/decster/hadoop-trunk/hadoop-common-project/hadoop-common/target/ native /target/usr/local/lib/libhadoop.so.1.0.0 zlib: true /lib/x86_64-linux-gnu/libz.so.1 snappy: true /usr/lib/libsnappy.so.1 lz4: true revision:43 bzip2: false system- native hadoop: true /home/decster/hadoop-trunk/hadoop-common-project/hadoop-common/target/ native /target/usr/local/lib/libhadoop.so.1.0.0 zlib: true /lib/x86_64-linux-gnu/libz.so.1 snappy: true /usr/lib/libsnappy.so.1 lz4: true revision:43 bzip2: true /lib/libbz2.so.1
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12592551/HADOOP-9164.v5.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2790//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2790//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12592551/HADOOP-9164.v5.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2790//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2790//console This message is automatically generated.
        Hide
        Luke Lu added a comment -

        It'd be nice that you print nothing instead of "system-native" if a library is not loaded, as system-native really doesn't make any sense in this case. Otherwise the patch lgtm.

        Show
        Luke Lu added a comment - It'd be nice that you print nothing instead of "system-native" if a library is not loaded, as system-native really doesn't make any sense in this case. Otherwise the patch lgtm.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12592919/HADOOP-9164.v6.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2807//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2807//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12592919/HADOOP-9164.v6.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2807//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2807//console This message is automatically generated.
        Hide
        Luke Lu added a comment -

        The patch lgtm. +1. Will commit shortly.

        Show
        Luke Lu added a comment - The patch lgtm. +1. Will commit shortly.
        Hide
        Colin Patrick McCabe added a comment -

        looks good to me

        Show
        Colin Patrick McCabe added a comment - looks good to me
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in Hadoop-trunk-Commit #4113 (See https://builds.apache.org/job/Hadoop-trunk-Commit/4113/)
        HADOOP-9164. Print paths of loaded native libraries in NativeLibraryChecker. (Binglin Chang via llu) (llu: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1504700)

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/Lz4Codec.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/SnappyCodec.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/bzip2/Bzip2Compressor.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/bzip2/Bzip2Factory.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Compressor.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/snappy/SnappyCompressor.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zlib/ZlibCompressor.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zlib/ZlibFactory.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NativeCodeLoader.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NativeLibraryChecker.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/bzip2/Bzip2Compressor.c
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyCompressor.c
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/NativeCodeLoader.c
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestNativeCodeLoader.java
        Show
        Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #4113 (See https://builds.apache.org/job/Hadoop-trunk-Commit/4113/ ) HADOOP-9164 . Print paths of loaded native libraries in NativeLibraryChecker. (Binglin Chang via llu) (llu: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1504700 ) /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/Lz4Codec.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/SnappyCodec.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/bzip2/Bzip2Compressor.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/bzip2/Bzip2Factory.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Compressor.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/snappy/SnappyCompressor.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zlib/ZlibCompressor.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zlib/ZlibFactory.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NativeCodeLoader.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NativeLibraryChecker.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/bzip2/Bzip2Compressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyCompressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/NativeCodeLoader.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestNativeCodeLoader.java
        Hide
        Luke Lu added a comment -

        Committed to trunk, branch-2, 2.1-beta, and 2.1.0-beta. Thanks Binglin for the patch and Colin for reviewing!

        Show
        Luke Lu added a comment - Committed to trunk, branch-2, 2.1-beta, and 2.1.0-beta. Thanks Binglin for the patch and Colin for reviewing!
        Hide
        Binglin Chang added a comment -

        @Colin and @Luke, Thanks for the reviewing!

        Show
        Binglin Chang added a comment - @Colin and @Luke, Thanks for the reviewing!
        Hide
        German Florez-Larrahondo added a comment -

        Regards
        I've been using this command and it is quite useful.
        However, shouldn't we also be checking for LZO being loaded?

        Thanks
        ./g

        Show
        German Florez-Larrahondo added a comment - Regards I've been using this command and it is quite useful. However, shouldn't we also be checking for LZO being loaded? Thanks ./g
        Hide
        Binglin Chang added a comment -

        Sorry, LZO is not in trunk code base so I can't add related code to LZO, I think it's because LZO is GPL.

        Show
        Binglin Chang added a comment - Sorry, LZO is not in trunk code base so I can't add related code to LZO, I think it's because LZO is GPL.
        Hide
        German Florez-Larrahondo added a comment -

        Understood Binglin

        I don't think there are implications by checking if a GPL library is loaded or not, but not being a lawyer... let's keep it safe as it is for now.

        thanks
        ./g

        Show
        German Florez-Larrahondo added a comment - Understood Binglin I don't think there are implications by checking if a GPL library is loaded or not, but not being a lawyer... let's keep it safe as it is for now. thanks ./g

          People

          • Assignee:
            Binglin Chang
            Reporter:
            Binglin Chang
          • Votes:
            0 Vote for this issue
            Watchers:
            13 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development