Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8346

libwebhdfs build fails during link due to unresolved external symbols.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.6.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: native
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      The libwebhdfs build is currently broken due to various unresolved external symbols during link. Multiple patches have introduced a few different forms of this breakage. See comments for full details.

        Issue Links

          Activity

          Hide
          cnauroth Chris Nauroth added a comment -
          • HDFS-573 implemented Windows compatibility for libhdfs. As part of this effort, we refactored OS-specific system calls into OS-specific source files. We also introduced htable.c for a platform-neutral hash table implementation. libwebhdfs includes jni_helper.c from libhdfs in its own build. jni_helper.c depends on the OS-specific sources and the hash table, but since the libwebhdfs build was not updated to include these files, this causes unresolved external symbols.
          • HADOOP-11403 introduced the terror function in exception.c as a cross-platform thread-safe error reporting function. libwebhdfs code was changed to call this function, but libwebhdfs doesn't use the exception.c from hadoop-common. Instead, it uses the exception.c from libhdfs, which does not have this function.
          Show
          cnauroth Chris Nauroth added a comment - HDFS-573 implemented Windows compatibility for libhdfs. As part of this effort, we refactored OS-specific system calls into OS-specific source files. We also introduced htable.c for a platform-neutral hash table implementation. libwebhdfs includes jni_helper.c from libhdfs in its own build. jni_helper.c depends on the OS-specific sources and the hash table, but since the libwebhdfs build was not updated to include these files, this causes unresolved external symbols. HADOOP-11403 introduced the terror function in exception.c as a cross-platform thread-safe error reporting function. libwebhdfs code was changed to call this function, but libwebhdfs doesn't use the exception.c from hadoop-common. Instead, it uses the exception.c from libhdfs, which does not have this function.
          Hide
          cnauroth Chris Nauroth added a comment -

          The attached patch fixes the libwebhdfs build by adding the required sources introduced in HDFS-573 and using the same strategy as HADOOP-11403 for a cross-platform thread-safe function similar to strerror.

          Show
          cnauroth Chris Nauroth added a comment - The attached patch fixes the libwebhdfs build by adding the required sources introduced in HDFS-573 and using the same strategy as HADOOP-11403 for a cross-platform thread-safe function similar to strerror .
          Hide
          cnauroth Chris Nauroth added a comment -

          HADOOP-10452 requests updates to BUILDING.txt. I've asked that we include information on -Drequire.libwebhdfs.

          I created HADOOP-11937 to request that test-patch.sh do a full native build, including all optional components like libwebhdfs. This would have caught the problem before the patches that caused the break were committed.

          Show
          cnauroth Chris Nauroth added a comment - HADOOP-10452 requests updates to BUILDING.txt. I've asked that we include information on -Drequire.libwebhdfs . I created HADOOP-11937 to request that test-patch.sh do a full native build, including all optional components like libwebhdfs. This would have caught the problem before the patches that caused the break were committed.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 5m 19s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 javac 7m 38s There were no new javac warning messages.
          +1 release audit 0m 20s The applied patch does not increase the total number of release audit warnings.
          +1 whitespace 0m 4s The patch has no lines that end in whitespace.
          +1 install 1m 33s mvn install still works.
          +1 eclipse:eclipse 0m 31s The patch built with eclipse:eclipse.
          +1 native 1m 11s Pre-build of native portion
          -1 hdfs tests 165m 40s Tests failed in hadoop-hdfs.
              182m 21s  



          Reason Tests
          Failed unit tests hadoop.tracing.TestTraceAdmin



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12731283/HDFS-8346.001.patch
          Optional Tests javac unit
          git revision trunk / b88700d
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/10851/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10851/testReport/
          Java 1.7.0_55
          uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10851/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 5m 19s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 38s There were no new javac warning messages. +1 release audit 0m 20s The applied patch does not increase the total number of release audit warnings. +1 whitespace 0m 4s The patch has no lines that end in whitespace. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 31s The patch built with eclipse:eclipse. +1 native 1m 11s Pre-build of native portion -1 hdfs tests 165m 40s Tests failed in hadoop-hdfs.     182m 21s   Reason Tests Failed unit tests hadoop.tracing.TestTraceAdmin Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12731283/HDFS-8346.001.patch Optional Tests javac unit git revision trunk / b88700d hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/10851/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10851/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10851/console This message was automatically generated.
          Hide
          wheat9 Haohui Mai added a comment -

          +1 I'll commit it shortly.

          Show
          wheat9 Haohui Mai added a comment - +1 I'll commit it shortly.
          Hide
          wheat9 Haohui Mai added a comment -

          I've committed the patch to trunk and branch-2. Thanks Chris Nauroth for the contribution.

          Show
          wheat9 Haohui Mai added a comment - I've committed the patch to trunk and branch-2. Thanks Chris Nauroth for the contribution.
          Hide
          cnauroth Chris Nauroth added a comment -

          Thanks very much, Haohui!

          Show
          cnauroth Chris Nauroth added a comment - Thanks very much, Haohui!
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #7776 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7776/)
          HDFS-8346. libwebhdfs build fails during link due to unresolved external symbols. Contributed by Chris Nauroth. (wheat9: rev f4ebbc6afc1297dced54bd2bd671e587c4ceb2fc)

          • hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_client.c
          • hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/CMakeLists.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_client.h
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7776 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7776/ ) HDFS-8346 . libwebhdfs build fails during link due to unresolved external symbols. Contributed by Chris Nauroth. (wheat9: rev f4ebbc6afc1297dced54bd2bd671e587c4ceb2fc) hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_client.c hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/CMakeLists.txt hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_client.h hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #191 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/191/)
          HDFS-8346. libwebhdfs build fails during link due to unresolved external symbols. Contributed by Chris Nauroth. (wheat9: rev f4ebbc6afc1297dced54bd2bd671e587c4ceb2fc)

          • hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_client.h
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_client.c
          • hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/CMakeLists.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #191 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/191/ ) HDFS-8346 . libwebhdfs build fails during link due to unresolved external symbols. Contributed by Chris Nauroth. (wheat9: rev f4ebbc6afc1297dced54bd2bd671e587c4ceb2fc) hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_client.h hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_client.c hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/CMakeLists.txt
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk #922 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/922/)
          HDFS-8346. libwebhdfs build fails during link due to unresolved external symbols. Contributed by Chris Nauroth. (wheat9: rev f4ebbc6afc1297dced54bd2bd671e587c4ceb2fc)

          • hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_client.h
          • hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/CMakeLists.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_client.c
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #922 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/922/ ) HDFS-8346 . libwebhdfs build fails during link due to unresolved external symbols. Contributed by Chris Nauroth. (wheat9: rev f4ebbc6afc1297dced54bd2bd671e587c4ceb2fc) hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_client.h hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/CMakeLists.txt hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_client.c hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2120 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2120/)
          HDFS-8346. libwebhdfs build fails during link due to unresolved external symbols. Contributed by Chris Nauroth. (wheat9: rev f4ebbc6afc1297dced54bd2bd671e587c4ceb2fc)

          • hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_client.h
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_client.c
          • hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/CMakeLists.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2120 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2120/ ) HDFS-8346 . libwebhdfs build fails during link due to unresolved external symbols. Contributed by Chris Nauroth. (wheat9: rev f4ebbc6afc1297dced54bd2bd671e587c4ceb2fc) hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_client.h hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_client.c hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/CMakeLists.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #180 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/180/)
          HDFS-8346. libwebhdfs build fails during link due to unresolved external symbols. Contributed by Chris Nauroth. (wheat9: rev f4ebbc6afc1297dced54bd2bd671e587c4ceb2fc)

          • hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/CMakeLists.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_client.c
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_client.h
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #180 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/180/ ) HDFS-8346 . libwebhdfs build fails during link due to unresolved external symbols. Contributed by Chris Nauroth. (wheat9: rev f4ebbc6afc1297dced54bd2bd671e587c4ceb2fc) hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/CMakeLists.txt hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_client.c hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_client.h
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #190 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/190/)
          HDFS-8346. libwebhdfs build fails during link due to unresolved external symbols. Contributed by Chris Nauroth. (wheat9: rev f4ebbc6afc1297dced54bd2bd671e587c4ceb2fc)

          • hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_client.h
          • hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_client.c
          • hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/CMakeLists.txt
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #190 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/190/ ) HDFS-8346 . libwebhdfs build fails during link due to unresolved external symbols. Contributed by Chris Nauroth. (wheat9: rev f4ebbc6afc1297dced54bd2bd671e587c4ceb2fc) hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_client.h hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_client.c hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/CMakeLists.txt hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2138 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2138/)
          HDFS-8346. libwebhdfs build fails during link due to unresolved external symbols. Contributed by Chris Nauroth. (wheat9: rev f4ebbc6afc1297dced54bd2bd671e587c4ceb2fc)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_client.c
          • hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/CMakeLists.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_client.h
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2138 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2138/ ) HDFS-8346 . libwebhdfs build fails during link due to unresolved external symbols. Contributed by Chris Nauroth. (wheat9: rev f4ebbc6afc1297dced54bd2bd671e587c4ceb2fc) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_client.c hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/CMakeLists.txt hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_client.h
          Hide
          djp Junping Du added a comment -

          Hi Chris Nauroth, do we need this fix in branch-2.6/branch-2.7?

          Show
          djp Junping Du added a comment - Hi Chris Nauroth , do we need this fix in branch-2.6/branch-2.7?
          Hide
          cnauroth Chris Nauroth added a comment -

          IMO, this is not worthwhile for a backport to branch-2.7 or branch-2.6. There is some debate about the actual utility of libwebhdfs, and we don't have any evidence that it's being used by anyone. If someone else wants to request a backport though, then I wouldn't object.

          Show
          cnauroth Chris Nauroth added a comment - IMO, this is not worthwhile for a backport to branch-2.7 or branch-2.6. There is some debate about the actual utility of libwebhdfs, and we don't have any evidence that it's being used by anyone. If someone else wants to request a backport though, then I wouldn't object.
          Hide
          djp Junping Du added a comment -

          Thanks Chris for clarification on this. I agree that this is not a very high priority bug to fix.
          This comes to my radar per Kihwal Lee's comments in HDFS-9445(https://issues.apache.org/jira/browse/HDFS-9445?focusedCommentId=15081254&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15081254). Kihwal Lee, do you still think we should backport this to 2.6/2.7 now or you are OK to defer to later?

          Show
          djp Junping Du added a comment - Thanks Chris for clarification on this. I agree that this is not a very high priority bug to fix. This comes to my radar per Kihwal Lee 's comments in HDFS-9445 ( https://issues.apache.org/jira/browse/HDFS-9445?focusedCommentId=15081254&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15081254 ). Kihwal Lee , do you still think we should backport this to 2.6/2.7 now or you are OK to defer to later?
          Hide
          cnauroth Chris Nauroth added a comment -

          This comes to my radar per Kihwal Lee's comments...

          I see. If it helps expedite HDFS-9445, then I see no harm in including this too. Let's see what Kihwal thinks.

          Show
          cnauroth Chris Nauroth added a comment - This comes to my radar per Kihwal Lee's comments... I see. If it helps expedite HDFS-9445 , then I see no harm in including this too. Let's see what Kihwal thinks.
          Hide
          kihwal Kihwal Lee added a comment -

          I don't see a point of maintaining libwebhdfs in 2.6 and 2.7. We might as well push for HDFS-9047. YETUS-223 is also related.

          Show
          kihwal Kihwal Lee added a comment - I don't see a point of maintaining libwebhdfs in 2.6 and 2.7. We might as well push for HDFS-9047 . YETUS-223 is also related.
          Hide
          djp Junping Du added a comment -

          So Kihwal, you are preferring to push HDFS-9047 (deprecate/remove libwebhdfs) instead of fixing the build failures like patch here. Isn't it? But that one is unassigned so far, and I didn't see who is actually working on it recently...

          Show
          djp Junping Du added a comment - So Kihwal, you are preferring to push HDFS-9047 (deprecate/remove libwebhdfs) instead of fixing the build failures like patch here. Isn't it? But that one is unassigned so far, and I didn't see who is actually working on it recently...
          Hide
          kihwal Kihwal Lee added a comment -

          The immediate goal will be to make the precommit work. We can go either way. We can fix it, but the fact that nobody complained about the breakage in 2.6 and 2.7 so far probably means no one is using it. If we remove it, it is different from deprecating functioning code. Rather, we will be removing code that has been unbuildable for a long time.

          Show
          kihwal Kihwal Lee added a comment - The immediate goal will be to make the precommit work. We can go either way. We can fix it, but the fact that nobody complained about the breakage in 2.6 and 2.7 so far probably means no one is using it. If we remove it, it is different from deprecating functioning code. Rather, we will be removing code that has been unbuildable for a long time.
          Hide
          kihwal Kihwal Lee added a comment -

          I felt it needs more discussion, so left a comment in HDFS-9047.

          Show
          kihwal Kihwal Lee added a comment - I felt it needs more discussion, so left a comment in HDFS-9047 .
          Hide
          djp Junping Du added a comment -

          Ok. Thanks for pushing it, Kihwal Lee!

          Show
          djp Junping Du added a comment - Ok. Thanks for pushing it, Kihwal Lee !

            People

            • Assignee:
              cnauroth Chris Nauroth
              Reporter:
              cnauroth Chris Nauroth
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development