Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13684

Snappy may complain Hadoop is built without snappy if libhadoop is not found.

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha2
    • Component/s: None
    • Labels:

      Description

      If for some reason libhadoop can not be found/loaded, Snappy complains Hadoop is not built with Snappy but it actually is.

      SnappyCodec.java
      public static void checkNativeCodeLoaded() {
            if (!NativeCodeLoader.isNativeCodeLoaded() ||
                !NativeCodeLoader.buildSupportsSnappy()) {
              throw new RuntimeException("native snappy library not available: " +
                  "this version of libhadoop was built without " +
                  "snappy support.");
            }
      

      This case may happen with MAPREDUCE-6577.

      1. HADOOP-13684.001.patch
        1 kB
        Wei-Chiu Chuang
      2. HADOOP-13684.002.patch
        2 kB
        Wei-Chiu Chuang

        Issue Links

          Activity

          Hide
          stevel@apache.org Steve Loughran added a comment -

          Before doing that, is it possible to have the setup: no native libhadoop, but snappy native code around and accessible. If so, it should be bypassing the native lib check; it just needs some other check

          Show
          stevel@apache.org Steve Loughran added a comment - Before doing that, is it possible to have the setup: no native libhadoop, but snappy native code around and accessible. If so, it should be bypassing the native lib check; it just needs some other check
          Hide
          xiaochen Xiao Chen added a comment - - edited

          The regression of this commit is because it changed

          if (!NativeCodeLoader.isNativeCodeLoaded() ||		
          	          !NativeCodeLoader.buildSupportsSnappy()) {		
          	        throw 
          	      }
          

          to

          	    if (!NativeCodeLoader.buildSupportsSnappy()) {
          	      throw
          	    }
          	    if (!NativeCodeLoader.isNativeCodeLoaded()) {
          	      throw 
          	    }
          

          .

          So, I was proposing we could fix this in HADOOP-14981 by doing

          	    if (!NativeCodeLoader.isNativeCodeLoaded()) {
          	      throw 
          	    }
          	    if (!NativeCodeLoader.buildSupportsSnappy()) {
          	      throw
          	    }
          

          which is behaviorally identical to before this commit, but with a different exception message. This should solve the 'original problem' IMO.

          We can continue this on HADOOP-14981. I believe Wei-Chiu Chuang will have cycles on this soon.

          Show
          xiaochen Xiao Chen added a comment - - edited The regression of this commit is because it changed if (!NativeCodeLoader.isNativeCodeLoaded() || !NativeCodeLoader.buildSupportsSnappy()) { throw } to if (!NativeCodeLoader.buildSupportsSnappy()) { throw } if (!NativeCodeLoader.isNativeCodeLoaded()) { throw } . So, I was proposing we could fix this in HADOOP-14981 by doing if (!NativeCodeLoader.isNativeCodeLoaded()) { throw } if (!NativeCodeLoader.buildSupportsSnappy()) { throw } which is behaviorally identical to before this commit, but with a different exception message. This should solve the 'original problem' IMO. We can continue this on HADOOP-14981 . I believe Wei-Chiu Chuang will have cycles on this soon.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          I think the original problem is that if snappy is present, but libhadooop missing, the error message is "no native lib". But the patch calls NativeCodeLoader.buildSupportsSnappy(), which triggers a native code load, which triggers the link exception if the native lib isn't there. Whatever was intended, the latest patch doesn't fix.

          so this doesn't fix the underlying problem, merely makes the error message even less informative than before.

          The only way to fix the original problem is to have a probe for snappy being present which doesn't depend on the native lib at all

          Show
          stevel@apache.org Steve Loughran added a comment - I think the original problem is that if snappy is present, but libhadooop missing, the error message is "no native lib". But the patch calls NativeCodeLoader.buildSupportsSnappy() , which triggers a native code load, which triggers the link exception if the native lib isn't there. Whatever was intended, the latest patch doesn't fix. so this doesn't fix the underlying problem, merely makes the error message even less informative than before. The only way to fix the original problem is to have a probe for snappy being present which doesn't depend on the native lib at all
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Steve Loughran.

          Wei-Chiu Chuang could you provide more details?
          I recall Wei-Chiu verified this internally with an escalation, where a confusing error message was seen.

          . Wei-Chiu Chuang added a comment - Aug/29/17 08:32 AM
          HADOOP-13684 did not fix a bug. It simply corrects an incorrect log message more precise.

          Looking at HADOOP-14981 and this one, one possible fix could be moving the isNativeCodeLoaded check before buildSupportsSnappy.

          Show
          xiaochen Xiao Chen added a comment - Thanks Steve Loughran . Wei-Chiu Chuang could you provide more details? I recall Wei-Chiu verified this internally with an escalation, where a confusing error message was seen. . Wei-Chiu Chuang added a comment - Aug/29/17 08:32 AM HADOOP-13684 did not fix a bug. It simply corrects an incorrect log message more precise. Looking at HADOOP-14981 and this one, one possible fix could be moving the isNativeCodeLoaded check before buildSupportsSnappy .
          Hide
          stevel@apache.org Steve Loughran added a comment -

          I've just opened HADOOP-14981. Essentially, I don't think this fix fixed anything.

          How was it originally tested?

          Show
          stevel@apache.org Steve Loughran added a comment - I've just opened HADOOP-14981 . Essentially, I don't think this fix fixed anything. How was it originally tested?
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10588 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10588/)
          HADOOP-13684. Snappy may complain Hadoop is built without snappy if (weichiu: rev 4b32b1420d98ea23460d05ae94f2698109b3d6f7)

          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/SnappyCodec.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10588 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10588/ ) HADOOP-13684 . Snappy may complain Hadoop is built without snappy if (weichiu: rev 4b32b1420d98ea23460d05ae94f2698109b3d6f7) (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/SnappyCodec.java
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Committed v2 patch to trunk, branch-2 and branch-2.8. Thanks Xiao Chen for reviewing the patch!

          Show
          jojochuang Wei-Chiu Chuang added a comment - Committed v2 patch to trunk, branch-2 and branch-2.8. Thanks Xiao Chen for reviewing the patch!
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Committing this based on Xiao Chen's +1.

          Show
          jojochuang Wei-Chiu Chuang added a comment - Committing this based on Xiao Chen 's +1.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Test failure is unrelated.

          Show
          jojochuang Wei-Chiu Chuang added a comment - Test failure is unrelated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 7m 38s trunk passed
          +1 compile 8m 16s trunk passed
          +1 checkstyle 0m 25s trunk passed
          +1 mvnsite 1m 4s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 1m 33s trunk passed
          +1 javadoc 0m 47s trunk passed
          +1 mvninstall 0m 41s the patch passed
          +1 compile 7m 58s the patch passed
          +1 javac 7m 58s the patch passed
          +1 checkstyle 0m 25s hadoop-common-project/hadoop-common: The patch generated 0 new + 4 unchanged - 9 fixed = 4 total (was 13)
          +1 mvnsite 1m 1s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 44s the patch passed
          +1 javadoc 0m 47s the patch passed
          -1 unit 10m 19s hadoop-common in the patch failed.
          +1 asflicense 0m 23s The patch does not generate ASF License warnings.
          45m 11s



          Reason Tests
          Failed junit tests hadoop.ha.TestZKFailoverController



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HADOOP-13684
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12832150/HADOOP-13684.002.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 44e56d540bb2 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 3059b25
          Default Java 1.8.0_101
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10701/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10701/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10701/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 38s trunk passed +1 compile 8m 16s trunk passed +1 checkstyle 0m 25s trunk passed +1 mvnsite 1m 4s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 33s trunk passed +1 javadoc 0m 47s trunk passed +1 mvninstall 0m 41s the patch passed +1 compile 7m 58s the patch passed +1 javac 7m 58s the patch passed +1 checkstyle 0m 25s hadoop-common-project/hadoop-common: The patch generated 0 new + 4 unchanged - 9 fixed = 4 total (was 13) +1 mvnsite 1m 1s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 44s the patch passed +1 javadoc 0m 47s the patch passed -1 unit 10m 19s hadoop-common in the patch failed. +1 asflicense 0m 23s The patch does not generate ASF License warnings. 45m 11s Reason Tests Failed junit tests hadoop.ha.TestZKFailoverController Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HADOOP-13684 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12832150/HADOOP-13684.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 44e56d540bb2 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 3059b25 Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10701/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10701/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10701/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          Nice improvement Wei-Chiu. +1 pending jenkins.

          Show
          xiaochen Xiao Chen added a comment - Nice improvement Wei-Chiu. +1 pending jenkins.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Attach v2 patch to clean up checkstyle warning. The test failure is unrelated.

          Show
          jojochuang Wei-Chiu Chuang added a comment - Attach v2 patch to clean up checkstyle warning. The test failure is unrelated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 8m 47s trunk passed
          +1 compile 9m 2s trunk passed
          +1 checkstyle 0m 26s trunk passed
          +1 mvnsite 1m 3s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 40s trunk passed
          +1 javadoc 0m 56s trunk passed
          +1 mvninstall 0m 49s the patch passed
          +1 compile 9m 5s the patch passed
          +1 javac 9m 5s the patch passed
          -0 checkstyle 0m 32s hadoop-common-project/hadoop-common: The patch generated 4 new + 13 unchanged - 1 fixed = 17 total (was 14)
          +1 mvnsite 1m 9s the patch passed
          +1 mvneclipse 0m 15s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 49s the patch passed
          +1 javadoc 0m 54s the patch passed
          -1 unit 17m 27s hadoop-common in the patch failed.
          +1 asflicense 0m 24s The patch does not generate ASF License warnings.
          56m 17s



          Reason Tests
          Failed junit tests hadoop.net.TestDNS
          Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HADOOP-13684
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12831780/HADOOP-13684.001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux dbb4dadfcfab 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / d6be1e7
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10675/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10675/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10675/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10675/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 8m 47s trunk passed +1 compile 9m 2s trunk passed +1 checkstyle 0m 26s trunk passed +1 mvnsite 1m 3s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 40s trunk passed +1 javadoc 0m 56s trunk passed +1 mvninstall 0m 49s the patch passed +1 compile 9m 5s the patch passed +1 javac 9m 5s the patch passed -0 checkstyle 0m 32s hadoop-common-project/hadoop-common: The patch generated 4 new + 13 unchanged - 1 fixed = 17 total (was 14) +1 mvnsite 1m 9s the patch passed +1 mvneclipse 0m 15s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 49s the patch passed +1 javadoc 0m 54s the patch passed -1 unit 17m 27s hadoop-common in the patch failed. +1 asflicense 0m 24s The patch does not generate ASF License warnings. 56m 17s Reason Tests Failed junit tests hadoop.net.TestDNS Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HADOOP-13684 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12831780/HADOOP-13684.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux dbb4dadfcfab 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d6be1e7 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10675/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10675/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10675/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10675/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Attach a simple fix.

          Show
          jojochuang Wei-Chiu Chuang added a comment - Attach a simple fix.

            People

            • Assignee:
              jojochuang Wei-Chiu Chuang
              Reporter:
              jojochuang Wei-Chiu Chuang
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development