Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-12955

Fix bugs in the initialization of the ISA-L library JNI bindings

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0-alpha1
    • Component/s: native
    • Labels:
      None
    • Target Version/s:

      Description

      Ref. the comment here.

      When run hadoop checknative, it also failed. Got something like below from log:

      Stack: [0x00007f2b9d405000,0x00007f2b9d506000],  sp=0x00007f2b9d504748,  free space=1021k
      Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
      V  [libjvm.so+0xa90c90]  UTF8::unicode_length(char const*)+0x0
      V  [libjvm.so+0x6ddfc3]  jni_NewStringUTF+0xc3
      j  org.apache.hadoop.io.erasurecode.ErasureCodeNative.getLibraryName()Ljava/lang/String;+0
      j  org.apache.hadoop.util.NativeLibraryChecker.main([Ljava/lang/String;)V+212
      v  ~StubRoutines::call_stub
      V  [libjvm.so+0x68c616]  JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x1056
      V  [libjvm.so+0x6cdc32]  jni_invoke_static(JNIEnv_*, JavaValue*, _jobject*, JNICallType, _jmethodID*, JNI_ArgumentPusher*, Thread*)+0x362
      V  [libjvm.so+0x6ea63a]  jni_CallStaticVoidMethod+0x17a
      C  [libjli.so+0x7bcc]  JavaMain+0x80c
      C  [libpthread.so.0+0x8182]  start_thread+0xc2
      
      Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
      j  org.apache.hadoop.io.erasurecode.ErasureCodeNative.getLibraryName()Ljava/lang/String;+0
      j  org.apache.hadoop.util.NativeLibraryChecker.main([Ljava/lang/String;)V+212
      v  ~StubRoutines::call_stub
      
      1. HADOOP-12955-v1.patch
        3 kB
        Kai Zheng
      2. HADOOP-12955-v2.patch
        5 kB
        Kai Zheng
      3. HADOOP-12955-v3.patch
        6 kB
        Kai Zheng
      4. HADOOP-12955-v4.patch
        6 kB
        Kai Zheng

        Issue Links

          Activity

          Hide
          drankye Kai Zheng added a comment -

          Uploaded a patch for the fix. The root causes for the crash when calling Java_org_apache_hadoop_io_erasurecode_ErasureCodeNative_getLibraryName are: 1) the library isn't loaded; 2) get_library_name returns pointer to local variable.

          Test passed locally as follows.

          [root@zkdesk hadoop-3.0.0-SNAPSHOT]# bin/hadoop checknative
          16/03/24 17:05:49 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
          16/03/24 17:05:49 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
          Native library checking:
          hadoop:  true /home/workspace/hadoop3/hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/lib/native/libhadoop.so.1.0.0
          zlib:    true /lib64/libz.so.1
          snappy:  true /lib64/libsnappy.so.1
          lz4:     true revision:99
          bzip2:   true /lib64/libbz2.so.1
          openssl: true /lib64/libcrypto.so
          ISA-L:   true /home/workspace/hadoop3/hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/lib/native/libisal.so.2
          

          Colin P. McCabe, sorry for not catching this in HADOOP-11996 because I forgot test the command while providing later revision of patches there. Would you help review this? Thanks.

          Show
          drankye Kai Zheng added a comment - Uploaded a patch for the fix. The root causes for the crash when calling Java_org_apache_hadoop_io_erasurecode_ErasureCodeNative_getLibraryName are: 1) the library isn't loaded; 2) get_library_name returns pointer to local variable. Test passed locally as follows. [root@zkdesk hadoop-3.0.0-SNAPSHOT]# bin/hadoop checknative 16/03/24 17:05:49 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native 16/03/24 17:05:49 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library Native library checking: hadoop: true /home/workspace/hadoop3/hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/lib/native/libhadoop.so.1.0.0 zlib: true /lib64/libz.so.1 snappy: true /lib64/libsnappy.so.1 lz4: true revision:99 bzip2: true /lib64/libbz2.so.1 openssl: true /lib64/libcrypto.so ISA-L: true /home/workspace/hadoop3/hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/lib/native/libisal.so.2 Colin P. McCabe , sorry for not catching this in HADOOP-11996 because I forgot test the command while providing later revision of patches there. Would you help review this? Thanks.
          Hide
          drankye Kai Zheng added a comment -

          I guess when YETUS-222 is committed, the ISA-L related building will be in the loop so such cases will be caught the first time.

          Show
          drankye Kai Zheng added a comment - I guess when YETUS-222 is committed, the ISA-L related building will be in the loop so such cases will be caught the first time.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 20s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 10m 6s trunk passed
          +1 compile 12m 2s trunk passed with JDK v1.8.0_74
          +1 compile 10m 20s trunk passed with JDK v1.7.0_95
          +1 mvnsite 1m 20s trunk passed
          +1 mvneclipse 0m 20s trunk passed
          +1 mvninstall 0m 57s the patch passed
          +1 compile 12m 1s the patch passed with JDK v1.8.0_74
          +1 cc 47m 26s root-jdk1.8.0_74 with JDK v1.8.0_74 generated 0 new + 19 unchanged - 2 fixed = 19 total (was 21)
          +1 cc 12m 1s root in the patch passed with JDK v1.8.0_74.
          +1 javac 12m 1s the patch passed
          +1 compile 10m 23s the patch passed with JDK v1.7.0_95
          -1 cc 57m 50s root-jdk1.7.0_95 with JDK v1.7.0_95 generated 3 new + 26 unchanged - 5 fixed = 29 total (was 31)
          +1 cc 10m 23s the patch passed
          +1 javac 10m 23s the patch passed
          +1 mvnsite 1m 18s the patch passed
          +1 mvneclipse 0m 22s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          -1 unit 10m 47s hadoop-common in the patch failed with JDK v1.8.0_74.
          -1 unit 9m 59s hadoop-common in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 34s Patch does not generate ASF License warnings.
          81m 17s



          Reason Tests
          JDK v1.8.0_74 Failed junit tests hadoop.ipc.TestRPCWaitForProxy
            hadoop.fs.shell.find.TestIname
          JDK v1.7.0_95 Failed junit tests hadoop.ipc.TestRPCWaitForProxy
            hadoop.fs.shell.find.TestPrint0



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:fbe3e86
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12794965/HADOOP-12955-v1.patch
          JIRA Issue HADOOP-12955
          Optional Tests asflicense compile cc mvnsite javac unit
          uname Linux 9e32ecfc6d42 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / a107cee
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          cc root-jdk1.7.0_95: https://builds.apache.org/job/PreCommit-HADOOP-Build/8906/artifact/patchprocess/diff-compile-cc-root-jdk1.7.0_95.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8906/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8906/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8906/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8906/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8906/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8906/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 10m 6s trunk passed +1 compile 12m 2s trunk passed with JDK v1.8.0_74 +1 compile 10m 20s trunk passed with JDK v1.7.0_95 +1 mvnsite 1m 20s trunk passed +1 mvneclipse 0m 20s trunk passed +1 mvninstall 0m 57s the patch passed +1 compile 12m 1s the patch passed with JDK v1.8.0_74 +1 cc 47m 26s root-jdk1.8.0_74 with JDK v1.8.0_74 generated 0 new + 19 unchanged - 2 fixed = 19 total (was 21) +1 cc 12m 1s root in the patch passed with JDK v1.8.0_74. +1 javac 12m 1s the patch passed +1 compile 10m 23s the patch passed with JDK v1.7.0_95 -1 cc 57m 50s root-jdk1.7.0_95 with JDK v1.7.0_95 generated 3 new + 26 unchanged - 5 fixed = 29 total (was 31) +1 cc 10m 23s the patch passed +1 javac 10m 23s the patch passed +1 mvnsite 1m 18s the patch passed +1 mvneclipse 0m 22s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. -1 unit 10m 47s hadoop-common in the patch failed with JDK v1.8.0_74. -1 unit 9m 59s hadoop-common in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 34s Patch does not generate ASF License warnings. 81m 17s Reason Tests JDK v1.8.0_74 Failed junit tests hadoop.ipc.TestRPCWaitForProxy   hadoop.fs.shell.find.TestIname JDK v1.7.0_95 Failed junit tests hadoop.ipc.TestRPCWaitForProxy   hadoop.fs.shell.find.TestPrint0 Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12794965/HADOOP-12955-v1.patch JIRA Issue HADOOP-12955 Optional Tests asflicense compile cc mvnsite javac unit uname Linux 9e32ecfc6d42 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / a107cee Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 cc root-jdk1.7.0_95: https://builds.apache.org/job/PreCommit-HADOOP-Build/8906/artifact/patchprocess/diff-compile-cc-root-jdk1.7.0_95.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8906/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8906/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8906/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8906/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8906/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8906/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          cmccabe Colin P. McCabe added a comment -

          Thanks, Kai Zheng. I agree that building using -Drequire.isal is important, and I hope we can get YETUS-222 in soon.

          124 void get_library_name(char* buf, size_t buf_len) {
          125 #ifdef UNIX 
          126   Dl_info dl_info;
          127
          128   if (isaLoader == NULL || isaLoader->ec_encode_data == NULL) {
          129     strncpy(buf, HADOOP_ISAL_LIBRARY, buf_len);
          130   }
          131
          132   if(dladdr(isaLoader->ec_encode_data, &dl_info)) {
          133     strncpy(buf, dl_info.dli_fname, buf_len);
          134   }
          135 #else
          136   LPTSTR filename = NULL;
          137
          138   if (isaLoader == NULL || isaLoader->libec == NULL) {
          139     strncpy(buf, HADOOP_ISAL_LIBRARY, buf_len);
          140   }
          141
          142   if (GetModuleFileName(isaLoader->libec, filename, 256) > 0) {
          143     strncpy(buf, filename, buf_len);
          144   }
          145 #endif
          

          Right now, buf is left uninitialized if neither if statement is taken in the non-UNIX code path. This case should be an error instead.

          Please do not use strncpy. It zeroes the entire end of the buffer, which is unnecessary in this case. It also can produce a buffer which is not null-terminated if src is longer than dst. Instead, use snprintf(buf, buf_len, "%s", source);.

          You need to check if the string you're trying to copy is too long, and return an error if that is so. Please do not silently truncate the name string.

          Why is there no return statement on line 140? Is there ever a case where we would want to overwrite the string we wrote on line 139 later in the function?

          The potential error coming from loadLib is not handled.

          Show
          cmccabe Colin P. McCabe added a comment - Thanks, Kai Zheng . I agree that building using -Drequire.isal is important, and I hope we can get YETUS-222 in soon. 124 void get_library_name( char * buf, size_t buf_len) { 125 #ifdef UNIX 126 Dl_info dl_info; 127 128 if (isaLoader == NULL || isaLoader->ec_encode_data == NULL) { 129 strncpy(buf, HADOOP_ISAL_LIBRARY, buf_len); 130 } 131 132 if (dladdr(isaLoader->ec_encode_data, &dl_info)) { 133 strncpy(buf, dl_info.dli_fname, buf_len); 134 } 135 # else 136 LPTSTR filename = NULL; 137 138 if (isaLoader == NULL || isaLoader->libec == NULL) { 139 strncpy(buf, HADOOP_ISAL_LIBRARY, buf_len); 140 } 141 142 if (GetModuleFileName(isaLoader->libec, filename, 256) > 0) { 143 strncpy(buf, filename, buf_len); 144 } 145 #endif Right now, buf is left uninitialized if neither if statement is taken in the non-UNIX code path. This case should be an error instead. Please do not use strncpy . It zeroes the entire end of the buffer, which is unnecessary in this case. It also can produce a buffer which is not null-terminated if src is longer than dst . Instead, use snprintf(buf, buf_len, "%s", source); . You need to check if the string you're trying to copy is too long, and return an error if that is so. Please do not silently truncate the name string. Why is there no return statement on line 140? Is there ever a case where we would want to overwrite the string we wrote on line 139 later in the function? The potential error coming from loadLib is not handled.
          Hide
          drankye Kai Zheng added a comment -

          Thanks Colin P. McCabe for the valuable review comments!

          buf is left uninitialized if neither if statement is taken in the non-UNIX code path.

          Yes buf should be ensured to be initialized by this call. Will fix this.

          Please do not use strncpy. ... Instead, use snprintf(buf, buf_len, "%s", source);

          Sounds great!

          You need to check if the string you're trying to copy is too long, and return an error if that is so. Please do not silently truncate the name string.

          OK, I will make the function return an error if any. Return null if no error at all.

          Why is there no return statement on line 140?

          It's my stupid mistake. Good catch. It should return immediately.

          The potential error coming from loadLib is not handled.

          I thought loadLib handles its error itself by throwing an exception.

          Show
          drankye Kai Zheng added a comment - Thanks Colin P. McCabe for the valuable review comments! buf is left uninitialized if neither if statement is taken in the non-UNIX code path. Yes buf should be ensured to be initialized by this call. Will fix this. Please do not use strncpy. ... Instead, use snprintf(buf, buf_len, "%s", source); Sounds great! You need to check if the string you're trying to copy is too long, and return an error if that is so. Please do not silently truncate the name string. OK, I will make the function return an error if any. Return null if no error at all. Why is there no return statement on line 140? It's my stupid mistake. Good catch. It should return immediately. The potential error coming from loadLib is not handled. I thought loadLib handles its error itself by throwing an exception.
          Hide
          drankye Kai Zheng added a comment -

          Updated the patch according to review comments.

          Show
          drankye Kai Zheng added a comment - Updated the patch according to review comments.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 10m 55s trunk passed
          +1 compile 14m 56s trunk passed with JDK v1.8.0_74
          +1 compile 11m 34s trunk passed with JDK v1.7.0_95
          +1 mvnsite 1m 27s trunk passed
          +1 mvneclipse 0m 24s trunk passed
          +1 mvninstall 1m 10s the patch passed
          +1 compile 14m 58s the patch passed with JDK v1.8.0_74
          +1 cc 55m 44s root-jdk1.8.0_74 with JDK v1.8.0_74 generated 0 new + 11 unchanged - 10 fixed = 11 total (was 21)
          +1 cc 14m 58s root in the patch passed with JDK v1.8.0_74.
          +1 javac 14m 58s the patch passed
          +1 compile 12m 20s the patch passed with JDK v1.7.0_95
          +1 cc 68m 5s root-jdk1.7.0_95 with JDK v1.7.0_95 generated 0 new + 21 unchanged - 10 fixed = 21 total (was 31)
          +1 cc 12m 20s root in the patch passed with JDK v1.7.0_95.
          +1 javac 12m 20s the patch passed
          +1 mvnsite 1m 33s the patch passed
          +1 mvneclipse 0m 25s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          -1 unit 14m 15s hadoop-common in the patch failed with JDK v1.8.0_74.
          -1 unit 12m 25s hadoop-common in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 40s Patch does not generate ASF License warnings.
          97m 48s



          Reason Tests
          JDK v1.8.0_74 Failed junit tests hadoop.ipc.TestRPCWaitForProxy
            hadoop.fs.shell.find.TestIname
            hadoop.fs.shell.find.TestPrint0
            hadoop.fs.shell.find.TestPrint
            hadoop.fs.shell.find.TestName
          JDK v1.7.0_95 Failed junit tests hadoop.ipc.TestRPCWaitForProxy
            hadoop.fs.shell.find.TestIname
            hadoop.security.token.delegation.TestZKDelegationTokenSecretManager
            hadoop.fs.shell.find.TestPrint0
            hadoop.fs.shell.find.TestPrint
            hadoop.fs.shell.find.TestName



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:fbe3e86
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12795177/HADOOP-12955-v2.patch
          JIRA Issue HADOOP-12955
          Optional Tests asflicense compile cc mvnsite javac unit
          uname Linux cb98028f283e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 19b645c
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8911/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8911/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8911/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8911/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8911/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8911/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 10m 55s trunk passed +1 compile 14m 56s trunk passed with JDK v1.8.0_74 +1 compile 11m 34s trunk passed with JDK v1.7.0_95 +1 mvnsite 1m 27s trunk passed +1 mvneclipse 0m 24s trunk passed +1 mvninstall 1m 10s the patch passed +1 compile 14m 58s the patch passed with JDK v1.8.0_74 +1 cc 55m 44s root-jdk1.8.0_74 with JDK v1.8.0_74 generated 0 new + 11 unchanged - 10 fixed = 11 total (was 21) +1 cc 14m 58s root in the patch passed with JDK v1.8.0_74. +1 javac 14m 58s the patch passed +1 compile 12m 20s the patch passed with JDK v1.7.0_95 +1 cc 68m 5s root-jdk1.7.0_95 with JDK v1.7.0_95 generated 0 new + 21 unchanged - 10 fixed = 21 total (was 31) +1 cc 12m 20s root in the patch passed with JDK v1.7.0_95. +1 javac 12m 20s the patch passed +1 mvnsite 1m 33s the patch passed +1 mvneclipse 0m 25s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. -1 unit 14m 15s hadoop-common in the patch failed with JDK v1.8.0_74. -1 unit 12m 25s hadoop-common in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 40s Patch does not generate ASF License warnings. 97m 48s Reason Tests JDK v1.8.0_74 Failed junit tests hadoop.ipc.TestRPCWaitForProxy   hadoop.fs.shell.find.TestIname   hadoop.fs.shell.find.TestPrint0   hadoop.fs.shell.find.TestPrint   hadoop.fs.shell.find.TestName JDK v1.7.0_95 Failed junit tests hadoop.ipc.TestRPCWaitForProxy   hadoop.fs.shell.find.TestIname   hadoop.security.token.delegation.TestZKDelegationTokenSecretManager   hadoop.fs.shell.find.TestPrint0   hadoop.fs.shell.find.TestPrint   hadoop.fs.shell.find.TestName Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12795177/HADOOP-12955-v2.patch JIRA Issue HADOOP-12955 Optional Tests asflicense compile cc mvnsite javac unit uname Linux cb98028f283e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 19b645c Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8911/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8911/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8911/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8911/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8911/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8911/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          cmccabe Colin P. McCabe added a comment -

          Thanks for the revisions. It is good to see checking for names which are too long, as well as fixes to the other little bugs.

          If the error string being returned is a constant string, it should be a const char* rather than a char*. Also, please document that the string does not need to be freed by the caller in the doxygen.

          I thought loadLib handles its error itself by throwing an exception.

          C does not have exceptions. When the C code calls a Java method via JNI, the C code must check for an error condition and return if that occurs. if the C caller does not check, execution will continue. The results of calling many JNI methods when an exception is raised are undefined. I believe the situation is similar when calling Java code through C++, even though that language does have exceptions. The important thing to keep in mind is that Java exceptions do not translate into the JNI code.

          Can you fix the cases where we are not checking for java exceptions?

          Show
          cmccabe Colin P. McCabe added a comment - Thanks for the revisions. It is good to see checking for names which are too long, as well as fixes to the other little bugs. If the error string being returned is a constant string, it should be a const char* rather than a char* . Also, please document that the string does not need to be freed by the caller in the doxygen. I thought loadLib handles its error itself by throwing an exception. C does not have exceptions. When the C code calls a Java method via JNI, the C code must check for an error condition and return if that occurs. if the C caller does not check, execution will continue. The results of calling many JNI methods when an exception is raised are undefined. I believe the situation is similar when calling Java code through C++, even though that language does have exceptions. The important thing to keep in mind is that Java exceptions do not translate into the JNI code. Can you fix the cases where we are not checking for java exceptions?
          Hide
          drankye Kai Zheng added a comment -

          Thanks Colin P. McCabe for the further helpful explanation. I will investigate the cases and fix them.

          Show
          drankye Kai Zheng added a comment - Thanks Colin P. McCabe for the further helpful explanation. I will investigate the cases and fix them.
          Hide
          drankye Kai Zheng added a comment -

          After looking into THROW and the question how to do exception handling in JNI, I agree we need to do the error checking after the call loadLib in Java_org_apache_hadoop_io_erasurecode_ErasureCodeNative_getLibraryName by using something like ExceptionOccurred.

          While looking around more, I found there is some codes to clean up in NativeLibraryChecker and also make consistent with other native things like openssl. So the resultant codes will remove the call to loadLib totally. The JNI function getLibraryName should assume loadLibrary is called already. If not called, then the simple constant like HADOOP_ISAL_LIBRARY will be returned.

          Will update the patch. Colin P. McCabe would you please help with check it, thanks!

          Show
          drankye Kai Zheng added a comment - After looking into THROW and the question how to do exception handling in JNI, I agree we need to do the error checking after the call loadLib in Java_org_apache_hadoop_io_erasurecode_ErasureCodeNative_getLibraryName by using something like ExceptionOccurred . While looking around more, I found there is some codes to clean up in NativeLibraryChecker and also make consistent with other native things like openssl. So the resultant codes will remove the call to loadLib totally. The JNI function getLibraryName should assume loadLibrary is called already. If not called, then the simple constant like HADOOP_ISAL_LIBRARY will be returned. Will update the patch. Colin P. McCabe would you please help with check it, thanks!
          Hide
          drankye Kai Zheng added a comment -

          Updated the patch according to above discussions. Tested and passed with or without ISA-L installed or the related options.

          Show
          drankye Kai Zheng added a comment - Updated the patch according to above discussions. Tested and passed with or without ISA-L installed or the related options.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 25s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 7m 26s trunk passed
          +1 compile 11m 6s trunk passed with JDK v1.8.0_74
          +1 compile 9m 36s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 24s trunk passed
          +1 mvnsite 1m 16s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          +1 findbugs 2m 7s trunk passed
          +1 javadoc 1m 14s trunk passed with JDK v1.8.0_74
          +1 javadoc 1m 6s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 44s the patch passed
          +1 compile 9m 4s the patch passed with JDK v1.8.0_74
          +1 cc 14m 26s root-jdk1.8.0_74 with JDK v1.8.0_74 generated 0 new + 11 unchanged - 10 fixed = 11 total (was 21)
          +1 cc 9m 4s root in the patch passed with JDK v1.8.0_74.
          +1 javac 9m 4s the patch passed
          +1 compile 7m 32s the patch passed with JDK v1.7.0_95
          +1 cc 21m 58s root-jdk1.7.0_95 with JDK v1.7.0_95 generated 0 new + 21 unchanged - 10 fixed = 21 total (was 31)
          +1 cc 7m 32s root in the patch passed with JDK v1.7.0_95.
          +1 javac 7m 32s the patch passed
          +1 checkstyle 0m 19s the patch passed
          +1 mvnsite 1m 1s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 1m 52s the patch passed
          +1 javadoc 1m 5s the patch passed with JDK v1.8.0_74
          +1 javadoc 1m 8s the patch passed with JDK v1.7.0_95
          -1 unit 13m 49s hadoop-common in the patch failed with JDK v1.8.0_74.
          +1 unit 10m 10s hadoop-common in the patch passed with JDK v1.7.0_95.
          -1 asflicense 0m 22s Patch generated 1 ASF License warnings.
          83m 23s



          Reason Tests
          JDK v1.8.0_74 Failed junit tests hadoop.fs.shell.find.TestPrint
            hadoop.fs.shell.find.TestIname



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:fbe3e86
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12795579/HADOOP-12955-v3.patch
          JIRA Issue HADOOP-12955
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux fe8d86ca24d7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 55ae143
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8938/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt
          unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8938/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8938/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/8938/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8938/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 25s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 26s trunk passed +1 compile 11m 6s trunk passed with JDK v1.8.0_74 +1 compile 9m 36s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 24s trunk passed +1 mvnsite 1m 16s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 2m 7s trunk passed +1 javadoc 1m 14s trunk passed with JDK v1.8.0_74 +1 javadoc 1m 6s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 44s the patch passed +1 compile 9m 4s the patch passed with JDK v1.8.0_74 +1 cc 14m 26s root-jdk1.8.0_74 with JDK v1.8.0_74 generated 0 new + 11 unchanged - 10 fixed = 11 total (was 21) +1 cc 9m 4s root in the patch passed with JDK v1.8.0_74. +1 javac 9m 4s the patch passed +1 compile 7m 32s the patch passed with JDK v1.7.0_95 +1 cc 21m 58s root-jdk1.7.0_95 with JDK v1.7.0_95 generated 0 new + 21 unchanged - 10 fixed = 21 total (was 31) +1 cc 7m 32s root in the patch passed with JDK v1.7.0_95. +1 javac 7m 32s the patch passed +1 checkstyle 0m 19s the patch passed +1 mvnsite 1m 1s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 52s the patch passed +1 javadoc 1m 5s the patch passed with JDK v1.8.0_74 +1 javadoc 1m 8s the patch passed with JDK v1.7.0_95 -1 unit 13m 49s hadoop-common in the patch failed with JDK v1.8.0_74. +1 unit 10m 10s hadoop-common in the patch passed with JDK v1.7.0_95. -1 asflicense 0m 22s Patch generated 1 ASF License warnings. 83m 23s Reason Tests JDK v1.8.0_74 Failed junit tests hadoop.fs.shell.find.TestPrint   hadoop.fs.shell.find.TestIname Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12795579/HADOOP-12955-v3.patch JIRA Issue HADOOP-12955 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux fe8d86ca24d7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 55ae143 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8938/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8938/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8938/testReport/ asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/8938/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8938/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          cmccabe Colin P. McCabe added a comment -
            if (error != NULL) {
              THROW(env, "java/lang/UnsatisfiedLinkError", error);
            }
          

          There needs to be a return null; here after the THROW macro.

          -      try {
          -        isalDetail = ErasureCodeNative.getLoadingFailureReason();
          -        isalDetail = ErasureCodeNative.getLibraryName();
          -        isalLoaded = true;
          -      } catch (UnsatisfiedLinkError e) {
          +      isalDetail = ErasureCodeNative.getLoadingFailureReason();
          +      if (isalDetail != null) {
                   isalLoaded = false;
          +      } else {
          +        try {
          +          isalDetail = ErasureCodeNative.getLibraryName();
          +          isalLoaded = true;
          +        } catch (UnsatisfiedLinkError e) {
          +          isalDetail = e.getMessage();
          +          isalLoaded = false;
          +        }
          

          I don't understand the rationale for doing this differently than the other native libraries. If there is an UnsatisfiedLinkError, this should be reflected by the return value of ErasureCodeNative#getLoadingFailureReason.

          Show
          cmccabe Colin P. McCabe added a comment - if (error != NULL) { THROW(env, "java/lang/UnsatisfiedLinkError" , error); } There needs to be a return null; here after the THROW macro. - try { - isalDetail = ErasureCodeNative.getLoadingFailureReason(); - isalDetail = ErasureCodeNative.getLibraryName(); - isalLoaded = true ; - } catch (UnsatisfiedLinkError e) { + isalDetail = ErasureCodeNative.getLoadingFailureReason(); + if (isalDetail != null ) { isalLoaded = false ; + } else { + try { + isalDetail = ErasureCodeNative.getLibraryName(); + isalLoaded = true ; + } catch (UnsatisfiedLinkError e) { + isalDetail = e.getMessage(); + isalLoaded = false ; + } I don't understand the rationale for doing this differently than the other native libraries. If there is an UnsatisfiedLinkError , this should be reflected by the return value of ErasureCodeNative#getLoadingFailureReason .
          Hide
          drankye Kai Zheng added a comment -

          Thanks Colin!

          There needs to be a return null; here after the THROW macro.

          Yes it's needed. Sorry I may accidentally lost the line while changing around.

          If there is an UnsatisfiedLinkError, this should be reflected by the return value of ErasureCodeNative#getLoadingFailureReason.

          Note getLoadingFailureReason can capture exceptions during loadLibrary. getLibraryName may also throw exception now. To be consistent, I'm wondering if we would do the same thing for other native things as well in case getLibraryName is allowed to throw exception in all the places.

          Show
          drankye Kai Zheng added a comment - Thanks Colin! There needs to be a return null; here after the THROW macro. Yes it's needed. Sorry I may accidentally lost the line while changing around. If there is an UnsatisfiedLinkError, this should be reflected by the return value of ErasureCodeNative#getLoadingFailureReason. Note getLoadingFailureReason can capture exceptions during loadLibrary. getLibraryName may also throw exception now. To be consistent, I'm wondering if we would do the same thing for other native things as well in case getLibraryName is allowed to throw exception in all the places.
          Hide
          drankye Kai Zheng added a comment -

          There needs to be a return null; here after the THROW macro.

          I thought you meant the code in getLibraryName. It's a new mistake. Thanks for catching it.

          Show
          drankye Kai Zheng added a comment - There needs to be a return null; here after the THROW macro. I thought you meant the code in getLibraryName. It's a new mistake. Thanks for catching it.
          Hide
          cmccabe Colin P. McCabe added a comment -

          In general, the other native libraries expect getLibraryName to succeed if the initialization succeeded. There may be some extremely rare cases where it doesn't, but this should reflect an internal bug 99% of the time. In contrast, initialization routinely fails because of missing libraries.

          I think it would be safer to have the initialization function set up a static variable so that it contained the library name. That way, we only have to expect a failure in one place, and we wouldn't even need a getLibraryName function.

          Show
          cmccabe Colin P. McCabe added a comment - In general, the other native libraries expect getLibraryName to succeed if the initialization succeeded. There may be some extremely rare cases where it doesn't, but this should reflect an internal bug 99% of the time. In contrast, initialization routinely fails because of missing libraries. I think it would be safer to have the initialization function set up a static variable so that it contained the library name. That way, we only have to expect a failure in one place, and we wouldn't even need a getLibraryName function.
          Hide
          drankye Kai Zheng added a comment -

          I think it would be safer to have the initialization function set up a static variable so that it contained the library name. That way, we only have to expect a failure in one place, and we wouldn't even need a getLibraryName function.

          Good idea and I agree with the new approach. Do you mean we'd refactor all the relate native codes? getLibraryName may be still needed as a JNI call to retrieve the library name to print in Java codes.

          Guess we'd like to have a small change for the aspect to move on this. How about right now getting away the exception catching for ErasureCodeNative.getLibraryName() and do the refactoring suggested above separately?

          Show
          drankye Kai Zheng added a comment - I think it would be safer to have the initialization function set up a static variable so that it contained the library name. That way, we only have to expect a failure in one place, and we wouldn't even need a getLibraryName function. Good idea and I agree with the new approach. Do you mean we'd refactor all the relate native codes? getLibraryName may be still needed as a JNI call to retrieve the library name to print in Java codes. Guess we'd like to have a small change for the aspect to move on this. How about right now getting away the exception catching for ErasureCodeNative.getLibraryName() and do the refactoring suggested above separately?
          Hide
          drankye Kai Zheng added a comment -

          we wouldn't even need a getLibraryName function.

          If you meant the c function const char* get_library_name(char* buf, size_t buf_len), yes it can be avoided in the new approach.
          I may use the new approach in this patch just for the erasure coding part. For other native things, we may consider it separately. Sounds good?

          Show
          drankye Kai Zheng added a comment - we wouldn't even need a getLibraryName function. If you meant the c function const char* get_library_name(char* buf, size_t buf_len) , yes it can be avoided in the new approach. I may use the new approach in this patch just for the erasure coding part. For other native things, we may consider it separately. Sounds good?
          Hide
          drankye Kai Zheng added a comment -

          Updated the patch:

          • Get the library name ready during loading phase. If loading successful, the library name is sure to be ready to get;
          • Put the library name into the IsaLibLoader structure as a field for later getting;
          • Got rid of the pure c function get_library_name and allow to access the loader object directly;
          • Removed the inconsistent codes in NativeLibraryChecker.

          The resultant codes look much clean than before. Thanks Colin for the more review in advance.

          Show
          drankye Kai Zheng added a comment - Updated the patch: Get the library name ready during loading phase. If loading successful, the library name is sure to be ready to get; Put the library name into the IsaLibLoader structure as a field for later getting; Got rid of the pure c function get_library_name and allow to access the loader object directly; Removed the inconsistent codes in NativeLibraryChecker . The resultant codes look much clean than before. Thanks Colin for the more review in advance.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 7m 31s trunk passed
          +1 compile 8m 56s trunk passed with JDK v1.8.0_74
          +1 compile 7m 55s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 22s trunk passed
          +1 mvnsite 1m 0s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 1m 43s trunk passed
          +1 javadoc 1m 0s trunk passed with JDK v1.8.0_74
          +1 javadoc 1m 4s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 46s the patch passed
          +1 compile 9m 9s the patch passed with JDK v1.8.0_74
          +1 cc 13m 54s root-jdk1.8.0_74 with JDK v1.8.0_74 generated 0 new + 11 unchanged - 10 fixed = 11 total (was 21)
          +1 cc 9m 9s root in the patch passed with JDK v1.8.0_74.
          +1 javac 9m 9s the patch passed
          +1 compile 7m 27s the patch passed with JDK v1.7.0_95
          -1 cc 21m 21s root-jdk1.7.0_95 with JDK v1.7.0_95 generated 1 new + 20 unchanged - 11 fixed = 21 total (was 31)
          +1 cc 7m 27s the patch passed
          +1 javac 7m 27s the patch passed
          +1 checkstyle 0m 22s the patch passed
          +1 mvnsite 1m 1s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 0s the patch passed
          +1 javadoc 1m 6s the patch passed with JDK v1.8.0_74
          +1 javadoc 1m 15s the patch passed with JDK v1.7.0_95
          -1 unit 14m 36s hadoop-common in the patch failed with JDK v1.8.0_74.
          -1 unit 14m 3s hadoop-common in the patch failed with JDK v1.7.0_95.
          -1 asflicense 0m 24s Patch generated 1 ASF License warnings.
          83m 37s



          Reason Tests
          JDK v1.8.0_74 Failed junit tests hadoop.fs.shell.find.TestPrint
            hadoop.fs.shell.find.TestPrint0
            hadoop.fs.shell.find.TestIname
            hadoop.fs.shell.find.TestName
            hadoop.ipc.TestRPCWaitForProxy
          JDK v1.7.0_95 Failed junit tests hadoop.net.TestDNS
            hadoop.fs.shell.find.TestPrint
            hadoop.fs.shell.find.TestPrint0
            hadoop.fs.shell.find.TestIname
            hadoop.fs.shell.find.TestName



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:fbe3e86
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12795813/HADOOP-12955-v4.patch
          JIRA Issue HADOOP-12955
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux a3517ca9210b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 0050fa5
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          cc root-jdk1.7.0_95: https://builds.apache.org/job/PreCommit-HADOOP-Build/8954/artifact/patchprocess/diff-compile-cc-root-jdk1.7.0_95.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8954/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8954/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8954/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8954/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8954/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/8954/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8954/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 31s trunk passed +1 compile 8m 56s trunk passed with JDK v1.8.0_74 +1 compile 7m 55s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 22s trunk passed +1 mvnsite 1m 0s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 43s trunk passed +1 javadoc 1m 0s trunk passed with JDK v1.8.0_74 +1 javadoc 1m 4s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 46s the patch passed +1 compile 9m 9s the patch passed with JDK v1.8.0_74 +1 cc 13m 54s root-jdk1.8.0_74 with JDK v1.8.0_74 generated 0 new + 11 unchanged - 10 fixed = 11 total (was 21) +1 cc 9m 9s root in the patch passed with JDK v1.8.0_74. +1 javac 9m 9s the patch passed +1 compile 7m 27s the patch passed with JDK v1.7.0_95 -1 cc 21m 21s root-jdk1.7.0_95 with JDK v1.7.0_95 generated 1 new + 20 unchanged - 11 fixed = 21 total (was 31) +1 cc 7m 27s the patch passed +1 javac 7m 27s the patch passed +1 checkstyle 0m 22s the patch passed +1 mvnsite 1m 1s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 0s the patch passed +1 javadoc 1m 6s the patch passed with JDK v1.8.0_74 +1 javadoc 1m 15s the patch passed with JDK v1.7.0_95 -1 unit 14m 36s hadoop-common in the patch failed with JDK v1.8.0_74. -1 unit 14m 3s hadoop-common in the patch failed with JDK v1.7.0_95. -1 asflicense 0m 24s Patch generated 1 ASF License warnings. 83m 37s Reason Tests JDK v1.8.0_74 Failed junit tests hadoop.fs.shell.find.TestPrint   hadoop.fs.shell.find.TestPrint0   hadoop.fs.shell.find.TestIname   hadoop.fs.shell.find.TestName   hadoop.ipc.TestRPCWaitForProxy JDK v1.7.0_95 Failed junit tests hadoop.net.TestDNS   hadoop.fs.shell.find.TestPrint   hadoop.fs.shell.find.TestPrint0   hadoop.fs.shell.find.TestIname   hadoop.fs.shell.find.TestName Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12795813/HADOOP-12955-v4.patch JIRA Issue HADOOP-12955 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux a3517ca9210b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0050fa5 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 cc root-jdk1.7.0_95: https://builds.apache.org/job/PreCommit-HADOOP-Build/8954/artifact/patchprocess/diff-compile-cc-root-jdk1.7.0_95.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8954/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8954/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8954/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8954/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8954/testReport/ asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/8954/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8954/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          drankye Kai Zheng added a comment -

          The cc warning and test failures are not relevant. Building passed on Windows.

          Show
          drankye Kai Zheng added a comment - The cc warning and test failures are not relevant. Building passed on Windows.
          Hide
          cmccabe Colin P. McCabe added a comment -

          +1. Thanks, Kai Zheng.

          Show
          cmccabe Colin P. McCabe added a comment - +1. Thanks, Kai Zheng .
          Hide
          drankye Kai Zheng added a comment -

          Thanks Colin for driving into this nice result and the commit!! Much learned.

          Show
          drankye Kai Zheng added a comment - Thanks Colin for driving into this nice result and the commit!! Much learned.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9538 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9538/)
          HADOOP-12955. Fix bugs in the initialization of the ISA-L library JNI (cmccabe: rev 19639785f5e9c483558ce585287b9dda9d626263)

          • hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/erasurecode/isal_load.h
          • hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/erasurecode/erasure_coder.c
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NativeLibraryChecker.java
          • hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/erasurecode/jni_erasure_code_native.c
          • hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/erasurecode/isal_load.c
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9538 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9538/ ) HADOOP-12955 . Fix bugs in the initialization of the ISA-L library JNI (cmccabe: rev 19639785f5e9c483558ce585287b9dda9d626263) hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/erasurecode/isal_load.h hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/erasurecode/erasure_coder.c hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NativeLibraryChecker.java hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/erasurecode/jni_erasure_code_native.c hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/erasurecode/isal_load.c

            People

            • Assignee:
              drankye Kai Zheng
              Reporter:
              drankye Kai Zheng
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development