Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-11484

hadoop-mapreduce-client-nativetask fails to build on ARM AARCH64 due to x86 asm statements

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0-alpha1
    • Component/s: native
    • Labels:
      None
    • Environment:

      ARM aarch64 development board

    • Target Version/s:

      Description

      Hadoop fails to build on ARM aarch64 (or any non x86 platform) because of the following in

      hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/primitives.h

      /**
       * little-endian to big-endian or vice versa
       */
      inline uint32_t bswap(uint32_t val) {
        __asm__("bswap %0" : "=r" (val) : "0" (val));
        return val;
      }
      
      inline uint64_t bswap64(uint64_t val) {
      #ifdef __X64
        __asm__("bswapq %0" : "=r" (val) : "0" (val));
      #else
      
        uint64_t lower = val & 0xffffffffU;
        uint32_t higher = (val >> 32) & 0xffffffffU;
      
        lower = bswap(lower);
        higher = bswap(higher);
      
        return (lower << 32) + higher;
      
      #endif
        return val;
      }
      

      The following also fails in

      hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Checksum.cc

      static uint32_t cpuid(uint32_t eax_in) {
        uint32_t eax, ebx, ecx, edx;
      #  if defined(__PIC__) && !defined(__LP64__)
      // 32-bit PIC code uses the ebx register for the base offset --
      // have to save and restore it on the stack
        asm("pushl %%ebx\n\t"
            "cpuid\n\t"
            "movl %%ebx, %[ebx]\n\t"
            "popl %%ebx" : "=a" (eax), [ebx] "=r"(ebx), "=c"(ecx), "=d"(edx) : "a" (eax_in) 
            : "cc");
      #  else
        asm("cpuid" : "=a" (eax), "=b"(ebx), "=c"(ecx), "=d"(edx) : "a"(eax_in)
            : "cc");
      #  endif
      
        return ecx;
      }
      

        Issue Links

          Activity

          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12692544/HADOOP-11484.001.patch
          against trunk revision ce29074.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5413//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5413//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12692544/HADOOP-11484.001.patch against trunk revision ce29074. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5413//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5413//console This message is automatically generated.
          Hide
          cmccabe Colin P. McCabe added a comment - - edited

          +1.

          btw, this should be against "3.0", not "trunk-win"... the latter branch is not being used any more.

          thanks

          Show
          cmccabe Colin P. McCabe added a comment - - edited +1. btw, this should be against "3.0", not "trunk-win"... the latter branch is not being used any more. thanks
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #6903 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6903/)
          HADOOP-11484. hadoop-mapreduce-client-nativetask fails to build on ARM AARCH64 due to x86 asm statements (Edward Nevill via Colin P. McCabe) (cmccabe: rev 220a49de479a632068ad37f11c6cdd17327dacd9)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/primitives.h
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Checksum.cc
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #6903 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6903/ ) HADOOP-11484 . hadoop-mapreduce-client-nativetask fails to build on ARM AARCH64 due to x86 asm statements (Edward Nevill via Colin P. McCabe) (cmccabe: rev 220a49de479a632068ad37f11c6cdd17327dacd9) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/primitives.h hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Checksum.cc hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #6904 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6904/)
          HADOOP-11484: move CHANGES.txt entry to 3.0 (cmccabe: rev 39c1bcf7d9c9331d25ca0aded85a293df04e0b52)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #6904 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6904/ ) HADOOP-11484 : move CHANGES.txt entry to 3.0 (cmccabe: rev 39c1bcf7d9c9331d25ca0aded85a293df04e0b52) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #81 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/81/)
          HADOOP-11484. hadoop-mapreduce-client-nativetask fails to build on ARM AARCH64 due to x86 asm statements (Edward Nevill via Colin P. McCabe) (cmccabe: rev 220a49de479a632068ad37f11c6cdd17327dacd9)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Checksum.cc
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/primitives.h
            HADOOP-11484: move CHANGES.txt entry to 3.0 (cmccabe: rev 39c1bcf7d9c9331d25ca0aded85a293df04e0b52)
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #81 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/81/ ) HADOOP-11484 . hadoop-mapreduce-client-nativetask fails to build on ARM AARCH64 due to x86 asm statements (Edward Nevill via Colin P. McCabe) (cmccabe: rev 220a49de479a632068ad37f11c6cdd17327dacd9) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Checksum.cc hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/primitives.h HADOOP-11484 : move CHANGES.txt entry to 3.0 (cmccabe: rev 39c1bcf7d9c9331d25ca0aded85a293df04e0b52) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #815 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/815/)
          HADOOP-11484. hadoop-mapreduce-client-nativetask fails to build on ARM AARCH64 due to x86 asm statements (Edward Nevill via Colin P. McCabe) (cmccabe: rev 220a49de479a632068ad37f11c6cdd17327dacd9)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Checksum.cc
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/primitives.h
            HADOOP-11484: move CHANGES.txt entry to 3.0 (cmccabe: rev 39c1bcf7d9c9331d25ca0aded85a293df04e0b52)
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #815 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/815/ ) HADOOP-11484 . hadoop-mapreduce-client-nativetask fails to build on ARM AARCH64 due to x86 asm statements (Edward Nevill via Colin P. McCabe) (cmccabe: rev 220a49de479a632068ad37f11c6cdd17327dacd9) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Checksum.cc hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/primitives.h HADOOP-11484 : move CHANGES.txt entry to 3.0 (cmccabe: rev 39c1bcf7d9c9331d25ca0aded85a293df04e0b52) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          stevel@apache.org Steve Loughran added a comment -

          this looks like it addresses the ARM compile issue (we'll have to trust you that it works), but there's some asm code in the bswap method that isn't ifdef'd out on non-x86 systems.

          Is there an easy way to downgrade to non-ASM alternatives on other CPU families, so make sure this patch helps compilation on all platforms?

          Show
          stevel@apache.org Steve Loughran added a comment - this looks like it addresses the ARM compile issue (we'll have to trust you that it works), but there's some asm code in the bswap method that isn't ifdef'd out on non-x86 systems. Is there an easy way to downgrade to non-ASM alternatives on other CPU families, so make sure this patch helps compilation on all platforms?
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2013 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2013/)
          HADOOP-11484. hadoop-mapreduce-client-nativetask fails to build on ARM AARCH64 due to x86 asm statements (Edward Nevill via Colin P. McCabe) (cmccabe: rev 220a49de479a632068ad37f11c6cdd17327dacd9)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/primitives.h
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Checksum.cc
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
            HADOOP-11484: move CHANGES.txt entry to 3.0 (cmccabe: rev 39c1bcf7d9c9331d25ca0aded85a293df04e0b52)
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2013 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2013/ ) HADOOP-11484 . hadoop-mapreduce-client-nativetask fails to build on ARM AARCH64 due to x86 asm statements (Edward Nevill via Colin P. McCabe) (cmccabe: rev 220a49de479a632068ad37f11c6cdd17327dacd9) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/primitives.h hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Checksum.cc hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt HADOOP-11484 : move CHANGES.txt entry to 3.0 (cmccabe: rev 39c1bcf7d9c9331d25ca0aded85a293df04e0b52) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #78 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/78/)
          HADOOP-11484. hadoop-mapreduce-client-nativetask fails to build on ARM AARCH64 due to x86 asm statements (Edward Nevill via Colin P. McCabe) (cmccabe: rev 220a49de479a632068ad37f11c6cdd17327dacd9)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Checksum.cc
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/primitives.h
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
            HADOOP-11484: move CHANGES.txt entry to 3.0 (cmccabe: rev 39c1bcf7d9c9331d25ca0aded85a293df04e0b52)
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #78 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/78/ ) HADOOP-11484 . hadoop-mapreduce-client-nativetask fails to build on ARM AARCH64 due to x86 asm statements (Edward Nevill via Colin P. McCabe) (cmccabe: rev 220a49de479a632068ad37f11c6cdd17327dacd9) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Checksum.cc hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/primitives.h hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt HADOOP-11484 : move CHANGES.txt entry to 3.0 (cmccabe: rev 39c1bcf7d9c9331d25ca0aded85a293df04e0b52) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #82 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/82/)
          HADOOP-11484. hadoop-mapreduce-client-nativetask fails to build on ARM AARCH64 due to x86 asm statements (Edward Nevill via Colin P. McCabe) (cmccabe: rev 220a49de479a632068ad37f11c6cdd17327dacd9)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Checksum.cc
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/primitives.h
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
            HADOOP-11484: move CHANGES.txt entry to 3.0 (cmccabe: rev 39c1bcf7d9c9331d25ca0aded85a293df04e0b52)
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #82 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/82/ ) HADOOP-11484 . hadoop-mapreduce-client-nativetask fails to build on ARM AARCH64 due to x86 asm statements (Edward Nevill via Colin P. McCabe) (cmccabe: rev 220a49de479a632068ad37f11c6cdd17327dacd9) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Checksum.cc hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/primitives.h hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt HADOOP-11484 : move CHANGES.txt entry to 3.0 (cmccabe: rev 39c1bcf7d9c9331d25ca0aded85a293df04e0b52) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2032 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2032/)
          HADOOP-11484. hadoop-mapreduce-client-nativetask fails to build on ARM AARCH64 due to x86 asm statements (Edward Nevill via Colin P. McCabe) (cmccabe: rev 220a49de479a632068ad37f11c6cdd17327dacd9)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/primitives.h
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Checksum.cc
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
            HADOOP-11484: move CHANGES.txt entry to 3.0 (cmccabe: rev 39c1bcf7d9c9331d25ca0aded85a293df04e0b52)
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2032 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2032/ ) HADOOP-11484 . hadoop-mapreduce-client-nativetask fails to build on ARM AARCH64 due to x86 asm statements (Edward Nevill via Colin P. McCabe) (cmccabe: rev 220a49de479a632068ad37f11c6cdd17327dacd9) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/primitives.h hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Checksum.cc hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt HADOOP-11484 : move CHANGES.txt entry to 3.0 (cmccabe: rev 39c1bcf7d9c9331d25ca0aded85a293df04e0b52) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          cmccabe Colin P. McCabe added a comment - - edited

          this looks like it addresses the ARM compile issue (we'll have to trust you that it works), but there's some asm code in the bswap method that isn't ifdef'd out on non-x86 systems. Is there an easy way to downgrade to non-ASM alternatives on other CPU families, so make sure this patch helps compilation on all platforms?

          Edit: Good find. I have filed HADOOP-11505 to fix this.

          Show
          cmccabe Colin P. McCabe added a comment - - edited this looks like it addresses the ARM compile issue (we'll have to trust you that it works), but there's some asm code in the bswap method that isn't ifdef'd out on non-x86 systems. Is there an easy way to downgrade to non-ASM alternatives on other CPU families, so make sure this patch helps compilation on all platforms? Edit: Good find. I have filed HADOOP-11505 to fix this.
          Hide
          gkasap Georgios Kasapoglou added a comment -

          Hello,
          I've used gcc's __builtin_bswap instead.

          +++ ./src/main/native/src/lib/primitives.h 2015-05-08 16:52:22.568992860 +0300
          @@ -97,13 +97,15 @@

          • little-endian to big-endian or vice versa
            */
            inline uint32_t bswap(uint32_t val) { - __asm__("bswap %0" : "=r" (val) : "0" (val)); +// __asm__("bswap %0" : "=r" (val) : "0" (val)); + val=__builtin_bswap32 (val); return val; }

          inline uint64_t bswap64(uint64_t val) {
          #ifdef __X64

          • _asm_("bswapq %0" : "=r" (val) : "0" (val));
            +// _asm_("bswapq %0" : "=r" (val) : "0" (val));
            + val=__builtin_bswap64(val);
            #else

          uint64_t lower = val & 0xffffffffU;

          Show
          gkasap Georgios Kasapoglou added a comment - Hello, I've used gcc's __builtin_bswap instead. +++ ./src/main/native/src/lib/primitives.h 2015-05-08 16:52:22.568992860 +0300 @@ -97,13 +97,15 @@ little-endian to big-endian or vice versa */ inline uint32_t bswap(uint32_t val) { - __asm__("bswap %0" : "=r" (val) : "0" (val)); +// __asm__("bswap %0" : "=r" (val) : "0" (val)); + val=__builtin_bswap32 (val); return val; } inline uint64_t bswap64(uint64_t val) { #ifdef __X64 _ asm _("bswapq %0" : "=r" (val) : "0" (val)); +// _ asm _("bswapq %0" : "=r" (val) : "0" (val)); + val=__builtin_bswap64(val); #else uint64_t lower = val & 0xffffffffU;

            People

            • Assignee:
              enevill Edward Nevill
              Reporter:
              enevill Edward Nevill
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development