Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1920

libhdfs does not build for ARM processors

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.23.0
    • Component/s: libhdfs
    • Labels:
      None
    • Environment:
    • Hadoop Flags:
      Reviewed

      Description

      $ ant compile -Dcompile.native=true -Dcompile.c++=1 -Dlibhdfs=1 -Dfusedfs=1
      ...
      create-libhdfs-configure:
      ...
      [exec] configure: error: Unsupported CPU architecture "armv7l"

      Once the CPU arch check is fixed in src/c++/libhdfs/m4/apsupport.m4, then next issue is -m32:

      $ ant compile -Dcompile.native=true -Dcompile.c++=1 -Dlibhdfs=1 -Dfusedfs=1
      ...

      compile-c++-libhdfs:
      [exec] /bin/bash ./libtool --tag=CC --mode=compile gcc -DPACKAGE_NAME=\"libhdfs\" -DPACKAGE_TARNAME=\"libhdfs\" -DPACKAGE_VERSION=\"0.1.0\" -DPACKAGE_STRING=\"libhdfs\ 0.1.0\" -DPACKAGE_BUGREPORT=\"omalley@apache.org\" -DPACKAGE_URL=\"\" -DPACKAGE=\"libhdfs\" -DVERSION=\"0.1.0\" -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1 -DHAVE_DLFCN_H=1 -DLT_OBJDIR=\".libs/\" -Dsize_t=unsigned\ int -Dconst=/**/ -Dvolatile=/**/ -I. -I/home/trobinson/dev/hadoop-hdfs/src/c+/libhdfs -g -O2 -DOS_LINUX -DDSO_DLFCN -DCPU=\"arm\" -m32 -I/usr/lib/jvm/java-6-openjdk/include -I/usr/lib/jvm/java-6-openjdk/include/arm -Wall -Wstrict-prototypes -MT hdfs.lo -MD -MP -MF .deps/hdfs.Tpo -c -o hdfs.lo /home/trobinson/dev/hadoop-hdfs/src/c+/libhdfs/hdfs.c
      [exec] make: Warning: File `.deps/hdfs_write.Po' has modification time 2.1 s in the future
      [exec] libtool: compile: gcc -DPACKAGE_NAME=\"libhdfs\" -DPACKAGE_TARNAME=\"libhdfs\" -DPACKAGE_VERSION=\"0.1.0\" "-DPACKAGE_STRING=\"libhdfs 0.1.0\"" -DPACKAGE_BUGREPORT=\"omalley@apache.org\" -DPACKAGE_URL=\"\" -DPACKAGE=\"libhdfs\" -DVERSION=\"0.1.0\" -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1 -DHAVE_DLFCN_H=1 -DLT_OBJDIR=\".libs/\" "-Dsize_t=unsigned int" "-Dconst=/*/" "-Dvolatile=/*/" -I. -I/home/trobinson/dev/hadoop-hdfs/src/c+/libhdfs -g -O2 -DOS_LINUX -DDSO_DLFCN -DCPU=\"arm\" -m32 -I/usr/lib/jvm/java-6-openjdk/include -I/usr/lib/jvm/java-6-openjdk/include/arm -Wall -Wstrict-prototypes -MT hdfs.lo -MD -MP -MF .deps/hdfs.Tpo -c /home/trobinson/dev/hadoop-hdfs/src/c+/libhdfs/hdfs.c -fPIC -DPIC -o .libs/hdfs.o
      [exec] cc1: error: unrecognized command line option "-m32"
      [exec] make: *** [hdfs.lo] Error 1

      Here, gcc does not support -m32 for the ARM target, so -m$

      {JVM_ARCH}

      must be omitted from CFLAGS and LDFLAGS.

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #680 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/680/)
          HDFS-1920. libhdfs does not build for ARM processors. Contributed by Trevor Robinson

          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1128473
          Files :

          • /hadoop/hdfs/trunk/src/c++/libhdfs/configure.ac
          • /hadoop/hdfs/trunk/CHANGES.txt
          • /hadoop/hdfs/trunk/src/c++/libhdfs/m4/apsupport.m4
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #680 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/680/ ) HDFS-1920 . libhdfs does not build for ARM processors. Contributed by Trevor Robinson eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1128473 Files : /hadoop/hdfs/trunk/src/c++/libhdfs/configure.ac /hadoop/hdfs/trunk/CHANGES.txt /hadoop/hdfs/trunk/src/c++/libhdfs/m4/apsupport.m4
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #693 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/693/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #693 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/693/ )
          Hide
          Eli Collins added a comment -

          I've committed this to trunk. Thanks Trevor!

          Show
          Eli Collins added a comment - I've committed this to trunk. Thanks Trevor!
          Hide
          Eli Collins added a comment -

          +1 looks good. (I also verified that x64 libhdfs still builds/runs).

          Show
          Eli Collins added a comment - +1 looks good. (I also verified that x64 libhdfs still builds/runs).
          Hide
          Trevor Robinson added a comment -

          No tests included because this change just fixes a build failure. Manually verified that x86-64 builds unchanged (-m64 is properly specified) and that ARM now builds (-m32 is not specified).

          Core unit test failures are existing and unrelated issues. This change only affects libhdfs.

          Would a committer please review the change?

          Show
          Trevor Robinson added a comment - No tests included because this change just fixes a build failure. Manually verified that x86-64 builds unchanged (-m64 is properly specified) and that ARM now builds (-m32 is not specified). Core unit test failures are existing and unrelated issues. This change only affects libhdfs. Would a committer please review the change?
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12478901/hadoop-hdfs-arm.patch
          against trunk revision 1102153.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.cli.TestHDFSCLI
          org.apache.hadoop.hdfs.TestDFSShell
          org.apache.hadoop.hdfs.TestDFSStorageStateRecovery
          org.apache.hadoop.hdfs.TestFileConcurrentReader
          org.apache.hadoop.tools.TestJMXGet

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/501//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/501//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/501//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12478901/hadoop-hdfs-arm.patch against trunk revision 1102153. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestHDFSCLI org.apache.hadoop.hdfs.TestDFSShell org.apache.hadoop.hdfs.TestDFSStorageStateRecovery org.apache.hadoop.hdfs.TestFileConcurrentReader org.apache.hadoop.tools.TestJMXGet +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/501//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/501//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/501//console This message is automatically generated.
          Hide
          Trevor Robinson added a comment -

          This patch adds a case for ARM to src/c++/libhdfs/m4/apsupport.m4 and omits -m$

          {JVM_ARCH}

          from CFLAGS and LDFLAGS if $host_cpu starts with "arm".

          Show
          Trevor Robinson added a comment - This patch adds a case for ARM to src/c++/libhdfs/m4/apsupport.m4 and omits -m$ {JVM_ARCH} from CFLAGS and LDFLAGS if $host_cpu starts with "arm".

            People

            • Assignee:
              Trevor Robinson
              Reporter:
              Trevor Robinson
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development