Hadoop Common
  1. Hadoop Common
  2. HADOOP-8368

Use CMake rather than autotools to build native code

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: 2.0.2-alpha
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Incompatible change, Reviewed

      Description

      It would be good to use cmake rather than autotools to build the native (C/C++) code in Hadoop.

      Rationale:
      1. automake depends on shell scripts, which often have problems running on different operating systems. It would be extremely difficult, and perhaps impossible, to use autotools under Windows. Even if it were possible, it might require horrible workarounds like installing cygwin. Even on Linux variants like Ubuntu 12.04, there are major build issues because /bin/sh is the Dash shell, rather than the Bash shell as it is in other Linux versions. It is currently impossible to build the native code under Ubuntu 12.04 because of this problem.

      CMake has robust cross-platform support, including Windows. It does not use shell scripts.

      2. automake error messages are very confusing. For example, "autoreconf: cannot empty /tmp/ar0.4849: Is a directory" or "Can't locate object method "path" via package "Autom4te..." are common error messages. In order to even start debugging automake problems you need to learn shell, m4, sed, and the a bunch of other things. With CMake, all you have to learn is the syntax of CMakeLists.txt, which is simple.

      CMake can do all the stuff autotools can, such as making sure that required libraries are installed. There is a Maven plugin for CMake as well.

      3. Different versions of autotools can have very different behaviors. For example, the version installed under openSUSE defaults to putting libraries in /usr/local/lib64, whereas the version shipped with Ubuntu 11.04 defaults to installing the same libraries under /usr/local/lib. (This is why the FUSE build is currently broken when using OpenSUSE.) This is another source of build failures and complexity. If things go wrong, you will often get an error message which is incomprehensible to normal humans (see point #2).

      CMake allows you to specify the minimum_required_version of CMake that a particular CMakeLists.txt will accept. In addition, CMake maintains strict backwards compatibility between different versions. This prevents build bugs due to version skew.

      4. autoconf, automake, and libtool are large and rather slow. This adds to build time.

      For all these reasons, I think we should switch to CMake for compiling native (C/C++) code in Hadoop.

      1. HADOOP-8368-b2.003.trimmed.patch
        43 kB
        Colin Patrick McCabe
      2. HADOOP-8368-b2.003.rm.patch
        2 kB
        Colin Patrick McCabe
      3. HADOOP-8368.030.trimmed.patch
        44 kB
        Colin Patrick McCabe
      4. HADOOP-8368.030.rm.patch
        2 kB
        Colin Patrick McCabe
      5. HADOOP-8368.030.patch
        101 kB
        Colin Patrick McCabe
      6. HADOOP-8368.030.patch
        101 kB
        Colin Patrick McCabe
      7. HADOOP-8368.029.patch
        102 kB
        Colin Patrick McCabe
      8. HADOOP-8368.028.trimmed.patch
        44 kB
        Colin Patrick McCabe
      9. HADOOP-8368.028.rm.patch
        2 kB
        Colin Patrick McCabe
      10. HADOOP-8368-b2.002.trimmed.patch
        43 kB
        Colin Patrick McCabe
      11. HADOOP-8368-b2.002.rm.patch
        2 kB
        Colin Patrick McCabe
      12. HADOOP-8368-b2.001.trimmed.patch
        47 kB
        Colin Patrick McCabe
      13. HADOOP-8368-b2.001.rm.patch
        2 kB
        Colin Patrick McCabe
      14. HADOOP-8368-b2.001.patch
        259 kB
        Colin Patrick McCabe
      15. HADOOP-8368.026.trimmed.patch
        47 kB
        Colin Patrick McCabe
      16. HADOOP-8368.026.rm.patch
        2 kB
        Colin Patrick McCabe
      17. HADOOP-8368.025.trimmed.patch
        56 kB
        Colin Patrick McCabe
      18. HADOOP-8368.024.trimmed.patch
        56 kB
        Colin Patrick McCabe
      19. HADOOP-8368.023.trimmed.patch
        58 kB
        Colin Patrick McCabe
      20. HADOOP-8368.021.trimmed.patch
        50 kB
        Colin Patrick McCabe
      21. HADOOP-8368.020.trimmed.patch
        50 kB
        Colin Patrick McCabe
      22. HADOOP-8368.020.rm.patch
        2 kB
        Colin Patrick McCabe
      23. HADOOP-8368.018.trimmed.patch
        65 kB
        Colin Patrick McCabe
      24. HADOOP-8368.016.trimmed.patch
        65 kB
        Colin Patrick McCabe
      25. HADOOP-8368.015.trimmed.patch
        53 kB
        Colin Patrick McCabe
      26. HADOOP-8368.014.trimmed.patch
        53 kB
        Colin Patrick McCabe
      27. HADOOP-8368.012.rm.patch
        3 kB
        Colin Patrick McCabe
      28. HADOOP-8368.012.half.patch
        221 kB
        Colin Patrick McCabe
      29. HADOOP-8368.012.patch
        445 kB
        Colin Patrick McCabe
      30. HADOOP-8368.010.patch
        443 kB
        Colin Patrick McCabe
      31. HADOOP-8368.009.patch
        443 kB
        Colin Patrick McCabe
      32. HADOOP-8368.008.patch
        445 kB
        Colin Patrick McCabe
      33. HADOOP-8368.007.patch
        445 kB
        Colin Patrick McCabe
      34. HADOOP-8368.006.patch
        445 kB
        Colin Patrick McCabe
      35. HADOOP-8368.005.patch
        444 kB
        Colin Patrick McCabe
      36. HADOOP-8368.001.patch
        34 kB
        Colin Patrick McCabe

        Issue Links

          Activity

          Colin Patrick McCabe created issue -
          Aaron T. Myers made changes -
          Field Original Value New Value
          Affects Version/s 2.0.0 [ 12320352 ]
          Target Version/s 2.0.0 [ 12320352 ]
          Colin Patrick McCabe made changes -
          Attachment HADOOP-8368.001.patch [ 12526227 ]
          Attachment HADOOP-8368.001.patch [ 12526228 ]
          Colin Patrick McCabe made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Colin Patrick McCabe made changes -
          Attachment HADOOP-8368.001.patch [ 12526228 ]
          Colin Patrick McCabe made changes -
          Attachment HADOOP-8368.001.patch [ 12526227 ]
          Colin Patrick McCabe made changes -
          Attachment HADOOP-8368.001.patch [ 12526229 ]
          Colin Patrick McCabe made changes -
          Attachment HADOOP-8368.005.patch [ 12527803 ]
          Colin Patrick McCabe made changes -
          Attachment HADOOP-8368.006.patch [ 12527883 ]
          Colin Patrick McCabe made changes -
          Attachment HADOOP-8368.007.patch [ 12527910 ]
          Colin Patrick McCabe made changes -
          Attachment HADOOP-8368.008.patch [ 12527914 ]
          Colin Patrick McCabe made changes -
          Attachment HADOOP-8368.009.patch [ 12528132 ]
          Colin Patrick McCabe made changes -
          Attachment HADOOP-8368.010.patch [ 12528181 ]
          Colin Patrick McCabe made changes -
          Attachment HADOOP-8368.012.patch [ 12528225 ]
          Colin Patrick McCabe made changes -
          Attachment HADOOP-8368.012.half.patch [ 12528606 ]
          Colin Patrick McCabe made changes -
          Attachment HADOOP-8368.012.rm.patch [ 12528611 ]
          Colin Patrick McCabe made changes -
          Attachment HADOOP-8368.trimmed.013.patch [ 12528682 ]
          Colin Patrick McCabe made changes -
          Attachment HADOOP-8368.014.trimmed.patch [ 12528699 ]
          Colin Patrick McCabe made changes -
          Attachment HADOOP-8368.trimmed.013.patch [ 12528682 ]
          Colin Patrick McCabe made changes -
          Attachment HADOOP-8368.015.trimmed.patch [ 12528700 ]
          Colin Patrick McCabe made changes -
          Attachment HADOOP-8368.016.trimmed.patch [ 12528761 ]
          Eli Collins made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Colin Patrick McCabe made changes -
          Attachment HADOOP-8368.018.trimmed.patch [ 12528802 ]
          Colin Patrick McCabe made changes -
          Attachment HADOOP-8368.020.rm.patch [ 12529665 ]
          Colin Patrick McCabe made changes -
          Attachment HADOOP-8368.020.trimmed.patch [ 12529666 ]
          Colin Patrick McCabe made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Allen Wittenauer made changes -
          Hadoop Flags Incompatible change [ 10342 ]
          Colin Patrick McCabe made changes -
          Hadoop Flags Incompatible change [ 10342 ]
          Allen Wittenauer made changes -
          Hadoop Flags Incompatible change [ 10342 ]
          Colin Patrick McCabe made changes -
          Attachment HADOOP-8368.021.trimmed.patch [ 12529772 ]
          Colin Patrick McCabe made changes -
          Attachment HADOOP-8368.023.trimmed.patch [ 12529837 ]
          Eli Collins made changes -
          Target Version/s 2.0.0-alpha [ 12320352 ] 2.0.1-alpha [ 12321441 ]
          Colin Patrick McCabe made changes -
          Attachment HADOOP-8368.024.trimmed.patch [ 12530104 ]
          Colin Patrick McCabe made changes -
          Attachment HADOOP-8368.025.trimmed.patch [ 12530136 ]
          Colin Patrick McCabe made changes -
          Attachment HADOOP-8368.026.trimmed.patch [ 12530295 ]
          Attachment HADOOP-8368.026.rm.patch [ 12530294 ]
          Alejandro Abdelnur made changes -
          Fix Version/s 3.0.0 [ 12320357 ]
          Colin Patrick McCabe made changes -
          Attachment HADOOP-8368-b2.001.patch [ 12530660 ]
          Attachment HADOOP-8368-b2.001.rm.patch [ 12530661 ]
          Attachment HADOOP-8368-b2.001.trimmed.patch [ 12530662 ]
          Thomas Graves made changes -
          Link This issue blocks MAPREDUCE-4267 [ MAPREDUCE-4267 ]
          Colin Patrick McCabe made changes -
          Attachment HADOOP-8368-b2.002.rm.patch [ 12530832 ]
          Attachment HADOOP-8368-b2.002.trimmed.patch [ 12530833 ]
          Alejandro Abdelnur made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Incompatible change [ 10342 ] Incompatible change,Reviewed [ 10342, 10343 ]
          Fix Version/s 2.0.1-alpha [ 12321441 ]
          Fix Version/s 3.0.0 [ 12320357 ]
          Resolution Fixed [ 1 ]
          Colin Patrick McCabe made changes -
          Link This issue is related to HADOOP-8480 [ HADOOP-8480 ]
          Colin Patrick McCabe made changes -
          Link This issue blocks HDFS-3250 [ HDFS-3250 ]
          Colin Patrick McCabe made changes -
          Link This issue requires HADOOP-8481 [ HADOOP-8481 ]
          Colin Patrick McCabe made changes -
          Link This issue requires HADOOP-8481 [ HADOOP-8481 ]
          Colin Patrick McCabe made changes -
          Link This issue is related to HADOOP-8481 [ HADOOP-8481 ]
          Colin Patrick McCabe made changes -
          Link This issue requires INFRA-4881 [ INFRA-4881 ]
          Colin Patrick McCabe made changes -
          Link This issue is related to HADOOP-8482 [ HADOOP-8482 ]
          Tsz Wo Nicholas Sze made changes -
          Resolution Fixed [ 1 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          Colin Patrick McCabe made changes -
          Attachment HADOOP-8368.028.rm.patch [ 12531142 ]
          Attachment HADOOP-8368.028.trimmed.patch [ 12531143 ]
          Alejandro Abdelnur made changes -
          Status Reopened [ 4 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Colin Patrick McCabe made changes -
          Link This issue is related to INFRA-4886 [ INFRA-4886 ]
          Colin Patrick McCabe made changes -
          Link This issue is related to HADOOP-8488 [ HADOOP-8488 ]
          Colin Patrick McCabe made changes -
          Link This issue is related to INFRA-4886 [ INFRA-4886 ]
          Tsz Wo Nicholas Sze made changes -
          Resolution Fixed [ 1 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          Colin Patrick McCabe made changes -
          Attachment HADOOP-8368.029.patch [ 12531292 ]
          Colin Patrick McCabe made changes -
          Status Reopened [ 4 ] Patch Available [ 10002 ]
          Colin Patrick McCabe made changes -
          Attachment HADOOP-8368.030.patch [ 12531327 ]
          Colin Patrick McCabe made changes -
          Attachment HADOOP-8368.030.patch [ 12531379 ]
          Colin Patrick McCabe made changes -
          Attachment HADOOP-8368.030.rm.patch [ 12531439 ]
          Attachment HADOOP-8368.030.trimmed.patch [ 12531440 ]
          Colin Patrick McCabe made changes -
          Attachment HADOOP-8368-b2.003.rm.patch [ 12531703 ]
          Attachment HADOOP-8368-b2.003.trimmed.patch [ 12531704 ]
          Alejandro Abdelnur made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Trevor Robinson made changes -
          Link This issue relates to HADOOP-8538 [ HADOOP-8538 ]
          Jeff Hammerbacher made changes -
          Link This issue relates to HADOOP-8620 [ HADOOP-8620 ]
          Arun C Murthy made changes -
          Fix Version/s 2.0.2-alpha [ 12322473 ]
          Fix Version/s 2.1.0-alpha [ 12321441 ]
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Gavin made changes -
          Link This issue blocks MAPREDUCE-4267 [ MAPREDUCE-4267 ]
          Gavin made changes -
          Link This issue is depended upon by MAPREDUCE-4267 [ MAPREDUCE-4267 ]
          Colin Patrick McCabe made changes -
          Link This issue is duplicated by HDFS-3607 [ HDFS-3607 ]

            People

            • Assignee:
              Colin Patrick McCabe
              Reporter:
              Colin Patrick McCabe
            • Votes:
              0 Vote for this issue
              Watchers:
              23 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development