Details

    • Type: Sub-task Sub-task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: HADOOP-10388
    • Component/s: None
    • Labels:
      None

      Description

      Add pom.xml infrastructure for hadoop-native-core, so that it builds under Maven. We can look to how we integrated CMake into hadoop-hdfs-project and hadoop-common-project for inspiration here. In the long term, it would be nice to use a Maven plugin here (see HADOOP-8887)

      1. HADOOP-10444.v1.patch
        415 kB
        Binglin Chang
      2. HADOOP-10444.v2.patch
        412 kB
        Binglin Chang

        Activity

        Hide
        Colin Patrick McCabe added a comment -

        +1. Thanks, Binglin.

        Show
        Colin Patrick McCabe added a comment - +1. Thanks, Binglin.
        Hide
        Colin Patrick McCabe added a comment -

        I think the issue with HADOOP-9648 is that it needs someone with more familiarity with YARN (and MacOS) to look at it. I know there are a few YARN developers who use MacOS, so maybe start by asking them?

        Show
        Colin Patrick McCabe added a comment - I think the issue with HADOOP-9648 is that it needs someone with more familiarity with YARN (and MacOS) to look at it. I know there are a few YARN developers who use MacOS, so maybe start by asking them?
        Hide
        Binglin Chang added a comment -

        If there is stuff that doesn't work on MacOS, we should just fix that stuff.

        This reminds me of HADOOP-9648, as you have done lot of native library stuff, could you help take a look?

        Show
        Binglin Chang added a comment - If there is stuff that doesn't work on MacOS, we should just fix that stuff. This reminds me of HADOOP-9648 , as you have done lot of native library stuff, could you help take a look?
        Hide
        Binglin Chang added a comment -

        Thanks for the review Colin. I updated the patch addressing your comments.
        Changes:
        1. change -Pnative-core to -Pnative
        2. remove assembly, we can add hdfs.h(or just share old one) and yarn.h latter when we have them.

        Show
        Binglin Chang added a comment - Thanks for the review Colin. I updated the patch addressing your comments. Changes: 1. change -Pnative-core to -Pnative 2. remove assembly, we can add hdfs.h(or just share old one) and yarn.h latter when we have them.
        Hide
        Colin Patrick McCabe added a comment -

        1. change project structure to maven, move all code from hadoop-native-core to hadoop-native-core/src/main/native

        OK

        2. change build dir from hadoop-native-core to hadoop-native-core/target/native

        OK

        3. add pom artifact hadoop-native-core, make according changes in hadoop-dist/pom.xml hadoop-project/pom.xml and pom.xml

        looks good

        4. add a new assembly hadoop-native-core-dist to copy .h files only.

        This is a little tricky. We don't want to copy our internal header files (they can't help anyone but us.) We will eventually want to copy the hdfs.h public API header file, but that's about it for headers.

        5. add a new profile: native-core to activate hadoop-native-core compile/test/package, I didn't reuse -Pnative, cause currently -Pnative can't work on MacOSX.

        I'd prefer to stick with -Pnative, since that's the profile everyone already knows about. Changing the way all of our build slaves (apache and otherwise) operate is a huge undertaking, not to mention READMEs, howtos, web pages, etc. If there is stuff that doesn't work on MacOS, we should just fix that stuff. In the meantime, how about running Maven from the hadoop-native-core directory... that way if there is stuff in other subprojects that doesn't work on MacOS, you won't have to worry about it?

        Show
        Colin Patrick McCabe added a comment - 1. change project structure to maven, move all code from hadoop-native-core to hadoop-native-core/src/main/native OK 2. change build dir from hadoop-native-core to hadoop-native-core/target/native OK 3. add pom artifact hadoop-native-core, make according changes in hadoop-dist/pom.xml hadoop-project/pom.xml and pom.xml looks good 4. add a new assembly hadoop-native-core-dist to copy .h files only. This is a little tricky. We don't want to copy our internal header files (they can't help anyone but us.) We will eventually want to copy the hdfs.h public API header file, but that's about it for headers. 5. add a new profile: native-core to activate hadoop-native-core compile/test/package, I didn't reuse -Pnative, cause currently -Pnative can't work on MacOSX. I'd prefer to stick with -Pnative , since that's the profile everyone already knows about. Changing the way all of our build slaves (apache and otherwise) operate is a huge undertaking, not to mention READMEs, howtos, web pages, etc. If there is stuff that doesn't work on MacOS, we should just fix that stuff. In the meantime, how about running Maven from the hadoop-native-core directory... that way if there is stuff in other subprojects that doesn't work on MacOS, you won't have to worry about it?
        Hide
        Colin Patrick McCabe added a comment -

        Thanks, Binglin. I'll try to take a look today. It will be nice to have maven integration

        Show
        Colin Patrick McCabe added a comment - Thanks, Binglin. I'll try to take a look today. It will be nice to have maven integration
        Hide
        Binglin Chang added a comment -

        Hi Colin, the patch moves all the code, so it's better to get this review and committed soon to avoid conflicts.

        Show
        Binglin Chang added a comment - Hi Colin, the patch moves all the code, so it's better to get this review and committed soon to avoid conflicts.
        Hide
        Binglin Chang added a comment -

        Changes:
        1. change project structure to maven, move all code from hadoop-native-core to hadoop-native-core/src/main/native
        2. change build dir from hadoop-native-core to hadoop-native-core/target/native
        3. add pom artifact hadoop-native-core, make according changes in hadoop-dist/pom.xml hadoop-project/pom.xml and pom.xml
        4. add a new assembly hadoop-native-core-dist to copy .h files only.
        5. add a new profile: native-core to activate hadoop-native-core compile/test/package, I didn't reuse -Pnative, cause currently -Pnative can't work on MacOSX.

        When invoke using: mvn package -Pdist -DskipTests -Pnative -Pnative-core
        native client libraries are packaged in following locations:

        decster@localhost:~/hadoop-trunk$ ll hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/{include/common,include/rpc,lib/native}
        hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/include/common:
        total 76
        -rw-rw-r-- 1 decster decster  2380 2014-05-29 00:10 hadoop_err.h
        -rw-rw-r-- 1 decster decster  1025 2014-05-29 00:10 net.h
        -rw-rw-r-- 1 decster decster 23781 2014-05-29 00:10 queue.h
        -rw-rw-r-- 1 decster decster  1349 2014-05-29 00:10 string.h
        -rw-rw-r-- 1 decster decster  4936 2014-05-29 00:10 test.h
        -rw-rw-r-- 1 decster decster 25776 2014-05-29 00:10 tree.h
        -rw-rw-r-- 1 decster decster  1189 2014-05-29 00:10 user.h
        
        hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/include/rpc:
        total 36
        -rw-rw-r-- 1 decster decster 3479 2014-05-29 00:10 call.h
        -rw-rw-r-- 1 decster decster 1561 2014-05-29 00:10 client_id.h
        -rw-rw-r-- 1 decster decster 6392 2014-05-29 00:10 conn.h
        -rw-rw-r-- 1 decster decster 2939 2014-05-29 00:10 messenger.h
        -rw-rw-r-- 1 decster decster 5395 2014-05-29 00:10 proxy.h
        -rw-rw-r-- 1 decster decster 3700 2014-05-29 00:10 reactor.h
        -rw-rw-r-- 1 decster decster 2647 2014-05-29 00:10 varint.h
        
        hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/lib/native:
        total 7464
        -rw-rw-r-- 1 decster decster 1239398 2014-05-29 00:10 libhadoop.a
        -rw-rw-r-- 1 decster decster 1374292 2014-05-29 00:10 libhadooppipes.a
        lrwxrwxrwx 1 decster decster      18 2014-05-29 00:10 libhadoop.so -> libhadoop.so.1.0.0*
        -rwxrwxr-x 1 decster decster  721343 2014-05-29 00:10 libhadoop.so.1.0.0*
        -rw-rw-r-- 1 decster decster  453186 2014-05-29 00:10 libhadooputils.a
        -rw-rw-r-- 1 decster decster  317370 2014-05-29 00:10 libhdfs.a
        lrwxrwxrwx 1 decster decster      17 2014-05-29 00:10 libhdfs-core.so -> libhdfs-core.so.1*
        lrwxrwxrwx 1 decster decster      21 2014-05-29 00:10 libhdfs-core.so.1 -> libhdfs-core.so.1.0.0*
        -rwxrwxr-x 1 decster decster 2747719 2014-05-29 00:10 libhdfs-core.so.1.0.0*
        lrwxrwxrwx 1 decster decster      16 2014-05-29 00:10 libhdfs.so -> libhdfs.so.0.0.0*
        -rwxrwxr-x 1 decster decster  212967 2014-05-29 00:10 libhdfs.so.0.0.0*
        lrwxrwxrwx 1 decster decster      17 2014-05-29 00:10 libyarn-core.so -> libyarn-core.so.1*
        lrwxrwxrwx 1 decster decster      21 2014-05-29 00:10 libyarn-core.so.1 -> libyarn-core.so.1.0.0*
        -rwxrwxr-x 1 decster decster  564923 2014-05-29 00:10 libyarn-core.so.1.0.0*
        
        Show
        Binglin Chang added a comment - Changes: 1. change project structure to maven, move all code from hadoop-native-core to hadoop-native-core/src/main/native 2. change build dir from hadoop-native-core to hadoop-native-core/target/native 3. add pom artifact hadoop-native-core, make according changes in hadoop-dist/pom.xml hadoop-project/pom.xml and pom.xml 4. add a new assembly hadoop-native-core-dist to copy .h files only. 5. add a new profile: native-core to activate hadoop-native-core compile/test/package, I didn't reuse -Pnative, cause currently -Pnative can't work on MacOSX. When invoke using: mvn package -Pdist -DskipTests -Pnative -Pnative-core native client libraries are packaged in following locations: decster@localhost:~/hadoop-trunk$ ll hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/{include/common,include/rpc,lib/ native } hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/include/common: total 76 -rw-rw-r-- 1 decster decster 2380 2014-05-29 00:10 hadoop_err.h -rw-rw-r-- 1 decster decster 1025 2014-05-29 00:10 net.h -rw-rw-r-- 1 decster decster 23781 2014-05-29 00:10 queue.h -rw-rw-r-- 1 decster decster 1349 2014-05-29 00:10 string.h -rw-rw-r-- 1 decster decster 4936 2014-05-29 00:10 test.h -rw-rw-r-- 1 decster decster 25776 2014-05-29 00:10 tree.h -rw-rw-r-- 1 decster decster 1189 2014-05-29 00:10 user.h hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/include/rpc: total 36 -rw-rw-r-- 1 decster decster 3479 2014-05-29 00:10 call.h -rw-rw-r-- 1 decster decster 1561 2014-05-29 00:10 client_id.h -rw-rw-r-- 1 decster decster 6392 2014-05-29 00:10 conn.h -rw-rw-r-- 1 decster decster 2939 2014-05-29 00:10 messenger.h -rw-rw-r-- 1 decster decster 5395 2014-05-29 00:10 proxy.h -rw-rw-r-- 1 decster decster 3700 2014-05-29 00:10 reactor.h -rw-rw-r-- 1 decster decster 2647 2014-05-29 00:10 varint.h hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/lib/ native : total 7464 -rw-rw-r-- 1 decster decster 1239398 2014-05-29 00:10 libhadoop.a -rw-rw-r-- 1 decster decster 1374292 2014-05-29 00:10 libhadooppipes.a lrwxrwxrwx 1 decster decster 18 2014-05-29 00:10 libhadoop.so -> libhadoop.so.1.0.0* -rwxrwxr-x 1 decster decster 721343 2014-05-29 00:10 libhadoop.so.1.0.0* -rw-rw-r-- 1 decster decster 453186 2014-05-29 00:10 libhadooputils.a -rw-rw-r-- 1 decster decster 317370 2014-05-29 00:10 libhdfs.a lrwxrwxrwx 1 decster decster 17 2014-05-29 00:10 libhdfs-core.so -> libhdfs-core.so.1* lrwxrwxrwx 1 decster decster 21 2014-05-29 00:10 libhdfs-core.so.1 -> libhdfs-core.so.1.0.0* -rwxrwxr-x 1 decster decster 2747719 2014-05-29 00:10 libhdfs-core.so.1.0.0* lrwxrwxrwx 1 decster decster 16 2014-05-29 00:10 libhdfs.so -> libhdfs.so.0.0.0* -rwxrwxr-x 1 decster decster 212967 2014-05-29 00:10 libhdfs.so.0.0.0* lrwxrwxrwx 1 decster decster 17 2014-05-29 00:10 libyarn-core.so -> libyarn-core.so.1* lrwxrwxrwx 1 decster decster 21 2014-05-29 00:10 libyarn-core.so.1 -> libyarn-core.so.1.0.0* -rwxrwxr-x 1 decster decster 564923 2014-05-29 00:10 libyarn-core.so.1.0.0*
        Hide
        Colin Patrick McCabe added a comment -

        It would also be nice to be able to spin up an HDFS cluster during a unit test. If we got hadoop-native-core-project to build after hadoop-hdfs-project, we should be able to just invoke a binary from that project creating a MiniDFSCluster.

        Show
        Colin Patrick McCabe added a comment - It would also be nice to be able to spin up an HDFS cluster during a unit test. If we got hadoop-native-core-project to build after hadoop-hdfs-project, we should be able to just invoke a binary from that project creating a MiniDFSCluster.

          People

          • Assignee:
            Binglin Chang
            Reporter:
            Colin Patrick McCabe
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development