Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: native
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Since our min version is now JDK7, there's hardlink support via Files. This means we can deprecate the JNI implementation and discontinue usage.

        Activity

        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #2167 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2167/)
        HADOOP-12055. Deprecate usage of NativeIO#link. Contributed by Andrew Wang. (cnauroth: rev e46cb800028c95f9bce575d05268cd10d0913222)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java
        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java
        • hadoop-common-project/hadoop-common/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2167 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2167/ ) HADOOP-12055 . Deprecate usage of NativeIO#link. Contributed by Andrew Wang. (cnauroth: rev e46cb800028c95f9bce575d05268cd10d0913222) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java hadoop-common-project/hadoop-common/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #219 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/219/)
        HADOOP-12055. Deprecate usage of NativeIO#link. Contributed by Andrew Wang. (cnauroth: rev e46cb800028c95f9bce575d05268cd10d0913222)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java
        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java
        • hadoop-common-project/hadoop-common/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #219 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/219/ ) HADOOP-12055 . Deprecate usage of NativeIO#link. Contributed by Andrew Wang. (cnauroth: rev e46cb800028c95f9bce575d05268cd10d0913222) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java hadoop-common-project/hadoop-common/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #210 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/210/)
        HADOOP-12055. Deprecate usage of NativeIO#link. Contributed by Andrew Wang. (cnauroth: rev e46cb800028c95f9bce575d05268cd10d0913222)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java
        • hadoop-common-project/hadoop-common/CHANGES.txt
        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #210 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/210/ ) HADOOP-12055 . Deprecate usage of NativeIO#link. Contributed by Andrew Wang. (cnauroth: rev e46cb800028c95f9bce575d05268cd10d0913222) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Hdfs-trunk #2149 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2149/)
        HADOOP-12055. Deprecate usage of NativeIO#link. Contributed by Andrew Wang. (cnauroth: rev e46cb800028c95f9bce575d05268cd10d0913222)

        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java
        • hadoop-common-project/hadoop-common/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #2149 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2149/ ) HADOOP-12055 . Deprecate usage of NativeIO#link. Contributed by Andrew Wang. (cnauroth: rev e46cb800028c95f9bce575d05268cd10d0913222) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #951 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/951/)
        HADOOP-12055. Deprecate usage of NativeIO#link. Contributed by Andrew Wang. (cnauroth: rev e46cb800028c95f9bce575d05268cd10d0913222)

        • hadoop-common-project/hadoop-common/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java
        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #951 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/951/ ) HADOOP-12055 . Deprecate usage of NativeIO#link. Contributed by Andrew Wang. (cnauroth: rev e46cb800028c95f9bce575d05268cd10d0913222) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #221 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/221/)
        HADOOP-12055. Deprecate usage of NativeIO#link. Contributed by Andrew Wang. (cnauroth: rev e46cb800028c95f9bce575d05268cd10d0913222)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java
        • hadoop-common-project/hadoop-common/CHANGES.txt
        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #221 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/221/ ) HADOOP-12055 . Deprecate usage of NativeIO#link. Contributed by Andrew Wang. (cnauroth: rev e46cb800028c95f9bce575d05268cd10d0913222) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #7982 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7982/)
        HADOOP-12055. Deprecate usage of NativeIO#link. Contributed by Andrew Wang. (cnauroth: rev e46cb800028c95f9bce575d05268cd10d0913222)

        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java
        • hadoop-common-project/hadoop-common/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7982 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7982/ ) HADOOP-12055 . Deprecate usage of NativeIO#link. Contributed by Andrew Wang. (cnauroth: rev e46cb800028c95f9bce575d05268cd10d0913222) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java hadoop-common-project/hadoop-common/CHANGES.txt
        Hide
        cnauroth Chris Nauroth added a comment -

        +1 for the patch. I tested it on Windows. I have committed this to trunk and branch-2. Thank you for the contribution, Andrew.

        I'm sorry for the delay. I ran into some problems with TestDatanodeLayoutUpgrade. Those problems turned out to be unrelated to this patch, so I filed a new patch on HDFS-8554 to fix it.

        I don't have a Windows dev environment.

        All Apache committers have been offered a free MSDN license. The details are documented here.

        https://svn.apache.org/repos/private/committers/donated-licenses/msdn-license-grants.txt

        After getting the free MSDN license, you can download the installation iso for Windows Server 2008 or 2010 and run it in a VirtualBox VM (or your hypervisor of choice). Instructions for setting up a Windows development environment have been in BUILDING.txt for a few years.

        This would prevent situations where patches are blocked from getting committed while waiting for me or any other individual to test. I'm not sure if the MSDN license information is widely known, so I'll send a reminder on the dev mailing lists.

        Show
        cnauroth Chris Nauroth added a comment - +1 for the patch. I tested it on Windows. I have committed this to trunk and branch-2. Thank you for the contribution, Andrew. I'm sorry for the delay. I ran into some problems with TestDatanodeLayoutUpgrade . Those problems turned out to be unrelated to this patch, so I filed a new patch on HDFS-8554 to fix it. I don't have a Windows dev environment. All Apache committers have been offered a free MSDN license. The details are documented here. https://svn.apache.org/repos/private/committers/donated-licenses/msdn-license-grants.txt After getting the free MSDN license, you can download the installation iso for Windows Server 2008 or 2010 and run it in a VirtualBox VM (or your hypervisor of choice). Instructions for setting up a Windows development environment have been in BUILDING.txt for a few years. This would prevent situations where patches are blocked from getting committed while waiting for me or any other individual to test. I'm not sure if the MSDN license information is widely known, so I'll send a reminder on the dev mailing lists.
        Hide
        cnauroth Chris Nauroth added a comment -

        Andrew Wang, please hold off. I'm investigating this now.

        Show
        cnauroth Chris Nauroth added a comment - Andrew Wang , please hold off. I'm investigating this now.
        Hide
        andrew.wang Andrew Wang added a comment -

        I'd like to commit this if there are no further comments. Please note that HADOOP-11483 already switched us over to the JDK7 hardlink for all the other usages and was released in 2.7.0. I also ran some performance tests myself and didn't see a difference between using our JNI implementation vs the JDK7 API.

        Show
        andrew.wang Andrew Wang added a comment - I'd like to commit this if there are no further comments. Please note that HADOOP-11483 already switched us over to the JDK7 hardlink for all the other usages and was released in 2.7.0. I also ran some performance tests myself and didn't see a difference between using our JNI implementation vs the JDK7 API.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 pre-patch 16m 50s Findbugs (version ) appears to be broken on trunk.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 javac 7m 31s There were no new javac warning messages.
        +1 javadoc 9m 39s There were no new javadoc warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 1m 50s There were no new checkstyle issues.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 37s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        +1 findbugs 5m 1s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 common tests 22m 32s Tests passed in hadoop-common.
        +1 hdfs tests 163m 41s Tests passed in hadoop-hdfs.
            229m 39s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12736997/hadoop-12055.001.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 03fb5c6
        hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6895/artifact/patchprocess/testrun_hadoop-common.txt
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6895/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6895/testReport/
        Java 1.7.0_55
        uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6895/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 16m 50s Findbugs (version ) appears to be broken on trunk. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 31s There were no new javac warning messages. +1 javadoc 9m 39s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 50s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 37s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 5m 1s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 22m 32s Tests passed in hadoop-common. +1 hdfs tests 163m 41s Tests passed in hadoop-hdfs.     229m 39s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12736997/hadoop-12055.001.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 03fb5c6 hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6895/artifact/patchprocess/testrun_hadoop-common.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6895/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6895/testReport/ Java 1.7.0_55 uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6895/console This message was automatically generated.
        Hide
        andrew.wang Andrew Wang added a comment -

        This function is only used in the one callsite I changed in the DN upgrade path. All our other hardlink usage already goes through HardLink which I believe does the OS abstraction. I haven't tested it though.

        Do we think this is sufficient? Else, Chris Nauroth is this something you could help test? I don't have a Windows dev environment.

        Show
        andrew.wang Andrew Wang added a comment - This function is only used in the one callsite I changed in the DN upgrade path. All our other hardlink usage already goes through HardLink which I believe does the OS abstraction. I haven't tested it though. Do we think this is sufficient? Else, Chris Nauroth is this something you could help test? I don't have a Windows dev environment.
        Hide
        cnauroth Chris Nauroth added a comment -

        Hi Andrew Wang. Has the change been tested on Windows? From a quick check of OpenJDK source, it looks to me like the Windows implementation of java.nio.file.Files#createLink is equivalent to our own hadoop.dll implementation, but it would be good to test.

        Show
        cnauroth Chris Nauroth added a comment - Hi Andrew Wang . Has the change been tested on Windows? From a quick check of OpenJDK source, it looks to me like the Windows implementation of java.nio.file.Files#createLink is equivalent to our own hadoop.dll implementation, but it would be good to test.
        Hide
        chris.douglas Chris Douglas added a comment -

        Does this change the set of supported platforms?

        Show
        chris.douglas Chris Douglas added a comment - Does this change the set of supported platforms?
        Hide
        cmccabe Colin P. McCabe added a comment -

        +1 pending jenkins

        Show
        cmccabe Colin P. McCabe added a comment - +1 pending jenkins
        Hide
        andrew.wang Andrew Wang added a comment -

        Patch attached.

        Show
        andrew.wang Andrew Wang added a comment - Patch attached.

          People

          • Assignee:
            andrew.wang Andrew Wang
            Reporter:
            andrew.wang Andrew Wang
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development