Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-14434

Use MoveFileEx to allow renaming a file when the destination exists

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.1, 2.7.3, 3.0.0-alpha1, 3.0.0-alpha2
    • Fix Version/s: 2.7.4, 3.0.0-alpha4
    • Component/s: native
    • Labels:
    • Hadoop Flags:
      Reviewed

      Description

      The NativeIO.c#renameTo0 currently uses MoveFile Windows system call, which fails when renaming a file to a destination that already exists.

      This makes the TestRollingUpgrade.testRollback test fail on Windows, as during that execution, a DataNode tries to rename block's meta file to a destination that exists.

      The proposal is to change to using MoveFileEx Windows call, and passing in MOVEFILE_REPLACE_EXISTING flag to force the renaming.

      1. HADOOP-14434.002.patch
        2 kB
        Chris Douglas
      2. HDFS-11713.001.patch
        5 kB
        Lukas Majercak

        Issue Links

          Activity

          Hide
          elgoiri Íñigo Goiri added a comment -

          The change from rename() to MoveFile() was done by Chris Nauroth in HADOOP-9437 as proposed by Ivan Mitic.
          Any comments here?
          I think this matches the semantics from Linux.

          Show
          elgoiri Íñigo Goiri added a comment - The change from rename() to MoveFile() was done by Chris Nauroth in HADOOP-9437 as proposed by Ivan Mitic . Any comments here? I think this matches the semantics from Linux.
          Hide
          lukmajercak Lukas Majercak added a comment -

          Exception in TestRollingUpgrade.testRollback :

          from E:\workspace\Hadoop\hadoop-hdfs-project\hadoop-hdfs\target\test\data\dfs\data\data1\current\BP-107441638-10.123.116.86-1493168359567\tmp\subdir0\subdir0\blk_1073741826_1002.meta to E:\workspace\Hadoop\hadoop-hdfs-project\hadoop-hdfs\target\test\data\dfs\data\data1\current\BP-107441638-10.123.116.86-1493168359567\current\finalized\subdir0\subdir0\blk_1073741826_1002.meta
              at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.moveBlockFiles(FsDatasetImpl.java:882)
              at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.addBlock(BlockPoolSlice.java:286)
              at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.addFinalizedBlock(FsVolumeImpl.java:799)
              at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.finalizeReplica(FsDatasetImpl.java:1621)
              at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.updateReplicaUnderRecovery(FsDatasetImpl.java:2526)
              at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.updateReplicaUnderRecovery(FsDatasetImpl.java:2451)
              at org.apache.hadoop.hdfs.server.datanode.DataNode.updateReplicaUnderRecovery(DataNode.java:2633)
              at org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$BlockRecord.updateReplicaUnderRecovery(BlockRecoveryWorker.java:87)
              at org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$BlockRecord.access$500(BlockRecoveryWorker.java:70)
              at org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$RecoveryTaskContiguous.syncBlock(BlockRecoveryWorker.java:241)
              at org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$RecoveryTaskContiguous.recover(BlockRecoveryWorker.java:153)
              at org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$1.run(BlockRecoveryWorker.java:541)
              at java.lang.Thread.run(Thread.java:745)
          Caused by: 183: Cannot create a file when that file already exists.
              at org.apache.hadoop.io.nativeio.NativeIO.renameTo0(Native Method)
              at org.apache.hadoop.io.nativeio.NativeIO.renameTo(NativeIO.java:902)
              at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.moveBlockFiles(FsDatasetImpl.java:880)
              ... 12 more
          
          Show
          lukmajercak Lukas Majercak added a comment - Exception in TestRollingUpgrade.testRollback : from E:\workspace\Hadoop\hadoop-hdfs-project\hadoop-hdfs\target\test\data\dfs\data\data1\current\BP-107441638-10.123.116.86-1493168359567\tmp\subdir0\subdir0\blk_1073741826_1002.meta to E:\workspace\Hadoop\hadoop-hdfs-project\hadoop-hdfs\target\test\data\dfs\data\data1\current\BP-107441638-10.123.116.86-1493168359567\current\finalized\subdir0\subdir0\blk_1073741826_1002.meta at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.moveBlockFiles(FsDatasetImpl.java:882) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.addBlock(BlockPoolSlice.java:286) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.addFinalizedBlock(FsVolumeImpl.java:799) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.finalizeReplica(FsDatasetImpl.java:1621) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.updateReplicaUnderRecovery(FsDatasetImpl.java:2526) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.updateReplicaUnderRecovery(FsDatasetImpl.java:2451) at org.apache.hadoop.hdfs.server.datanode.DataNode.updateReplicaUnderRecovery(DataNode.java:2633) at org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$BlockRecord.updateReplicaUnderRecovery(BlockRecoveryWorker.java:87) at org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$BlockRecord.access$500(BlockRecoveryWorker.java:70) at org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$RecoveryTaskContiguous.syncBlock(BlockRecoveryWorker.java:241) at org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$RecoveryTaskContiguous.recover(BlockRecoveryWorker.java:153) at org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$1.run(BlockRecoveryWorker.java:541) at java.lang. Thread .run( Thread .java:745) Caused by: 183: Cannot create a file when that file already exists. at org.apache.hadoop.io.nativeio.NativeIO.renameTo0(Native Method) at org.apache.hadoop.io.nativeio.NativeIO.renameTo(NativeIO.java:902) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.moveBlockFiles(FsDatasetImpl.java:880) ... 12 more
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 20s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 14s Maven dependency ordering for branch
          +1 mvninstall 12m 34s trunk passed
          +1 compile 15m 6s trunk passed
          +1 checkstyle 1m 47s trunk passed
          +1 mvnsite 2m 3s trunk passed
          +1 mvneclipse 0m 45s trunk passed
          -1 findbugs 1m 17s hadoop-common-project/hadoop-common in trunk has 17 extant Findbugs warnings.
          -1 findbugs 1m 34s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings.
          +1 javadoc 1m 37s trunk passed
          0 mvndep 0m 13s Maven dependency ordering for patch
          +1 mvninstall 1m 19s the patch passed
          +1 compile 13m 3s the patch passed
          +1 cc 13m 3s the patch passed
          +1 javac 13m 3s the patch passed
          +1 checkstyle 1m 49s root: The patch generated 0 new + 69 unchanged - 4 fixed = 69 total (was 73)
          +1 mvnsite 2m 0s the patch passed
          +1 mvneclipse 0m 46s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 15s the patch passed
          +1 javadoc 1m 42s the patch passed
          +1 unit 7m 9s hadoop-common in the patch passed.
          -1 unit 64m 15s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 41s The patch does not generate ASF License warnings.
          134m 53s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
            hadoop.hdfs.web.TestWebHdfsTimeouts



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ac17dc
          JIRA Issue HDFS-11713
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12865427/HDFS-11713.001.patch
          Optional Tests asflicense compile cc mvnsite javac unit javadoc mvninstall findbugs checkstyle
          uname Linux cbbd8cdf3653 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 9460721
          Default Java 1.8.0_121
          findbugs v3.1.0-RC1
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19222/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19222/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/19222/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19222/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19222/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 14s Maven dependency ordering for branch +1 mvninstall 12m 34s trunk passed +1 compile 15m 6s trunk passed +1 checkstyle 1m 47s trunk passed +1 mvnsite 2m 3s trunk passed +1 mvneclipse 0m 45s trunk passed -1 findbugs 1m 17s hadoop-common-project/hadoop-common in trunk has 17 extant Findbugs warnings. -1 findbugs 1m 34s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings. +1 javadoc 1m 37s trunk passed 0 mvndep 0m 13s Maven dependency ordering for patch +1 mvninstall 1m 19s the patch passed +1 compile 13m 3s the patch passed +1 cc 13m 3s the patch passed +1 javac 13m 3s the patch passed +1 checkstyle 1m 49s root: The patch generated 0 new + 69 unchanged - 4 fixed = 69 total (was 73) +1 mvnsite 2m 0s the patch passed +1 mvneclipse 0m 46s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 15s the patch passed +1 javadoc 1m 42s the patch passed +1 unit 7m 9s hadoop-common in the patch passed. -1 unit 64m 15s hadoop-hdfs in the patch failed. +1 asflicense 0m 41s The patch does not generate ASF License warnings. 134m 53s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.web.TestWebHdfsTimeouts Subsystem Report/Notes Docker Image:yetus/hadoop:0ac17dc JIRA Issue HDFS-11713 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12865427/HDFS-11713.001.patch Optional Tests asflicense compile cc mvnsite javac unit javadoc mvninstall findbugs checkstyle uname Linux cbbd8cdf3653 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 9460721 Default Java 1.8.0_121 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19222/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19222/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/19222/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19222/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19222/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          lukmajercak Lukas Majercak added a comment -

          None of the findbugs/unit test warnings seem to be related to the change.

          Show
          lukmajercak Lukas Majercak added a comment - None of the findbugs/unit test warnings seem to be related to the change.
          Hide
          elgoiri Íñigo Goiri added a comment -

          Anybody available for review? Arpit Agarwal, Chuan Liu, Suresh Srinivas, Chris Nauroth?

          Show
          elgoiri Íñigo Goiri added a comment - Anybody available for review? Arpit Agarwal , Chuan Liu , Suresh Srinivas , Chris Nauroth ?
          Hide
          chuanliu Chuan Liu added a comment -

          +1 LGTM

          Show
          chuanliu Chuan Liu added a comment - +1 LGTM
          Hide
          chris.douglas Chris Douglas added a comment -

          v002

          • Reverted spurious change to TestRollingUpgrade
          • TestNativeIO deletes TEST_DIR in a @Before method, so the try/finally change is redundant

          From my read of the docs this change looks correct, but I haven't tested it. Lukas Majercak, does this look OK to you?

          Show
          chris.douglas Chris Douglas added a comment - v002 Reverted spurious change to TestRollingUpgrade TestNativeIO deletes TEST_DIR in a @Before method, so the try/finally change is redundant From my read of the docs this change looks correct, but I haven't tested it. Lukas Majercak , does this look OK to you?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 12m 42s trunk passed
          +1 compile 12m 14s trunk passed
          +1 checkstyle 0m 29s trunk passed
          +1 mvnsite 0m 59s trunk passed
          +1 mvneclipse 0m 17s trunk passed
          -1 findbugs 1m 22s hadoop-common-project/hadoop-common in trunk has 19 extant Findbugs warnings.
          +1 javadoc 0m 46s trunk passed
          +1 mvninstall 0m 42s the patch passed
          +1 compile 12m 56s the patch passed
          +1 cc 12m 56s the patch passed
          +1 javac 12m 56s the patch passed
          +1 checkstyle 0m 31s the patch passed
          +1 mvnsite 0m 56s the patch passed
          +1 mvneclipse 0m 17s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 31s the patch passed
          +1 javadoc 0m 46s the patch passed
          +1 unit 7m 30s hadoop-common in the patch passed.
          +1 asflicense 0m 29s The patch does not generate ASF License warnings.
          56m 25s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue HADOOP-14434
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12868610/HADOOP-14434.002.patch
          Optional Tests asflicense compile cc mvnsite javac unit javadoc mvninstall findbugs checkstyle
          uname Linux bc0dd014c761 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / eb7791b
          Default Java 1.8.0_131
          findbugs v3.1.0-RC1
          findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/12349/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12349/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12349/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 12m 42s trunk passed +1 compile 12m 14s trunk passed +1 checkstyle 0m 29s trunk passed +1 mvnsite 0m 59s trunk passed +1 mvneclipse 0m 17s trunk passed -1 findbugs 1m 22s hadoop-common-project/hadoop-common in trunk has 19 extant Findbugs warnings. +1 javadoc 0m 46s trunk passed +1 mvninstall 0m 42s the patch passed +1 compile 12m 56s the patch passed +1 cc 12m 56s the patch passed +1 javac 12m 56s the patch passed +1 checkstyle 0m 31s the patch passed +1 mvnsite 0m 56s the patch passed +1 mvneclipse 0m 17s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 31s the patch passed +1 javadoc 0m 46s the patch passed +1 unit 7m 30s hadoop-common in the patch passed. +1 asflicense 0m 29s The patch does not generate ASF License warnings. 56m 25s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HADOOP-14434 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12868610/HADOOP-14434.002.patch Optional Tests asflicense compile cc mvnsite javac unit javadoc mvninstall findbugs checkstyle uname Linux bc0dd014c761 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / eb7791b Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/12349/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12349/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12349/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          chris.douglas Chris Douglas added a comment -

          Sorry Lukas Majercak, I missed your comment

          The change to TestRollingUpgrade looked like leftover debugging, since it just added a timeout to the test. What's going on, there?

          Show
          chris.douglas Chris Douglas added a comment - Sorry Lukas Majercak , I missed your comment The change to TestRollingUpgrade looked like leftover debugging, since it just added a timeout to the test. What's going on, there?
          Hide
          chris.douglas Chris Douglas added a comment -

          I should read the description before commenting.

          +1 I committed this. Thanks Lukas.

          Show
          chris.douglas Chris Douglas added a comment - I should read the description before commenting. +1 I committed this. Thanks Lukas.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11746 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11746/)
          HADOOP-14434. Use MoveFileEx to allow renaming a file when the (cdouglas: rev ef9e536a7137d209985d25a4769712c2db2c52bf)

          • (edit) hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/nativeio/NativeIO.c
          • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/nativeio/TestNativeIO.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11746 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11746/ ) HADOOP-14434 . Use MoveFileEx to allow renaming a file when the (cdouglas: rev ef9e536a7137d209985d25a4769712c2db2c52bf) (edit) hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/nativeio/NativeIO.c (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/nativeio/TestNativeIO.java

            People

            • Assignee:
              lukmajercak Lukas Majercak
              Reporter:
              lukmajercak Lukas Majercak
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development