Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-2759

Pre-allocate HDFS edit log files after writing version number

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.24.0
    • Fix Version/s: 2.0.2-alpha
    • Component/s: ha, namenode
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      In HDFS-2709 it was discovered that there's a potential race wherein edits log files are pre-allocated before the version number is written into the header of the file. This can cause the NameNode to read an invalid HDFS layout version, and hence fail to read the edit log file. We should write the header, then pre-allocate the rest of the file after this point.

      1. HDFS-2759.patch
        3 kB
        Aaron T. Myers
      2. HDFS-2759.patch
        3 kB
        Aaron T. Myers
      3. HDFS-2759.patch
        0.9 kB
        Aaron T. Myers

        Issue Links

          Activity

          Aaron T. Myers created issue -
          Aaron T. Myers made changes -
          Field Original Value New Value
          Component/s ha [ 12316609 ]
          Aaron T. Myers made changes -
          Description In HDFS-2709 it was discovered that there's a potential race wherein edits log files are pre-allocated before the version number is written into the header of the file. This can cause the standby to read an invalid version. We should write the header, then pre-allocate the rest of the file after this point. In HDFS-2709 it was discovered that there's a potential race wherein edits log files are pre-allocated before the version number is written into the header of the file. This can cause the NameNode to read an invalid HDFS layout version, and hence fail to read the edit log file. We should write the header, then pre-allocate the rest of the file after this point.
          Aaron T. Myers made changes -
          Parent HDFS-1623 [ 12498318 ]
          Issue Type Sub-task [ 7 ] Bug [ 1 ]
          Hide
          Aaron T. Myers added a comment -

          Moved this to a top-level issue, since it's not specific to HA.

          Show
          Aaron T. Myers added a comment - Moved this to a top-level issue, since it's not specific to HA.
          Hide
          Aaron T. Myers added a comment -

          Here's a patch which addresses the issue.

          Show
          Aaron T. Myers added a comment - Here's a patch which addresses the issue.
          Aaron T. Myers made changes -
          Attachment HDFS-2759.patch [ 12511911 ]
          Hide
          Aaron T. Myers added a comment -

          Sorry, I should have mentioned what I did for testing. No tests are included since writing a test seems like it would require adding a test hook in the EditLogFileOutputStream#flushAndSync method, which seems undesirable since that method is highly performance-sensitive. I verified that the bug exists by adding a "System.exit(0)" after pre-allocation, but before calling FileChannel#force. This indeed resulted in a file full of FF, which if an NN were to try to read would be interpreted as an invalid header version number. By moving the pre-allocation after the call to FileChannel#force, we guarantee that the valid data will hit the edit log before writing FFs for the purpose of pre-allocation.

          In the course of preparing and testing this patch, I think I've discovered another potential issue, though. Note that in the call to FileChannel#force, we pass "false" which indicates that FileChannel#force does not need to wait for metadata of the file to be synced to disk, only the data. I'm not sure of the precise semantics of not syncing metadata. In particular, is the file length included? If not, I think this has the potential to cause some edits to not actually be read back from disk after a crash.

          The explanation, per the comment, is that syncing metadata is unnecessary because of pre-allocation. I don't think that's reasonable, though, since EditLogFileOutputStream#preallocate doesn't call sync itself, which means that the file length might never get updated upon returning from EditLogFileOutputStream#flushAndSync.

          Do people agree this is a potential problem? If so, I can either fix it here, or file a new JIRA.

          Show
          Aaron T. Myers added a comment - Sorry, I should have mentioned what I did for testing. No tests are included since writing a test seems like it would require adding a test hook in the EditLogFileOutputStream#flushAndSync method, which seems undesirable since that method is highly performance-sensitive. I verified that the bug exists by adding a "System.exit(0)" after pre-allocation, but before calling FileChannel#force. This indeed resulted in a file full of FF, which if an NN were to try to read would be interpreted as an invalid header version number. By moving the pre-allocation after the call to FileChannel#force, we guarantee that the valid data will hit the edit log before writing FFs for the purpose of pre-allocation. In the course of preparing and testing this patch, I think I've discovered another potential issue, though. Note that in the call to FileChannel#force, we pass "false" which indicates that FileChannel#force does not need to wait for metadata of the file to be synced to disk, only the data. I'm not sure of the precise semantics of not syncing metadata. In particular, is the file length included? If not, I think this has the potential to cause some edits to not actually be read back from disk after a crash. The explanation, per the comment, is that syncing metadata is unnecessary because of pre-allocation. I don't think that's reasonable, though, since EditLogFileOutputStream#preallocate doesn't call sync itself, which means that the file length might never get updated upon returning from EditLogFileOutputStream#flushAndSync. Do people agree this is a potential problem? If so, I can either fix it here, or file a new JIRA.
          Aaron T. Myers made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Affects Version/s 0.24.0 [ 12317653 ]
          Affects Version/s HA branch (HDFS-1623) [ 12317568 ]
          Target Version/s 0.24.0 [ 12317653 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12511911/HDFS-2759.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 javadoc. The javadoc tool appears to have generated 21 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hdfs.server.namenode.TestEditLogFileOutputStream

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1812//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1812//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1812//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12511911/HDFS-2759.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated 21 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.server.namenode.TestEditLogFileOutputStream +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1812//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1812//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1812//console This message is automatically generated.
          Hide
          Aaron T. Myers added a comment -

          A few asserts in tests needed to be updated since a freshly pre-allocated log file is now 4 bytes longer, since pre-allocation happens after writing the 4-byte header.

          Show
          Aaron T. Myers added a comment - A few asserts in tests needed to be updated since a freshly pre-allocated log file is now 4 bytes longer, since pre-allocation happens after writing the 4-byte header.
          Aaron T. Myers made changes -
          Attachment HDFS-2759.patch [ 12511956 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12511956/HDFS-2759.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          -1 javadoc. The javadoc tool appears to have generated 21 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1814//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1814//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1814//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12511956/HDFS-2759.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 21 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1814//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1814//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1814//console This message is automatically generated.
          Hide
          Aaron T. Myers added a comment -

          The javadoc and findbugs warnings are unrelated to this patch.

          Show
          Aaron T. Myers added a comment - The javadoc and findbugs warnings are unrelated to this patch.
          Hide
          Todd Lipcon added a comment -

          This seems reasonable. I remember you ran a benchmark at some point to check for edit log throughput - if you have that around still would you mind re-running to make sure this doesn't cause any unforeseen regression?

          Show
          Todd Lipcon added a comment - This seems reasonable. I remember you ran a benchmark at some point to check for edit log throughput - if you have that around still would you mind re-running to make sure this doesn't cause any unforeseen regression?
          Hide
          Todd Lipcon added a comment -

          The explanation, per the comment, is that syncing metadata is unnecessary because of pre-allocation. I don't think that's reasonable, though, since EditLogFileOutputStream#preallocate doesn't call sync itself, which means that the file length might never get updated upon returning from EditLogFileOutputStream#flushAndSync.

          This isn't quite true - the difference between force(true) and force(false) is that force(false) calls fdatasync(). The man page says:

          fdatasync() is similar to fsync(), but does not flush modified metadata
          unless that metadata is needed in order to allow a subsequent data
          retrieval to be correctly handled. For example, changes to st_atime or
          st_mtime (respectively, time of last access and time of last modifica‐
          tion; see stat(2)) do not require flushing because they are not neces‐
          sary for a subsequent data read to be handled correctly. On the other
          hand, a change to the file size (st_size, as made by say ftruncate(2)),
          would require a metadata flush.

          (emphasis mine)

          So I think we're OK here.

          Show
          Todd Lipcon added a comment - The explanation, per the comment, is that syncing metadata is unnecessary because of pre-allocation. I don't think that's reasonable, though, since EditLogFileOutputStream#preallocate doesn't call sync itself, which means that the file length might never get updated upon returning from EditLogFileOutputStream#flushAndSync. This isn't quite true - the difference between force(true) and force(false) is that force(false) calls fdatasync() . The man page says: fdatasync() is similar to fsync(), but does not flush modified metadata unless that metadata is needed in order to allow a subsequent data retrieval to be correctly handled . For example, changes to st_atime or st_mtime (respectively, time of last access and time of last modifica‐ tion; see stat(2)) do not require flushing because they are not neces‐ sary for a subsequent data read to be handled correctly. On the other hand, a change to the file size (st_size, as made by say ftruncate(2)), would require a metadata flush . (emphasis mine) So I think we're OK here.
          Hide
          Aaron T. Myers added a comment -

          So I think we're OK here.

          Awesome. I agree with your analysis.

          I remember you ran a benchmark at some point to check for edit log throughput - if you have that around still would you mind re-running to make sure this doesn't cause any unforeseen regression?

          I bet I can dig that up and will be happy to run it again with this patch.

          Show
          Aaron T. Myers added a comment - So I think we're OK here. Awesome. I agree with your analysis. I remember you ran a benchmark at some point to check for edit log throughput - if you have that around still would you mind re-running to make sure this doesn't cause any unforeseen regression? I bet I can dig that up and will be happy to run it again with this patch.
          Hide
          Aaron T. Myers added a comment -

          I remember you ran a benchmark at some point to check for edit log throughput - if you have that around still would you mind re-running to make sure this doesn't cause any unforeseen regression?

          I just re-ran the benchmark, and saw no performance regressions. In fact, pre-allocation ops and the slowest 0.01% of ops seem to have actually sped up a little bit, since we can now return from the flushAndSync call without waiting for the pre-allocation data to be synced to disk.

          Here's a (very slightly) updated patch which is identical to the last, except that it also fixes up an incorrect comment justifying why we don't need to force the syncing of metadata when logging an op.

          Show
          Aaron T. Myers added a comment - I remember you ran a benchmark at some point to check for edit log throughput - if you have that around still would you mind re-running to make sure this doesn't cause any unforeseen regression? I just re-ran the benchmark, and saw no performance regressions. In fact, pre-allocation ops and the slowest 0.01% of ops seem to have actually sped up a little bit, since we can now return from the flushAndSync call without waiting for the pre-allocation data to be synced to disk. Here's a (very slightly) updated patch which is identical to the last, except that it also fixes up an incorrect comment justifying why we don't need to force the syncing of metadata when logging an op.
          Aaron T. Myers made changes -
          Attachment HDFS-2759.patch [ 12512335 ]
          Hide
          Aaron T. Myers added a comment -

          I just went ahead and committed this to trunk since the difference between the reviewed patch and the latest one is just a small comment change.

          Thanks a lot for the review, Todd.

          Show
          Aaron T. Myers added a comment - I just went ahead and committed this to trunk since the difference between the reviewed patch and the latest one is just a small comment change. Thanks a lot for the review, Todd.
          Aaron T. Myers made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Fix Version/s 0.24.0 [ 12317653 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #1613 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1613/)
          HDFS-2759. Pre-allocate HDFS edit log files after writing version number. Contributed by Aaron T. Myers.

          atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1237153
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogFileOutputStream.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1613 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1613/ ) HDFS-2759 . Pre-allocate HDFS edit log files after writing version number. Contributed by Aaron T. Myers. atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1237153 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogFileOutputStream.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #1685 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1685/)
          HDFS-2759. Pre-allocate HDFS edit log files after writing version number. Contributed by Aaron T. Myers.

          atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1237153
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogFileOutputStream.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1685 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1685/ ) HDFS-2759 . Pre-allocate HDFS edit log files after writing version number. Contributed by Aaron T. Myers. atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1237153 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogFileOutputStream.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #1629 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1629/)
          HDFS-2759. Pre-allocate HDFS edit log files after writing version number. Contributed by Aaron T. Myers.

          atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1237153
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogFileOutputStream.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1629 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1629/ ) HDFS-2759 . Pre-allocate HDFS edit log files after writing version number. Contributed by Aaron T. Myers. atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1237153 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogFileOutputStream.java
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12512335/HDFS-2759.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hdfs.TestFSInputChecker

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1823//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1823//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1823//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12512335/HDFS-2759.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.TestFSInputChecker +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1823//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1823//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1823//console This message is automatically generated.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #940 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/940/)
          HDFS-2759. Pre-allocate HDFS edit log files after writing version number. Contributed by Aaron T. Myers.

          atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1237153
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogFileOutputStream.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #940 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/940/ ) HDFS-2759 . Pre-allocate HDFS edit log files after writing version number. Contributed by Aaron T. Myers. atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1237153 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogFileOutputStream.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #973 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/973/)
          HDFS-2759. Pre-allocate HDFS edit log files after writing version number. Contributed by Aaron T. Myers.

          atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1237153
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogFileOutputStream.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #973 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/973/ ) HDFS-2759 . Pre-allocate HDFS edit log files after writing version number. Contributed by Aaron T. Myers. atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1237153 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogFileOutputStream.java
          Arun C Murthy made changes -
          Fix Version/s 3.0.0 [ 12320356 ]
          Fix Version/s 0.24.0 [ 12317653 ]
          Hide
          Todd Lipcon added a comment -

          I'd like to backport this to branch-2. Any objections?

          Show
          Todd Lipcon added a comment - I'd like to backport this to branch-2. Any objections?
          Hide
          Aaron T. Myers added a comment -

          I'd like to backport this to branch-2. Any objections?

          +1, this should definitely be in branch-2.

          Show
          Aaron T. Myers added a comment - I'd like to backport this to branch-2. Any objections? +1, this should definitely be in branch-2.
          Hide
          Todd Lipcon added a comment -

          Committed to branch-2 (the patch applied as is). Thanks.

          Show
          Todd Lipcon added a comment - Committed to branch-2 (the patch applied as is). Thanks.
          Todd Lipcon made changes -
          Fix Version/s 2.0.0 [ 12320353 ]
          Fix Version/s 3.0.0 [ 12320356 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2317 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2317/)
          Move HDFS-2759 to branch-2 section of CHANGES.txt. (Revision 1338490)

          Result = SUCCESS
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1338490
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2317 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2317/ ) Move HDFS-2759 to branch-2 section of CHANGES.txt. (Revision 1338490) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1338490 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2243 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2243/)
          Move HDFS-2759 to branch-2 section of CHANGES.txt. (Revision 1338490)

          Result = SUCCESS
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1338490
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2243 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2243/ ) Move HDFS-2759 to branch-2 section of CHANGES.txt. (Revision 1338490) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1338490 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2260 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2260/)
          Move HDFS-2759 to branch-2 section of CHANGES.txt. (Revision 1338490)

          Result = FAILURE
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1338490
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2260 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2260/ ) Move HDFS-2759 to branch-2 section of CHANGES.txt. (Revision 1338490) Result = FAILURE todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1338490 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Todd Lipcon made changes -
          Fix Version/s 2.0.1 [ 12321440 ]
          Fix Version/s 2.0.0 [ 12320353 ]
          Jeff Hammerbacher made changes -
          Link This issue is related to HDFS-2709 [ HDFS-2709 ]
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Arun C Murthy made changes -
          Fix Version/s 2.0.2-alpha [ 12322472 ]
          Fix Version/s 2.1.0-alpha [ 12321440 ]

            People

            • Assignee:
              Aaron T. Myers
              Reporter:
              Aaron T. Myers
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development