Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-3531

EditLogFileOutputStream#preallocate should check for incomplete writes

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: 2.0.2-alpha
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      We need to check for short writes when using WritableByteChannel#write and related methods in EditLogFileOutputStream#preallocate so we ensure we've preallocated sufficient space.

      1. HADOOP-8491.001.patch
        5 kB
        Colin Patrick McCabe
      2. HADOOP-8491.002.patch
        5 kB
        Colin Patrick McCabe
      3. hadoop-8491.txt
        5 kB
        Eli Collins
      4. hadoop-8491.txt
        5 kB
        Eli Collins

        Issue Links

          Activity

          Colin Patrick McCabe created issue -
          Hide
          Colin Patrick McCabe added a comment -

          I re-read the documents, and I'm no longer so sure about this one.

          The documentation for WritableByteChannel#write says:

          Writes a sequence of bytes to this channel from the given buffer.

          An attempt is made to write up to r bytes to the channel, where r is the number of bytes remaining in the buffer, that is, src.remaining(), at the moment this method is invoked.

          Suppose that a byte sequence of length n is written, where 0 <= n <= r. This byte sequence will be transferred from the buffer starting at index p, where p is the buffer's position at the moment this method is invoked; the index of the last byte written will be p + n - 1. Upon return the buffer's position will be equal to p + n; its limit will not have changed.

          Unless otherwise specified, a write operation will return only after writing all of the r requested bytes. Some types of channels, depending upon their state, may write only some of the bytes or possibly none at all. A socket channel in non-blocking mode, for example, cannot write any more bytes than are free in the socket's output buffer.

          This method may be invoked at any time. If another thread has already initiated a write operation upon this channel, however, then an invocation of this method will block until the first operation is complete.

          Notice the key sentence here: Unless otherwise specified, a write operation will return only after writing all of the r requested bytes.

          Then the documentation for FileChannel#write says:

          Writes a sequence of bytes to this channel from the given buffer.

          Bytes are written starting at this channel's current file position unless the channel is in append mode, in which case the position is first advanced to the end of the file. The file is grown, if necessary, to accommodate the written bytes, and then the file position is updated with the number of bytes actually written. Otherwise this method behaves exactly as specified by the WritableByteChannel interface.

          I don't see it "otherwise specified" here, so in essence, write should be equivalent to writeFully for FileChannel (but not for certain other subclasses of WritableByteChannel.) Or am I missing something?

          Show
          Colin Patrick McCabe added a comment - I re-read the documents, and I'm no longer so sure about this one. The documentation for WritableByteChannel#write says: Writes a sequence of bytes to this channel from the given buffer. An attempt is made to write up to r bytes to the channel, where r is the number of bytes remaining in the buffer, that is, src.remaining(), at the moment this method is invoked. Suppose that a byte sequence of length n is written, where 0 <= n <= r. This byte sequence will be transferred from the buffer starting at index p, where p is the buffer's position at the moment this method is invoked; the index of the last byte written will be p + n - 1. Upon return the buffer's position will be equal to p + n; its limit will not have changed. Unless otherwise specified, a write operation will return only after writing all of the r requested bytes. Some types of channels, depending upon their state, may write only some of the bytes or possibly none at all. A socket channel in non-blocking mode, for example, cannot write any more bytes than are free in the socket's output buffer. This method may be invoked at any time. If another thread has already initiated a write operation upon this channel, however, then an invocation of this method will block until the first operation is complete. Notice the key sentence here: Unless otherwise specified, a write operation will return only after writing all of the r requested bytes. Then the documentation for FileChannel#write says: Writes a sequence of bytes to this channel from the given buffer. Bytes are written starting at this channel's current file position unless the channel is in append mode, in which case the position is first advanced to the end of the file. The file is grown, if necessary, to accommodate the written bytes, and then the file position is updated with the number of bytes actually written. Otherwise this method behaves exactly as specified by the WritableByteChannel interface. I don't see it "otherwise specified" here, so in essence, write should be equivalent to writeFully for FileChannel (but not for certain other subclasses of WritableByteChannel.) Or am I missing something?
          Colin Patrick McCabe made changes -
          Field Original Value New Value
          Link This issue relates to HDFS-3479 [ HDFS-3479 ]
          Colin Patrick McCabe made changes -
          Link This issue relates to HDFS-3510 [ HDFS-3510 ]
          Hide
          Colin Patrick McCabe added a comment -

          I just took a quick look at the source code to OpenJDK 6. It's possible that I'm missing something in the JDK code, but I didn't see anything in there that would guard against short writes. I think we're better off just implementing writeFully and using that.

          Show
          Colin Patrick McCabe added a comment - I just took a quick look at the source code to OpenJDK 6. It's possible that I'm missing something in the JDK code, but I didn't see anything in there that would guard against short writes. I think we're better off just implementing writeFully and using that.
          Hide
          Colin Patrick McCabe added a comment -

          here's a patch for trunk that adds a writeFully method.

          Show
          Colin Patrick McCabe added a comment - here's a patch for trunk that adds a writeFully method.
          Colin Patrick McCabe made changes -
          Attachment HADOOP-8491.001.patch [ 12531435 ]
          Colin Patrick McCabe made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Eli Collins added a comment -

          I agree w your analysis that this is necessary.

          Nit: The 1st writeFully method javadoc says FileChannel and uses "fc" but the arg is a WritableByteChannel.

          +1 otherwise, looks excellent

          Show
          Eli Collins added a comment - I agree w your analysis that this is necessary. Nit: The 1st writeFully method javadoc says FileChannel and uses "fc" but the arg is a WritableByteChannel. +1 otherwise, looks excellent
          Eli Collins made changes -
          Affects Version/s 2.0.0-alpha [ 12320352 ]
          Target Version/s 2.0.1-alpha [ 12321441 ]
          Hide
          Colin Patrick McCabe added a comment -
          • fix javadoc nit
          Show
          Colin Patrick McCabe added a comment - fix javadoc nit
          Colin Patrick McCabe made changes -
          Attachment HADOOP-8491.002.patch [ 12531694 ]
          Hide
          Colin Patrick McCabe added a comment -
          • update Text unit test to handle new handling of 'maximum' (formerly, we were not allowing strings of the 'maximum' length, but now we do.)

          This is less confusing because it means that 'max' is a true maximum, not just an upper limit that can't be reached.

          Show
          Colin Patrick McCabe added a comment - update Text unit test to handle new handling of 'maximum' (formerly, we were not allowing strings of the 'maximum' length, but now we do.) This is less confusing because it means that 'max' is a true maximum, not just an upper limit that can't be reached.
          Colin Patrick McCabe made changes -
          Attachment HADOOP-8491.003.patch [ 12531697 ]
          Colin Patrick McCabe made changes -
          Attachment HADOOP-8491.003.patch [ 12531697 ]
          Hide
          Colin Patrick McCabe added a comment -

          please ignore the proceeding comment, it relates to a different JIRA.

          Show
          Colin Patrick McCabe added a comment - please ignore the proceeding comment, it relates to a different JIRA.
          Hide
          Eli Collins added a comment -

          +1 looks great

          Show
          Eli Collins added a comment - +1 looks great
          Eli Collins made changes -
          Hadoop Flags Reviewed [ 10343 ]
          Summary We need to check for short writes when using FileChannel#write and related methods Check for short writes when using FileChannel#write and related methods
          Hide
          Eli Collins added a comment -

          I've committed this and merged to branch-2. Thanks Colin.

          Show
          Eli Collins added a comment - I've committed this and merged to branch-2. Thanks Colin.
          Eli Collins made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Target Version/s 2.0.1-alpha [ 12321441 ]
          Fix Version/s 2.0.1-alpha [ 12321441 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2363 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2363/)
          HADOOP-8491. Check for short writes when using FileChannel#write and related methods. Contributed by Colin Patrick McCabe (Revision 1349019)

          Result = SUCCESS
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349019
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/IOUtils.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtils.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2363 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2363/ ) HADOOP-8491 . Check for short writes when using FileChannel#write and related methods. Contributed by Colin Patrick McCabe (Revision 1349019) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349019 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/IOUtils.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtils.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12531694/HADOOP-8491.002.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1106//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12531694/HADOOP-8491.002.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1106//console This message is automatically generated.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2342 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2342/)
          HADOOP-8491. Check for short writes when using FileChannel#write and related methods. Contributed by Colin Patrick McCabe (Revision 1349019)

          Result = SUCCESS
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349019
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/IOUtils.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtils.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2342 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2342/ ) HADOOP-8491 . Check for short writes when using FileChannel#write and related methods. Contributed by Colin Patrick McCabe (Revision 1349019) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349019 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/IOUtils.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtils.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2415 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2415/)
          HADOOP-8491. Check for short writes when using FileChannel#write and related methods. Contributed by Colin Patrick McCabe (Revision 1349019)

          Result = SUCCESS
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349019
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/IOUtils.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtils.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2415 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2415/ ) HADOOP-8491 . Check for short writes when using FileChannel#write and related methods. Contributed by Colin Patrick McCabe (Revision 1349019) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349019 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/IOUtils.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtils.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java
          Hide
          Tsz Wo Nicholas Sze added a comment -

          [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/IOUtils.java:260: warning - @param argument "offset" is not a parameter name.

          Why the patch was committed without Jerkins' +1?

          Show
          Tsz Wo Nicholas Sze added a comment - [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/IOUtils.java:260: warning - @param argument "offset" is not a parameter name. Why the patch was committed without Jerkins' +1?
          Tsz Wo Nicholas Sze made changes -
          Resolution Fixed [ 1 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          Hide
          Eli Collins added a comment -

          Good catch Nicholas. Reverted, here's an updated patch which fixes the javadoc error.

          Show
          Eli Collins added a comment - Good catch Nicholas. Reverted, here's an updated patch which fixes the javadoc error.
          Eli Collins made changes -
          Attachment hadoop-8491.txt [ 12531777 ]
          Eli Collins made changes -
          Status Reopened [ 4 ] Patch Available [ 10002 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2368 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2368/)
          Revert HADOOP-8491 (Revision 1349133)

          Result = FAILURE
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349133
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/IOUtils.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtils.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2368 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2368/ ) Revert HADOOP-8491 (Revision 1349133) Result = FAILURE eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349133 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/IOUtils.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtils.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2418 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2418/)
          Revert HADOOP-8491 (Revision 1349133)

          Result = SUCCESS
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349133
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/IOUtils.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtils.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2418 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2418/ ) Revert HADOOP-8491 (Revision 1349133) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349133 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/IOUtils.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtils.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2345 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2345/)
          Revert HADOOP-8491 (Revision 1349133)

          Result = SUCCESS
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349133
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/IOUtils.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtils.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2345 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2345/ ) Revert HADOOP-8491 (Revision 1349133) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349133 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/IOUtils.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtils.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1074 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1074/)
          Revert HADOOP-8491 (Revision 1349133)
          HADOOP-8491. Check for short writes when using FileChannel#write and related methods. Contributed by Colin Patrick McCabe (Revision 1349019)

          Result = SUCCESS
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349133
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/IOUtils.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtils.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java

          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349019
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/IOUtils.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtils.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1074 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1074/ ) Revert HADOOP-8491 (Revision 1349133) HADOOP-8491 . Check for short writes when using FileChannel#write and related methods. Contributed by Colin Patrick McCabe (Revision 1349019) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349133 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/IOUtils.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtils.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349019 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/IOUtils.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtils.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1107 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1107/)
          Revert HADOOP-8491 (Revision 1349133)
          HADOOP-8491. Check for short writes when using FileChannel#write and related methods. Contributed by Colin Patrick McCabe (Revision 1349019)

          Result = FAILURE
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349133
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/IOUtils.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtils.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java

          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349019
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/IOUtils.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtils.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1107 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1107/ ) Revert HADOOP-8491 (Revision 1349133) HADOOP-8491 . Check for short writes when using FileChannel#write and related methods. Contributed by Colin Patrick McCabe (Revision 1349019) Result = FAILURE eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349133 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/IOUtils.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtils.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349019 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/IOUtils.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtils.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java
          Hide
          Eli Collins added a comment -

          Same patch to kick jenkins.

          Show
          Eli Collins added a comment - Same patch to kick jenkins.
          Eli Collins made changes -
          Attachment hadoop-8491.txt [ 12531884 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12531884/hadoop-8491.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.TestFileAppend2
          org.apache.hadoop.hdfs.TestFileLengthOnClusterRestart

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1116//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1116//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12531884/hadoop-8491.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestFileAppend2 org.apache.hadoop.hdfs.TestFileLengthOnClusterRestart +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1116//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1116//console This message is automatically generated.
          Eli Collins made changes -
          Project Hadoop Common [ 12310240 ] Hadoop HDFS [ 12310942 ]
          Key HADOOP-8491 HDFS-3531
          Affects Version/s 2.0.0-alpha [ 12320353 ]
          Affects Version/s 2.0.0-alpha [ 12320352 ]
          Fix Version/s 2.0.1-alpha [ 12321440 ]
          Fix Version/s 2.0.1-alpha [ 12321441 ]
          Eli Collins made changes -
          Summary Check for short writes when using FileChannel#write and related methods EditLogFileOutputStream#preallocate should check for incomplete writes
          Description We need to check for short writes when using WritableByteChannel#write and related methods. We need to check for short writes when using WritableByteChannel#write and related methods in EditLogFileOutputStream#preallocate so we ensure we've preallocated sufficient space.
          Eli Collins made changes -
          Fix Version/s 2.0.1-alpha [ 12321440 ]
          Target Version/s 2.0.1-alpha [ 12321440 ]
          Component/s name-node [ 12312926 ]
          Hide
          Eli Collins added a comment -

          The test failures are unrelated. Filed HDFS-3530 for TestFileAppend2 as there wasn't already a jira.

          I've committed this and merged to branch-2. Thanks Colin.

          Show
          Eli Collins added a comment - The test failures are unrelated. Filed HDFS-3530 for TestFileAppend2 as there wasn't already a jira. I've committed this and merged to branch-2. Thanks Colin.
          Eli Collins made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Target Version/s 2.0.1-alpha [ 12321440 ]
          Fix Version/s 2.0.1-alpha [ 12321440 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2353 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2353/)
          HDFS-3531. EditLogFileOutputStream#preallocate should check for incomplete writes. Contributed by Colin Patrick McCabe (Revision 1349616)

          Result = SUCCESS
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349616
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/IOUtils.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtils.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2353 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2353/ ) HDFS-3531 . EditLogFileOutputStream#preallocate should check for incomplete writes. Contributed by Colin Patrick McCabe (Revision 1349616) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349616 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/IOUtils.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtils.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2426 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2426/)
          HDFS-3531. EditLogFileOutputStream#preallocate should check for incomplete writes. Contributed by Colin Patrick McCabe (Revision 1349616)

          Result = SUCCESS
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349616
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/IOUtils.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtils.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2426 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2426/ ) HDFS-3531 . EditLogFileOutputStream#preallocate should check for incomplete writes. Contributed by Colin Patrick McCabe (Revision 1349616) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349616 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/IOUtils.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtils.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12531884/hadoop-8491.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.fs.viewfs.TestViewFsTrash
          org.apache.hadoop.hdfs.TestFileLengthOnClusterRestart

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2651//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2651//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12531884/hadoop-8491.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.fs.viewfs.TestViewFsTrash org.apache.hadoop.hdfs.TestFileLengthOnClusterRestart +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2651//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2651//console This message is automatically generated.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2375 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2375/)
          HDFS-3531. EditLogFileOutputStream#preallocate should check for incomplete writes. Contributed by Colin Patrick McCabe (Revision 1349616)

          Result = FAILURE
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349616
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/IOUtils.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtils.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2375 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2375/ ) HDFS-3531 . EditLogFileOutputStream#preallocate should check for incomplete writes. Contributed by Colin Patrick McCabe (Revision 1349616) Result = FAILURE eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349616 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/IOUtils.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtils.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1075 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1075/)
          HDFS-3531. EditLogFileOutputStream#preallocate should check for incomplete writes. Contributed by Colin Patrick McCabe (Revision 1349616)

          Result = SUCCESS
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349616
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/IOUtils.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtils.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1075 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1075/ ) HDFS-3531 . EditLogFileOutputStream#preallocate should check for incomplete writes. Contributed by Colin Patrick McCabe (Revision 1349616) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349616 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/IOUtils.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtils.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1108 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1108/)
          HDFS-3531. EditLogFileOutputStream#preallocate should check for incomplete writes. Contributed by Colin Patrick McCabe (Revision 1349616)

          Result = FAILURE
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349616
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/IOUtils.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtils.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1108 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1108/ ) HDFS-3531 . EditLogFileOutputStream#preallocate should check for incomplete writes. Contributed by Colin Patrick McCabe (Revision 1349616) Result = FAILURE eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349616 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/IOUtils.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtils.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java
          Arun C Murthy made changes -
          Fix Version/s 2.0.2-alpha [ 12322472 ]
          Fix Version/s 2.1.0-alpha [ 12321440 ]
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Patch Available Patch Available
          1d 1h 29m 1 Colin Patrick McCabe 08/Jun/12 21:08
          Resolved Resolved Reopened Reopened
          7h 49m 1 Tsz Wo Nicholas Sze 12/Jun/12 06:15
          Reopened Reopened Patch Available Patch Available
          1h 9m 1 Eli Collins 12/Jun/12 07:24
          Patch Available Patch Available Resolved Resolved
          3d 21h 42m 2 Eli Collins 13/Jun/12 03:49
          Resolved Resolved Closed Closed
          120d 14h 57m 1 Arun C Murthy 11/Oct/12 18:46

            People

            • Assignee:
              Colin Patrick McCabe
              Reporter:
              Colin Patrick McCabe
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development