Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-2054

BlockSender.sendChunk() prints ERROR for connection closures encountered during transferToFully()

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.22.0, 0.23.0
    • Fix Version/s: 0.22.0, 0.23.0
    • Component/s: datanode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The addition of ERROR was part of HDFS-1527. In environments where clients tear down FSInputStream/connection before reaching the end of stream, this error message often pops up. Since these are not really errors and especially not the fault of data node, the message should be toned down at least.

      1. HDFS-2054.patch
        0.7 kB
        Kihwal Lee
      2. HDFS-2054.patch
        1 kB
        Kihwal Lee
      3. HDFS-2054-1.patch
        1 kB
        Kihwal Lee
      4. HDFS-2054-2.patch
        1 kB
        Kihwal Lee
      5. HDFS-2054.patch
        1 kB
        Kihwal Lee

        Activity

        Hide
        Suresh Srinivas added a comment -

        I missed Jakob's comment as I had not looked though all the comments. I looked at HDFS-2054-2.patch, I still see it. The committed code does not have it though.

        Just wanted to make sure we are all aware of need to remove StringifyException.

        Show
        Suresh Srinivas added a comment - I missed Jakob's comment as I had not looked though all the comments. I looked at HDFS-2054 -2.patch, I still see it. The committed code does not have it though. Just wanted to make sure we are all aware of need to remove StringifyException.
        Hide
        Kihwal Lee added a comment -

        @suresh Is it something different from what Jakob mentioned above? This patch does not use StringifyException as suggested by Jakob and the rest were cleaned up in HDFS-1977.

        Show
        Kihwal Lee added a comment - @suresh Is it something different from what Jakob mentioned above? This patch does not use StringifyException as suggested by Jakob and the rest were cleaned up in HDFS-1977 .
        Hide
        Suresh Srinivas added a comment -

        Sorry for the late comment. We should cleanup using stringifyException. This could be cleaned up later by a single jira though.

        Show
        Suresh Srinivas added a comment - Sorry for the late comment. We should cleanup using stringifyException. This could be cleaned up later by a single jira though.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #723 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/723/)
        HDFS-2054 BlockSender.sendChunk() prints ERROR for connection closures encountered during transferToFully() – moved the change notice into 0.22 section (i'd originally committed it in trunk section)
        HDFS-2054 BlockSender.sendChunk() prints ERROR for connection closures encountered during transferToFully()

        stack : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1145753
        Files :

        • /hadoop/common/trunk/hdfs/CHANGES.txt

        stack : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1145751
        Files :

        • /hadoop/common/trunk/hdfs/CHANGES.txt
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #723 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/723/ ) HDFS-2054 BlockSender.sendChunk() prints ERROR for connection closures encountered during transferToFully() – moved the change notice into 0.22 section (i'd originally committed it in trunk section) HDFS-2054 BlockSender.sendChunk() prints ERROR for connection closures encountered during transferToFully() stack : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1145753 Files : /hadoop/common/trunk/hdfs/CHANGES.txt stack : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1145751 Files : /hadoop/common/trunk/hdfs/CHANGES.txt /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-22-branch #71 (See https://builds.apache.org/job/Hadoop-Hdfs-22-branch/71/)
        HDFS-2054 BlockSender.sendChunk() prints ERROR for connection closures encountered during transferToFully()

        stack : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1145752
        Files :

        • /hadoop/common/branches/branch-0.22/hdfs/CHANGES.txt
        • /hadoop/common/branches/branch-0.22/hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-22-branch #71 (See https://builds.apache.org/job/Hadoop-Hdfs-22-branch/71/ ) HDFS-2054 BlockSender.sendChunk() prints ERROR for connection closures encountered during transferToFully() stack : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1145752 Files : /hadoop/common/branches/branch-0.22/hdfs/CHANGES.txt /hadoop/common/branches/branch-0.22/hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #782 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/782/)
        HDFS-2054 BlockSender.sendChunk() prints ERROR for connection closures encountered during transferToFully() – moved the change notice into 0.22 section (i'd originally committed it in trunk section)
        HDFS-2054 BlockSender.sendChunk() prints ERROR for connection closures encountered during transferToFully()

        stack : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1145753
        Files :

        • /hadoop/common/trunk/hdfs/CHANGES.txt

        stack : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1145751
        Files :

        • /hadoop/common/trunk/hdfs/CHANGES.txt
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #782 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/782/ ) HDFS-2054 BlockSender.sendChunk() prints ERROR for connection closures encountered during transferToFully() – moved the change notice into 0.22 section (i'd originally committed it in trunk section) HDFS-2054 BlockSender.sendChunk() prints ERROR for connection closures encountered during transferToFully() stack : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1145753 Files : /hadoop/common/trunk/hdfs/CHANGES.txt stack : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1145751 Files : /hadoop/common/trunk/hdfs/CHANGES.txt /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java
        Hide
        stack added a comment -

        Committed to 0.22 branch and to trunk. Thanks for the patch Kihwal (Nice reviewing lads, Todd and Jakob)

        Show
        stack added a comment - Committed to 0.22 branch and to trunk. Thanks for the patch Kihwal (Nice reviewing lads, Todd and Jakob)
        Hide
        Jakob Homan added a comment -

        +1

        Show
        Jakob Homan added a comment - +1
        Hide
        Kihwal Lee added a comment -

        -1 tests included. The patch doesn't appear to include any new or modified tests.

        The previous justification also applies to the new patch.

        Show
        Kihwal Lee added a comment - -1 tests included. The patch doesn't appear to include any new or modified tests. The previous justification also applies to the new patch.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12486211/HDFS-2054.patch
        against trunk revision 1145428.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/909//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/909//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/909//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12486211/HDFS-2054.patch against trunk revision 1145428. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/909//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/909//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/909//console This message is automatically generated.
        Hide
        Kihwal Lee added a comment -

        I incoporated Jakob's comment and a new patch has been posted.

        Show
        Kihwal Lee added a comment - I incoporated Jakob's comment and a new patch has been posted.
        Hide
        Jakob Homan added a comment -

        Since we're trying to remove the calls to StringifyException (HDFS-1977), can we do so with this patch as well?

        Show
        Jakob Homan added a comment - Since we're trying to remove the calls to StringifyException ( HDFS-1977 ), can we do so with this patch as well?
        Hide
        stack added a comment -

        +1 on patch. Nice comment. Small change. Will commit in next couple of hours unless objection.

        Show
        stack added a comment - +1 on patch. Nice comment. Small change. Will commit in next couple of hours unless objection.
        Hide
        Kihwal Lee added a comment -

        No test was added since the patch only changes the log message.

        Show
        Kihwal Lee added a comment - No test was added since the patch only changes the log message.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12483738/HDFS-2054-2.patch
        against trunk revision 1139124.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/832//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/832//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/832//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12483738/HDFS-2054-2.patch against trunk revision 1139124. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/832//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/832//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/832//console This message is automatically generated.
        Hide
        Kihwal Lee added a comment -

        Thanks. I will repost the patch. Last time I did that Jenkins didn't pick it up.

        Show
        Kihwal Lee added a comment - Thanks. I will repost the patch. Last time I did that Jenkins didn't pick it up.
        Hide
        Aaron T. Myers added a comment -

        Hey Kihwal, simply switching from patch available -> open -> patch available won't trigger a new pre-commit test-patch run. You either need to upload a new file (which could have the same contents) or ask someone with Hudson power to kick it for you. Looks like there's presently something missing from my Hudson privs, otherwise I would do it for you.

        Show
        Aaron T. Myers added a comment - Hey Kihwal, simply switching from patch available -> open -> patch available won't trigger a new pre-commit test-patch run. You either need to upload a new file (which could have the same contents) or ask someone with Hudson power to kick it for you. Looks like there's presently something missing from my Hudson privs, otherwise I would do it for you.
        Hide
        Kihwal Lee added a comment -

        The new patch has been uploaded.

        Show
        Kihwal Lee added a comment - The new patch has been uploaded.
        Hide
        Kihwal Lee added a comment -

        I found the following from Sun's NIO example.

        j2se/share/sample/nio/server/RequestHandler.java

                } catch (IOException x) {
                    String m = x.getMessage();
                    if (!m.equals("Broken pipe") &&
                            !m.equals("Connection reset by peer")) {
                        System.err.println("RequestHandler: " + x.toString());
                    }
        

        I will add the check.

        Show
        Kihwal Lee added a comment - I found the following from Sun's NIO example. j2se/share/sample/nio/server/RequestHandler.java } catch (IOException x) { String m = x.getMessage(); if (!m.equals( "Broken pipe" ) && !m.equals( "Connection reset by peer" )) { System .err.println( "RequestHandler: " + x.toString()); } I will add the check.
        Hide
        Todd Lipcon added a comment -

        Hey Kihwal. Don't we want to check for "Connection reset by peer" also? I've seen that as well.

        Show
        Todd Lipcon added a comment - Hey Kihwal. Don't we want to check for "Connection reset by peer" also? I've seen that as well.
        Hide
        Kihwal Lee added a comment -

        Test failures:

        TestHDFSCLI: The quota related test failures are not due to this patch. They also failed in build #696.
        TestHDFSTrash : It was failing in other recent pre-commit builds: e.g. https://builds.apache.org/job/PreCommit-HDFS-Build/786/

        Tests included: No test is included as justified above.

        Show
        Kihwal Lee added a comment - Test failures: TestHDFSCLI: The quota related test failures are not due to this patch. They also failed in build #696. TestHDFSTrash : It was failing in other recent pre-commit builds: e.g. https://builds.apache.org/job/PreCommit-HDFS-Build/786/ Tests included: No test is included as justified above.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12482699/HDFS-2054.patch
        against trunk revision 1136132.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these core unit tests:
        org.apache.hadoop.cli.TestHDFSCLI
        org.apache.hadoop.hdfs.TestHDFSTrash

        +1 contrib tests. The patch passed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/788//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/788//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/788//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12482699/HDFS-2054.patch against trunk revision 1136132. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestHDFSCLI org.apache.hadoop.hdfs.TestHDFSTrash +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/788//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/788//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/788//console This message is automatically generated.
        Hide
        Kihwal Lee added a comment -

        No test is added since it only changes the log message.

        Show
        Kihwal Lee added a comment - No test is added since it only changes the log message.
        Hide
        Todd Lipcon added a comment -

        Makes sense, let's just do (a) then

        Show
        Todd Lipcon added a comment - Makes sense, let's just do (a) then
        Hide
        Kihwal Lee added a comment -

        @Todd: (b) may not work either. TransferTo() accesses fd directly and nothing sets the class private variables for indicating the connection status in objects that uses it (e.g. Socket). SocketChannel.write() will also cause EPIPE and an ioe at this point. Only after close() method is explicitly called, the private variables are set and subsequently the open/close check methods will work as expected. Reading may work if the read buffer is not empty even after a write got EPIPE. So testing the connection with read is not reliable either. o.a.h.net.SocketOutputStream.write() used in other portions of sendChunks() does catch IOException and close the stream. That's the source of the isOpen() inconsistency I reported above.

        The NIO could check for EPIPE and make an AsynchronousCloseException thrown in FileChannel. Otherwise it is very hard to handle different types of exceptions in different ways. For SocketChannel, most Java code assumes IOException means connection closure. The same assumption cannot be made in FileChannel (e.g. HDFS-1527).

        Short of the support from NIO, (a) seems to be the only option. We could do this in o.a.h.net.SocketOutputStream.transferToFully(), but the hadoop-common class might be used by others and connection closure be handled by catching IOException. So the safest thing we can do at this point is adding (a) to BlockSender.

        Show
        Kihwal Lee added a comment - @Todd: (b) may not work either. TransferTo() accesses fd directly and nothing sets the class private variables for indicating the connection status in objects that uses it (e.g. Socket). SocketChannel.write() will also cause EPIPE and an ioe at this point. Only after close() method is explicitly called, the private variables are set and subsequently the open/close check methods will work as expected. Reading may work if the read buffer is not empty even after a write got EPIPE. So testing the connection with read is not reliable either. o.a.h.net.SocketOutputStream.write() used in other portions of sendChunks() does catch IOException and close the stream. That's the source of the isOpen() inconsistency I reported above. The NIO could check for EPIPE and make an AsynchronousCloseException thrown in FileChannel. Otherwise it is very hard to handle different types of exceptions in different ways. For SocketChannel, most Java code assumes IOException means connection closure. The same assumption cannot be made in FileChannel (e.g. HDFS-1527 ). Short of the support from NIO, (a) seems to be the only option. We could do this in o.a.h.net.SocketOutputStream.transferToFully(), but the hadoop-common class might be used by others and connection closure be handled by catching IOException. So the safest thing we can do at this point is adding (a) to BlockSender.
        Hide
        Todd Lipcon added a comment -

        hrm... that's a pain. I guess our options are (a) parsing exception messages, or (b) passing the Socket object itself to BlockSender such that it can determine whether it's still open. Any other good ideas?

        Show
        Todd Lipcon added a comment - hrm... that's a pain. I guess our options are (a) parsing exception messages, or (b) passing the Socket object itself to BlockSender such that it can determine whether it's still open. Any other good ideas?
        Hide
        Kihwal Lee added a comment -

        I tried SocketOutputStream.isOpen() in BlockSender.sendChunk(), but it seems even after EPIPE, isOpen() is not guaranteed to return false.

        Show
        Kihwal Lee added a comment - I tried SocketOutputStream.isOpen() in BlockSender.sendChunk(), but it seems even after EPIPE, isOpen() is not guaranteed to return false.
        Hide
        Kihwal Lee added a comment -

        >I'm not in favor of parsing the exception text for behavior-altering
        >things. But for deciding whether to log at debug vs warn level, it
        >seems OK to me.

        This sounds reasonable.

        >Another thought is to check something like socket.isInputShutdown()
        >or socket.isConnected()? Maybe we can assume that any case where we
        >get an IOE but the socket was then found to be disconnected is OK.
        >If we had a local IOE with the transferto, the socket would still be up.

        This is even better, IMO.

        Show
        Kihwal Lee added a comment - >I'm not in favor of parsing the exception text for behavior-altering >things. But for deciding whether to log at debug vs warn level, it >seems OK to me. This sounds reasonable. >Another thought is to check something like socket.isInputShutdown() >or socket.isConnected()? Maybe we can assume that any case where we >get an IOE but the socket was then found to be disconnected is OK. >If we had a local IOE with the transferto, the socket would still be up. This is even better, IMO.
        Hide
        Todd Lipcon added a comment -

        Yea, it sucks that Java doesn't give us a way to get at the underlying errno in these cases. For the IOEs thrown by the hadoop-native code in common, we actually have an Errno enum that makes life easy.

        I'm not in favor of parsing the exception text for behavior-altering things. But for deciding whether to log at debug vs warn level, it seems OK to me.

        Another thought is to check something like socket.isInputShutdown() or socket.isConnected()? Maybe we can assume that any case where we get an IOE but the socket was then found to be disconnected is OK. If we had a local IOE with the transferto, the socket would still be up.

        Show
        Todd Lipcon added a comment - Yea, it sucks that Java doesn't give us a way to get at the underlying errno in these cases. For the IOEs thrown by the hadoop-native code in common, we actually have an Errno enum that makes life easy. I'm not in favor of parsing the exception text for behavior-altering things. But for deciding whether to log at debug vs warn level, it seems OK to me. Another thought is to check something like socket.isInputShutdown() or socket.isConnected()? Maybe we can assume that any case where we get an IOE but the socket was then found to be disconnected is OK. If we had a local IOE with the transferto, the socket would still be up.
        Hide
        Kihwal Lee added a comment -

        Last time I tried what you said with EAGAIN in transferTo() in attempt to avoid doing epoll() evey time even before sending anything. Some folks were not thrilled about parsing the text. If it can be done in portable/i14n friendly way and people do not object the idea itself...

        Show
        Kihwal Lee added a comment - Last time I tried what you said with EAGAIN in transferTo() in attempt to avoid doing epoll() evey time even before sending anything. Some folks were not thrilled about parsing the text. If it can be done in portable/i14n friendly way and people do not object the idea itself...
        Hide
        Todd Lipcon added a comment -

        Maybe we can check the exception type and message, and only log warning for unexpected ones? EG "Connection reset by peer" and "Broken pipe" are expected exceptions, but anything else should be logged at WARN level.

        Show
        Todd Lipcon added a comment - Maybe we can check the exception type and message, and only log warning for unexpected ones? EG "Connection reset by peer" and "Broken pipe" are expected exceptions, but anything else should be logged at WARN level.
        Hide
        Kihwal Lee added a comment -

        To minimum, it will get rid of the annoying stack trace.

        transferTo() is not exactly making it easy to deal with different exceptions differently. I believe things like EAGAIN was fixed now, but to deal with others you have to parse the error itself, which is rather gross. Ideally we want to deal with EAGAIN, EPIPE, etc. separately and if something else happens print an error message.

        Show
        Kihwal Lee added a comment - To minimum, it will get rid of the annoying stack trace. transferTo() is not exactly making it easy to deal with different exceptions differently. I believe things like EAGAIN was fixed now, but to deal with others you have to parse the error itself, which is rather gross. Ideally we want to deal with EAGAIN, EPIPE, etc. separately and if something else happens print an error message.
        Hide
        stack added a comment -

        @Kihwal Do we think this enough to address this issue? I see loads of it running hbase loadings on 0.22.

        Show
        stack added a comment - @Kihwal Do we think this enough to address this issue? I see loads of it running hbase loadings on 0.22.
        Hide
        Kihwal Lee added a comment -

        It may reveal interesting errors in the future, so the log level is being lowered to warn.

        Show
        Kihwal Lee added a comment - It may reveal interesting errors in the future, so the log level is being lowered to warn.
        Hide
        Patrick Kling added a comment -

        If I remember correctly, the bug fixed by HDFS-1527 was causing the affected transfers to fail silently. That's why I added this message. If it is polluting the log file, I have no objection to downgrading this to a warning.

        Show
        Patrick Kling added a comment - If I remember correctly, the bug fixed by HDFS-1527 was causing the affected transfers to fail silently. That's why I added this message. If it is polluting the log file, I have no objection to downgrading this to a warning.

          People

          • Assignee:
            Kihwal Lee
            Reporter:
            Kihwal Lee
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development