Details
-
Bug
-
Status: Open
-
Minor
-
Resolution: Unresolved
-
None
-
None
-
None
Description
Recently while reviewing the HDFSBolt code because of a question on the mailing list, I noticed that the abstract bolt will fail a tuple if an IOException is thrown while trying to write it out, and then force a sync in those cases.
A RuntimeException thrown by the formatter, on the other hand, bubbles up and forces the the worker to restart.
Any IOException thrown by an Hdfs Output Stream means at that point the stream is closed and cannot be used any more. As part of our recovery we will try to sync, but this will also fail because the stream is closed by the exception that was thrown, and will result in the sync failing an a RuntimeException being thrown, and the entire worker being restarted.
The current code "works" and eventually will recover from these issues, but it may take a while. It also means that we are likely to have more data loss than needed for some output formats.
I would suggest that we try to recover from RuntimeExceptions in the same say that we are trying to recover from IOExceptions now. I also would suggest that we handle the special case where the tupleBatch.size() is 0 but we got an IOException from the writer, as the forceSync will not happen so tuples will continue to fail until the sync policy decides to sync, at which point the worker will crash and then recover.