Xiao Chen Thanks for your response.
From a quick look I doubt using the correct......
I'm afraid not. The pipeline of transform data is:
webhdfs-client ->(1) webhdfs server in datanode ->(2) hdfs block.
And exceptionCaught only takes effect when pipeline(1) encounter exceptions. The exception of this issue happens in pipeline(2).
What's more, pipeline(2) relies on DataStreamer, it's a asynchronous data transfer with buffer in another thread. Only when closing OutputStream and flushing data , we have the chance to catch the exception.
It's not elegant to ignore the exception in my code change by the way, I'll bring up a new way to fix it.
For example, would MiniDFSCluster#shutdownDataNodes right..
Good suggestion, It will certainly work in this test case. I will apply it in my next patch.