Description
Pls find the analysis. Correct me if am wrong
2012-01-15 05:14:02,374 FATAL org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: WriterThread-9 Got while writing log entry to log
java.io.IOException: All datanodes 10.18.40.200:50010 are bad. Aborting...
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:3373)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2811)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:3026)
Here we have an exception in one of the writer threads. If any exception we try to hold it in an Atomic variable
private void writerThreadError(Throwable t) { thrown.compareAndSet(null, t); }
In the finally block of splitLog we try to close the streams.
for (WriterThread t: writerThreads) { try { t.join(); } catch (InterruptedException ie) { throw new IOException(ie); } checkForErrors(); } LOG.info("Split writers finished"); return closeStreams();
Inside checkForErrors
private void checkForErrors() throws IOException { Throwable thrown = this.thrown.get(); if (thrown == null) return; if (thrown instanceof IOException) { throw (IOException)thrown; } else { throw new RuntimeException(thrown); } } So once we throw the exception the DFSStreamer threads are not getting closed.
Attachments
Attachments
Activity
Gavin McDonald
made changes -
Workflow | patch-available, re-open possible [ 13758207 ] | no-reopen-closed, patch-avail [ 13797199 ] |
Gavin McDonald
made changes -
Workflow | no-reopen-closed, patch-avail [ 12649857 ] | patch-available, re-open possible [ 13758207 ] |
Lars Francke
made changes -
Status | Resolved [ 5 ] | Closed [ 6 ] |
ramkrishna.s.vasudevan
made changes -
Resolution | Fixed [ 1 ] | |
Status | Reopened [ 4 ] | Resolved [ 5 ] |
Ted Yu
made changes -
Resolution | Fixed [ 1 ] | |
Status | Resolved [ 5 ] | Reopened [ 4 ] |
ramkrishna.s.vasudevan
made changes -
Hadoop Flags | Reviewed [ 10343 ] | |
Resolution | Fixed [ 1 ] | |
Status | Patch Available [ 10002 ] | Resolved [ 5 ] |
ramkrishna.s.vasudevan
made changes -
Attachment | HBASE-5235_0.90_2.patch [ 12511427 ] |
ramkrishna.s.vasudevan
made changes -
Status | Open [ 1 ] | Patch Available [ 10002 ] |
ramkrishna.s.vasudevan
made changes -
Attachment | HBASE-5235_trunk.patch [ 12511401 ] |
ramkrishna.s.vasudevan
made changes -
Attachment | HBASE-5235_0.90_1.patch [ 12511397 ] |
ramkrishna.s.vasudevan
made changes -
Attachment | HBASE-5235_0.90.patch [ 12511376 ] |
ramkrishna.s.vasudevan
made changes -
Field | Original Value | New Value |
---|---|---|
Summary | HLogSplitter writer threads not getting closed when any of the writer threads has exceptions. | HLogSplitter writer thread's streams not getting closed when any of the writer threads has exceptions. |
ramkrishna.s.vasudevan
created issue -
This issue was closed as part of a bulk closing operation on 2015-11-20. All issues that have been resolved and where all fixVersions have been released have been closed (following discussions on the mailing list).