Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Duplicate
-
0.20.2
-
None
-
None
-
None
Description
We recently started experiencing problems where Secondary Namenode fails to fetch the image from the NameNode. The basic problem is described in HDFS-1024, but that JIRA was only dealing with possible data corruption, since then we got to a place where we could not compact the fsimage anymore because failures were 100% of the time.
Here is what we have found out:
The fetch still fails with the same exception as the HDFS-1024 (Jetty closes the connection before the file is sent)
We suspect the underlying reason to be extensive garbage collection on the NameNode (1/5 of all time is being spent in garbage collection). And the reason for that might be the bug that is solved with HADOOP-6577 (we have a lot of large RPC requests, which means we allocate and free a lot of memory all the time).
Because of GC the speed of the transfer drops to 700Kb/s
Having said all of that current mechanism of fetching the image is still potentially flawed. When dealing with large images namenode is under stress of sending multigig files over the wire to the client while still serving requests.
This JIRA is to discuss the possible ways of separating NameNode and the image fetching by the secondary namenode.
One thought we had was fetching the image using SCP rather than HTTP download from the NameNode.
This way the NameNode will have less pressure on it, on the other hand this will introduce new components that are not exactly under hadoop control (ssh client and server).
To deal with possible data corruption with SCP copy we would also want to extend CheckpointSignature to have checksum on the file, so it can be checked on the client side.
Please let me know what you think.