Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
3.3.3
-
None
-
None
Description
Hi,
I am working on a HDFS smart storage management project recently. It is based on the Mover in Hadoop-hdfs project. I noticed that most code in Mover is similar to Balancer. However, Mover doesn't clean up MoverExecutor as Balancer does.
If we have multiple NameSystem for Namenode Connectors or have a large number of datanodes, Mover will result in threads leaking because there might be numerous iterations to process these namespaces. Like our project, we modified some source code so that we can use mover.run() once we found the blocks did not match the expected storage policies. So our application will initialize Namenode Connector and Mover continually. It turns out we have thousands of threads or threads pools for MoverExecutor.
here is what it looks like. We can see here are 9000+ threads like this in WAIT condition.
I know generally users may not use Mover like us. They might use it by CLI. But more and more users are planing to apply RBF or multiple NameSystems, or with a large cluster of datanodes. Mover CLI have to keep more than thousands of thread after pressing the enter key.
I have pulled a quick fix code, if you guys are interested, plz take a look at it.
thx.
