Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
2.0.0
-
None
Description
When a job having large amount of parallelism components( total parallelism rises to 5000 for example) been submmited to storm cluster, Nimubs might get crashed, the work flow is as below:
1) Nimbus computting assignment
2) Nimbus sending assignment to zk
3) When assignment mapping info string is too long due to total parallelism of job being too large, sending this info to zk will fail (zNode datalength set default is 1M )
4) Nimbus keeps trying sending this assignment info, after some times, it gives up and crashed, with that happend, the stablity of the cluster will be greatly impacted
