> The question is not if we should handle these cases, but rather how.
The first thing is that we should handle limitations. Current code does not, the patch does. I am glad we agree on that.
On "how", as I said before 256 comes from practical observations. I have seen cases when nodes were struggling to handle
more than that, and I'd rather be conservative here than leaving the problem unsolved by setting the limit too high.
> this will lead to problems and will limit Hadoop functionality.
On the contrary, currently the functionality of hadoop is bounded by the lack of thread limitation because nodes become dysfunctional.
Introducing the limit will make it functional again.
The 256 limit does not look low if you look at it from the point of view of how many clients can simultaneously do transfers.
On a 2000 node cluster it is about 500,000 of them. It is pretty big even if you divide it by the replication factor of 3 for writes.
Although I agree it would be better to have a method of calculating the limit based on some natural criteria like hardware
configuration or heap size. I would be glad to hear ideas in this direction.