MAPREDUCE-1943 was introduced to avoid memory corruption, 50 groups
and 120 counters is an awfully low limit. The submitter complained of
jobs which had millions of counters. I believe that the current limits
will cap Counters after consuming around 32KB of memory, which IMHO is
not a reasonable tradeoff in footprint vs inconvenience.
Regarding inconvenience: note that:
(1) The counter & group limits are currently hard fails. Jobs will stop working.
(2) it's a change to the current behavior; jobs which worked under 0.20 will now fail.
If the limits were raised to (for example):
1000 char name limit
they would prevent crashing all but the most comically limited clusters
(I measure Counters consuming 21mb with these limits), and trip up far fewer
programmers. For the same reason, I think Tom's patch should be applied so they're
expandable without requiring a recompilation of hadoop.