FieldInfos.fieldInfo(int) looks up a field by number and returns its FieldInfo.
This method is called per-field-per-doc in things like stored fields and vectors readers.
Unfortunately, today this method is always backed by a TreeMap. In most cases a simple array is better, its faster and uses less memory.
These changes made significant difference in stored fields checkindex time with my test index (had only 10 fields). Maybe it helps merge as well.