This is an issue could affect all the releases which includes
Basically, we use regex match repeatedly when we read mapper/reducer resource request from config files. When we have large config file, and large number of splits, it could take a long time.
We saw AM could take hours to parse config when we have 200k+ splits, with a large config file (hundreds of kbs).
The problematic part is this:
Inside ResourceUtils.getRequestedResourcesFromConfig(), we call Configuration.getValByRegex() which goes through all property keys that come from the MapReduce job configuration (jobconf.xml). If the job config is large (eg. due to being part of an MR pipeline and it was populated by an earlier job), then this results in running a regexp match unnecessarily for all properties over and over again. This is not necessary, because all mappers and reducers will have the same config, respectively.
We should do proper caching for pre-configured resource requests.