[HADOOP-9570] Configuration.addResource() should only parse the new resource

• Type: Improvement
• Status: Patch Available
• Priority: Minor
• Resolution: Unresolved
• Component/s: conf

Environment

Ubuntu LXC

• Assignee: Daryn Sharp
• Reporter: Gopal Vijayaraghavan
• Created: 17/May/13 17:57
• Updated: 12/Jun/19 10:22

Description

Hadoop configuration parsing re-parses all configuration files when a new resource is added to the configuration object.

This is wasteful and runs through every deprecation check unnecessarily.

JobConf clone = new JobConf(job);
clone.addResource(...);

will reparse every file including core-site, hdfs-site etc.

Resource addition can be taken care of cleanly, if the addResourceObject() call applies the new resource, followed by applying the overlay, without re-parsing any of the older files already loaded into the hashtable.

Issue Links

• is related to: HIVE-4486 FetchOperator slows down SMB map joins by 50% when there are many partitions