[SPARK-8135] Don't load defaults when reconstituting Hadoop Configurations

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.5.0
• Component/s: Spark Core
• Target Version/s: 1.5.0
• Assignee: Sandy Ryza
• Reporter: Sandy Ryza
• Created: 05/Jun/15 23:24
• Updated: 19/Jun/15 02:36
• Resolved: 19/Jun/15 02:36

Description

Calling "new Configuration()" is an expensive operation because it loads any Hadoop configuration XMLs from disk.

In SerializableWritable, we call new Configuration needlessly when instantiating an ObjectWritable. The ObjectWritable only needs the Configuration for its class cache, not for any Hadoop properties that might be in XML files, so it should be ok to call new Configuration with loadDefaults = false.

Issue Links

• links to: [Github] Pull Request #6679 (sryza)