[SPARK-8135] Don't load defaults when reconstituting Hadoop Configurations - ASF JIRA

Attach files

Attach Screenshot

Voters

Watch issue

Watchers

Create sub-task

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Improvement
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: None
Fix Version/s: 1.5.0
Component/s: Spark Core
Labels:
None

Target Version/s:

1.5.0

Description

Calling "new Configuration()" is an expensive operation because it loads any Hadoop configuration XMLs from disk.

In SerializableWritable, we call new Configuration needlessly when instantiating an ObjectWritable. The ObjectWritable only needs the Configuration for its class cache, not for any Hadoop properties that might be in XML files, so it should be ok to call new Configuration with loadDefaults = false.