NPE in handling deprecated configuration keys.

• Key: HADOOP-6243
• Type: Bug
• Status: Resolved
• Resolution: Fixed
• Priority: Blocker
• Assignee: Sreekanth Ramakrishnan
• Reporter: Konstantin Shvachko
• Component/s: conf
• Affects Version/s: 0.21.0
• Fix Version/s: 0.21.0

Description

I run TestFileCreation in Eclipse and get a NullPointerException. Debugging shows that Configuration.populateDeprecationMapping() sets Configuration.properties to null.
This code was introduced in HADOOP-6105.

Comments

[Konstantin Shvachko]

Running the same test using ant command line works fine. Don't know what the difference is.

[Hemanth Yamijala]

Chaitanya, Can you please take a look ? Please note Konstantin's helpful clues about difference in environment.

[Hemanth Yamijala]

I am marking this a blocker, because it is breaking builds. It was not my intent that the common jar containing HADOOP-6105 is committed to the M/R and HDFS projects. We were still running tests with Hudson jars. However, it appears that commit of HADOOP-6133 inadvertently included this fix as well.

[Sreekanth Ramakrishnan]

The patch fixes the NullPointerException which was thrown when Configuration.get() method is invoked.

The Configuration.processDeprecation() used to load the classes which were present as part of hadoop.conf.extra.classes key.

If one of those classes have a static block which does an Configuration.addResource(), the property map maintained by the Configuration is set to null causing the NullPointerException

After, offline discussion with Hemanth and Chaitanya with regards to correct fix, a new JIRA would be opened in order to discuss how best to expose the usage of HADOOP-6105 to consumers, and this JIRA would provide a short term fix in which consumers are expected to change the static block in Configuration class for using HADOOP-6105

[Hadoop QA]

+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12418892/HADOOP-6243-1.patch
against trunk revision 812317.

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 3 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 core tests. The patch passed core unit tests.

+1 contrib tests. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/24/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/24/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/24/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/24/console

This message is automatically generated.

[Hemanth Yamijala]

This seems fine to me. +1.

After, offline discussion with Hemanth and Chaitanya with regards to correct fix, a new JIRA would be opened in order to discuss how best to expose the usage of HADOOP-6105 to consumers

Opened HADOOP-6245 for tracking this.

[Hemanth Yamijala]

I committed this to trunk. Thanks, Sreekanth !

[Hemanth Yamijala]

Leaving this open until I commit the jars to the affected projects.

[Hudson]

Integrated in Hadoop-Common-trunk #89 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/89/)
. Fix a NullPointerException in processing deprecated keys. Contributed by Sreekanth Ramakrishnan.

[Hudson]

Integrated in Hadoop-Common-trunk-Commit #19 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/19/)
. Fix a NullPointerException in processing deprecated keys. Contributed by Sreekanth Ramakrishnan.

[Hemanth Yamijala]

I committed the hadoop common jars to mapreduce project after verifying that jobs are running fine where they previously did not.

[Hudson]

Integrated in Hadoop-Mapreduce-trunk-Commit #23 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/23/)
. Fixes a NullPointerException in handling deprecated keys. Contributed by Sreekanth Ramakrishnan.

[Hemanth Yamijala]

I ran HDFS tests with the jars got from http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/19/ and http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/23/ and they passed on my machine. Going to commit the jars to HDFS based on this.

[Hemanth Yamijala]

I committed common and mapreduce jars to HDFS thus completing the fix. Thanks, Sreekanth !

[Hudson]

Integrated in Hadoop-Hdfs-trunk-Commit #20 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/20/)
. Updating Common and Mapreduce jars fixing an NPE in handling deprecated keys. Contributed by Sreekanth Ramakrishnan.

[Konstantin Shvachko]

Worked for me, thanks.

[Hudson]

Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #5 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/5/)
. Updating Common and Mapreduce jars fixing an NPE in handling deprecated keys. Contributed by Sreekanth Ramakrishnan.

[Hudson]

Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #21 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/21/)