Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-6243

NPE in handling deprecated configuration keys.

Details

    • Bug
    • Status: Closed
    • Blocker
    • Resolution: Fixed
    • 0.21.0
    • 0.21.0
    • conf
    • None
    • Reviewed

    Description

      I run TestFileCreation in Eclipse and get a NullPointerException. Debugging shows that Configuration.populateDeprecationMapping() sets Configuration.properties to null.
      This code was introduced in HADOOP-6105.

      Attachments

        1. HADOOP-6243-1.patch
          7 kB
          Sreekanth Ramakrishnan

        Issue Links

          Activity

            Running the same test using ant command line works fine. Don't know what the difference is.

            shv Konstantin Shvachko added a comment - Running the same test using ant command line works fine. Don't know what the difference is.

            Chaitanya, Can you please take a look ? Please note Konstantin's helpful clues about difference in environment.

            yhemanth Hemanth Yamijala added a comment - Chaitanya, Can you please take a look ? Please note Konstantin's helpful clues about difference in environment.

            I am marking this a blocker, because it is breaking builds. It was not my intent that the common jar containing HADOOP-6105 is committed to the M/R and HDFS projects. We were still running tests with Hudson jars. However, it appears that commit of HADOOP-6133 inadvertently included this fix as well.

            yhemanth Hemanth Yamijala added a comment - I am marking this a blocker, because it is breaking builds. It was not my intent that the common jar containing HADOOP-6105 is committed to the M/R and HDFS projects. We were still running tests with Hudson jars. However, it appears that commit of HADOOP-6133 inadvertently included this fix as well.

            The patch fixes the NullPointerException which was thrown when Configuration.get() method is invoked.

            The Configuration.processDeprecation() used to load the classes which were present as part of hadoop.conf.extra.classes key.

            If one of those classes have a static block which does an Configuration.addResource(), the property map maintained by the Configuration is set to null causing the NullPointerException

            After, offline discussion with Hemanth and Chaitanya with regards to correct fix, a new JIRA would be opened in order to discuss how best to expose the usage of HADOOP-6105 to consumers, and this JIRA would provide a short term fix in which consumers are expected to change the static block in Configuration class for using HADOOP-6105

            sreekanth Sreekanth Ramakrishnan added a comment - The patch fixes the NullPointerException which was thrown when Configuration.get() method is invoked. The Configuration.processDeprecation() used to load the classes which were present as part of hadoop.conf.extra.classes key. If one of those classes have a static block which does an Configuration.addResource() , the property map maintained by the Configuration is set to null causing the NullPointerException After, offline discussion with Hemanth and Chaitanya with regards to correct fix, a new JIRA would be opened in order to discuss how best to expose the usage of HADOOP-6105 to consumers, and this JIRA would provide a short term fix in which consumers are expected to change the static block in Configuration class for using HADOOP-6105
            hadoopqa Hadoop QA added a comment -

            +1 overall. Here are the results of testing the latest attachment
            http://issues.apache.org/jira/secure/attachment/12418892/HADOOP-6243-1.patch
            against trunk revision 812317.

            +1 @author. The patch does not contain any @author tags.

            +1 tests included. The patch appears to include 3 new or modified tests.

            +1 javadoc. The javadoc tool did not generate any warning messages.

            +1 javac. The applied patch does not increase the total number of javac compiler warnings.

            +1 findbugs. The patch does not introduce any new Findbugs warnings.

            +1 release audit. The applied patch does not increase the total number of release audit warnings.

            +1 core tests. The patch passed core unit tests.

            +1 contrib tests. The patch passed contrib unit tests.

            Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/24/testReport/
            Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/24/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
            Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/24/artifact/trunk/build/test/checkstyle-errors.html
            Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/24/console

            This message is automatically generated.

            hadoopqa Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12418892/HADOOP-6243-1.patch against trunk revision 812317. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/24/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/24/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/24/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/24/console This message is automatically generated.

            This seems fine to me. +1.

            After, offline discussion with Hemanth and Chaitanya with regards to correct fix, a new JIRA would be opened in order to discuss how best to expose the usage of HADOOP-6105 to consumers

            Opened HADOOP-6245 for tracking this.

            yhemanth Hemanth Yamijala added a comment - This seems fine to me. +1. After, offline discussion with Hemanth and Chaitanya with regards to correct fix, a new JIRA would be opened in order to discuss how best to expose the usage of HADOOP-6105 to consumers Opened HADOOP-6245 for tracking this.

            I committed this to trunk. Thanks, Sreekanth !

            yhemanth Hemanth Yamijala added a comment - I committed this to trunk. Thanks, Sreekanth !

            Leaving this open until I commit the jars to the affected projects.

            yhemanth Hemanth Yamijala added a comment - Leaving this open until I commit the jars to the affected projects.
            hudson Hudson added a comment -

            Integrated in Hadoop-Common-trunk #89 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/89/)
            . Fix a NullPointerException in processing deprecated keys. Contributed by Sreekanth Ramakrishnan.

            hudson Hudson added a comment - Integrated in Hadoop-Common-trunk #89 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/89/ ) . Fix a NullPointerException in processing deprecated keys. Contributed by Sreekanth Ramakrishnan.
            hudson Hudson added a comment -

            Integrated in Hadoop-Common-trunk-Commit #19 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/19/)
            . Fix a NullPointerException in processing deprecated keys. Contributed by Sreekanth Ramakrishnan.

            hudson Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #19 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/19/ ) . Fix a NullPointerException in processing deprecated keys. Contributed by Sreekanth Ramakrishnan.

            I committed the hadoop common jars to mapreduce project after verifying that jobs are running fine where they previously did not.

            yhemanth Hemanth Yamijala added a comment - I committed the hadoop common jars to mapreduce project after verifying that jobs are running fine where they previously did not.
            hudson Hudson added a comment -

            Integrated in Hadoop-Mapreduce-trunk-Commit #23 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/23/)
            . Fixes a NullPointerException in handling deprecated keys. Contributed by Sreekanth Ramakrishnan.

            hudson Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #23 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/23/ ) . Fixes a NullPointerException in handling deprecated keys. Contributed by Sreekanth Ramakrishnan.

            I ran HDFS tests with the jars got from http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/19/ and http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/23/ and they passed on my machine. Going to commit the jars to HDFS based on this.

            yhemanth Hemanth Yamijala added a comment - I ran HDFS tests with the jars got from http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/19/ and http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/23/ and they passed on my machine. Going to commit the jars to HDFS based on this.

            I committed common and mapreduce jars to HDFS thus completing the fix. Thanks, Sreekanth !

            yhemanth Hemanth Yamijala added a comment - I committed common and mapreduce jars to HDFS thus completing the fix. Thanks, Sreekanth !
            hudson Hudson added a comment -

            Integrated in Hadoop-Hdfs-trunk-Commit #20 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/20/)
            . Updating Common and Mapreduce jars fixing an NPE in handling deprecated keys. Contributed by Sreekanth Ramakrishnan.

            hudson Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #20 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/20/ ) . Updating Common and Mapreduce jars fixing an NPE in handling deprecated keys. Contributed by Sreekanth Ramakrishnan.

            Worked for me, thanks.

            shv Konstantin Shvachko added a comment - Worked for me, thanks.
            hudson Hudson added a comment -

            Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #5 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/5/)
            . Updating Common and Mapreduce jars fixing an NPE in handling deprecated keys. Contributed by Sreekanth Ramakrishnan.

            hudson Hudson added a comment - Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #5 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/5/ ) . Updating Common and Mapreduce jars fixing an NPE in handling deprecated keys. Contributed by Sreekanth Ramakrishnan.
            hudson Hudson added a comment -

            Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #21 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/21/)

            hudson Hudson added a comment - Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #21 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/21/ )

            People

              sreekanth Sreekanth Ramakrishnan
              shv Konstantin Shvachko
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: