Hadoop Common
  1. Hadoop Common
  2. HADOOP-6243

NPE in handling deprecated configuration keys.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.0
    • Component/s: conf
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      I run TestFileCreation in Eclipse and get a NullPointerException. Debugging shows that Configuration.populateDeprecationMapping() sets Configuration.properties to null.
      This code was introduced in HADOOP-6105.

      1. HADOOP-6243-1.patch
        7 kB
        Sreekanth Ramakrishnan

        Issue Links

          Activity

          Tom White made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Hide
          Hudson added a comment -

          Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #21 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/21/)

          Show
          Hudson added a comment - Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #21 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/21/ )
          Hide
          Hudson added a comment -

          Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #5 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/5/)
          . Updating Common and Mapreduce jars fixing an NPE in handling deprecated keys. Contributed by Sreekanth Ramakrishnan.

          Show
          Hudson added a comment - Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #5 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/5/ ) . Updating Common and Mapreduce jars fixing an NPE in handling deprecated keys. Contributed by Sreekanth Ramakrishnan.
          Hide
          Konstantin Shvachko added a comment -

          Worked for me, thanks.

          Show
          Konstantin Shvachko added a comment - Worked for me, thanks.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #20 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/20/)
          . Updating Common and Mapreduce jars fixing an NPE in handling deprecated keys. Contributed by Sreekanth Ramakrishnan.

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #20 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/20/ ) . Updating Common and Mapreduce jars fixing an NPE in handling deprecated keys. Contributed by Sreekanth Ramakrishnan.
          Hemanth Yamijala made changes -
          Assignee V.V.Chaitanya Krishna [ chaitk ] Sreekanth Ramakrishnan [ sreekanth ]
          Hemanth Yamijala made changes -
          Status Reopened [ 4 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          Hemanth Yamijala added a comment -

          I committed common and mapreduce jars to HDFS thus completing the fix. Thanks, Sreekanth !

          Show
          Hemanth Yamijala added a comment - I committed common and mapreduce jars to HDFS thus completing the fix. Thanks, Sreekanth !
          Hide
          Hemanth Yamijala added a comment -

          I ran HDFS tests with the jars got from http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/19/ and http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/23/ and they passed on my machine. Going to commit the jars to HDFS based on this.

          Show
          Hemanth Yamijala added a comment - I ran HDFS tests with the jars got from http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/19/ and http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/23/ and they passed on my machine. Going to commit the jars to HDFS based on this.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #23 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/23/)
          . Fixes a NullPointerException in handling deprecated keys. Contributed by Sreekanth Ramakrishnan.

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #23 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/23/ ) . Fixes a NullPointerException in handling deprecated keys. Contributed by Sreekanth Ramakrishnan.
          Hide
          Hemanth Yamijala added a comment -

          I committed the hadoop common jars to mapreduce project after verifying that jobs are running fine where they previously did not.

          Show
          Hemanth Yamijala added a comment - I committed the hadoop common jars to mapreduce project after verifying that jobs are running fine where they previously did not.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #19 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/19/)
          . Fix a NullPointerException in processing deprecated keys. Contributed by Sreekanth Ramakrishnan.

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #19 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/19/ ) . Fix a NullPointerException in processing deprecated keys. Contributed by Sreekanth Ramakrishnan.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk #89 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/89/)
          . Fix a NullPointerException in processing deprecated keys. Contributed by Sreekanth Ramakrishnan.

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk #89 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/89/ ) . Fix a NullPointerException in processing deprecated keys. Contributed by Sreekanth Ramakrishnan.
          Hemanth Yamijala made changes -
          Resolution Fixed [ 1 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          Hide
          Hemanth Yamijala added a comment -

          Leaving this open until I commit the jars to the affected projects.

          Show
          Hemanth Yamijala added a comment - Leaving this open until I commit the jars to the affected projects.
          Hemanth Yamijala made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Resolution Fixed [ 1 ]
          Hide
          Hemanth Yamijala added a comment -

          I committed this to trunk. Thanks, Sreekanth !

          Show
          Hemanth Yamijala added a comment - I committed this to trunk. Thanks, Sreekanth !
          Hide
          Hemanth Yamijala added a comment -

          This seems fine to me. +1.

          After, offline discussion with Hemanth and Chaitanya with regards to correct fix, a new JIRA would be opened in order to discuss how best to expose the usage of HADOOP-6105 to consumers

          Opened HADOOP-6245 for tracking this.

          Show
          Hemanth Yamijala added a comment - This seems fine to me. +1. After, offline discussion with Hemanth and Chaitanya with regards to correct fix, a new JIRA would be opened in order to discuss how best to expose the usage of HADOOP-6105 to consumers Opened HADOOP-6245 for tracking this.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12418892/HADOOP-6243-1.patch
          against trunk revision 812317.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/24/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/24/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/24/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/24/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12418892/HADOOP-6243-1.patch against trunk revision 812317. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/24/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/24/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/24/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/24/console This message is automatically generated.
          Sreekanth Ramakrishnan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Sreekanth Ramakrishnan made changes -
          Attachment HADOOP-6243-1.patch [ 12418892 ]
          Hide
          Sreekanth Ramakrishnan added a comment -

          The patch fixes the NullPointerException which was thrown when Configuration.get() method is invoked.

          The Configuration.processDeprecation() used to load the classes which were present as part of hadoop.conf.extra.classes key.

          If one of those classes have a static block which does an Configuration.addResource(), the property map maintained by the Configuration is set to null causing the NullPointerException

          After, offline discussion with Hemanth and Chaitanya with regards to correct fix, a new JIRA would be opened in order to discuss how best to expose the usage of HADOOP-6105 to consumers, and this JIRA would provide a short term fix in which consumers are expected to change the static block in Configuration class for using HADOOP-6105

          Show
          Sreekanth Ramakrishnan added a comment - The patch fixes the NullPointerException which was thrown when Configuration.get() method is invoked. The Configuration.processDeprecation() used to load the classes which were present as part of hadoop.conf.extra.classes key. If one of those classes have a static block which does an Configuration.addResource() , the property map maintained by the Configuration is set to null causing the NullPointerException After, offline discussion with Hemanth and Chaitanya with regards to correct fix, a new JIRA would be opened in order to discuss how best to expose the usage of HADOOP-6105 to consumers, and this JIRA would provide a short term fix in which consumers are expected to change the static block in Configuration class for using HADOOP-6105
          Hemanth Yamijala made changes -
          Priority Major [ 3 ] Blocker [ 1 ]
          Hide
          Hemanth Yamijala added a comment -

          I am marking this a blocker, because it is breaking builds. It was not my intent that the common jar containing HADOOP-6105 is committed to the M/R and HDFS projects. We were still running tests with Hudson jars. However, it appears that commit of HADOOP-6133 inadvertently included this fix as well.

          Show
          Hemanth Yamijala added a comment - I am marking this a blocker, because it is breaking builds. It was not my intent that the common jar containing HADOOP-6105 is committed to the M/R and HDFS projects. We were still running tests with Hudson jars. However, it appears that commit of HADOOP-6133 inadvertently included this fix as well.
          Hemanth Yamijala made changes -
          Assignee V.V.Chaitanya Krishna [ chaitk ]
          Hide
          Hemanth Yamijala added a comment -

          Chaitanya, Can you please take a look ? Please note Konstantin's helpful clues about difference in environment.

          Show
          Hemanth Yamijala added a comment - Chaitanya, Can you please take a look ? Please note Konstantin's helpful clues about difference in environment.
          Hide
          Konstantin Shvachko added a comment -

          Running the same test using ant command line works fine. Don't know what the difference is.

          Show
          Konstantin Shvachko added a comment - Running the same test using ant command line works fine. Don't know what the difference is.
          Konstantin Shvachko made changes -
          Field Original Value New Value
          Link This issue is related to HADOOP-6105 [ HADOOP-6105 ]
          Konstantin Shvachko created issue -

            People

            • Assignee:
              Sreekanth Ramakrishnan
              Reporter:
              Konstantin Shvachko
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development