Details

    • Type: Bug Bug
    • Status: Patch Available
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: 0.20.1, 0.21.0
    • Fix Version/s: 0.20.1
    • Component/s: io
    • Labels:
      None
    • Environment:

      Hadoop 0.20.1, Centos

    • Release Note:
      Found a NullPointerException with MapWritable. The root cause is the counter newClasses in AbstractMapWritable is allowed to get out of sync with the id to class mapping tables when addToMap(Class) is called.

      Description

      We have encountered a NullPointerException when we use MapWritable with custom Writable objects.

      The root cause is the counter newClasses in AbstractMapWritable is allowed to get out of sync with the id to class mapping tables when addToMap(Class) is called. We have a patch to AbstractMapWritable.addToMap(Class) that handles the case when newClasses gets out of sync with the id to class mapping tables and adds a serialization optimization to minimize the number of class names to write out per MapWritable object.

      1. HADOOP-7021.patch
        3 kB
        John Lee
      2. HADOOP-7021.patch
        3 kB
        John Lee
      3. HADOOP-7021.patch
        3 kB
        John Lee
      4. HADOOP-7021.patch
        3 kB
        tom pierce

        Activity

        John Lee created issue -
        Hide
        John Lee added a comment -

        Attached are a patch to AbstractMapWritable and unit test that demonstrates the NullPointerException for current version of MapWritable.

        Show
        John Lee added a comment - Attached are a patch to AbstractMapWritable and unit test that demonstrates the NullPointerException for current version of MapWritable.
        John Lee made changes -
        Field Original Value New Value
        Attachment TestMapWritableNPE.java [ 12458908 ]
        Attachment AbstractMapWritable.diff [ 12458909 ]
        Attachment AbstractMapWritable.java [ 12458910 ]
        John Lee made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Release Note Found a NullPointerException with MapWritable. The root cause is the counter newClasses in AbstractMapWritable is allowed to get out of sync with the id to class mapping tables when addToMap(Class) is called.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12458910/AbstractMapWritable.java
        against trunk revision 1031422.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 patch. The patch command could not apply the patch.

        Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/57//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12458910/AbstractMapWritable.java against trunk revision 1031422. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/57//console This message is automatically generated.
        Hide
        John Lee added a comment -

        Could anyone help me out with properly submitting a patch and use Hudson? I have attached 3 files:

        • AbstractMapWritable.diff which is the output of svn diff.
        • AbstractMapWritable.java which is the full source with the patch.
        • TestMapWritableNPE.java which is a unit test that demonstrates when the NullPointerException is thrown.

        Thanks!

        Show
        John Lee added a comment - Could anyone help me out with properly submitting a patch and use Hudson? I have attached 3 files: AbstractMapWritable.diff which is the output of svn diff. AbstractMapWritable.java which is the full source with the patch. TestMapWritableNPE.java which is a unit test that demonstrates when the NullPointerException is thrown. Thanks!
        Hide
        tom pierce added a comment -

        I tidied the new test a bit, putting it into the current TestMapWritable class and using UTF8 as the "foreign class" instead of a one-off example Writable.

        Hopefully Hudson will find this patch more digestible; I followed the submission guidelines except for building the tar, which proved cumbersome on CentOS.

        Show
        tom pierce added a comment - I tidied the new test a bit, putting it into the current TestMapWritable class and using UTF8 as the "foreign class" instead of a one-off example Writable. Hopefully Hudson will find this patch more digestible; I followed the submission guidelines except for building the tar, which proved cumbersome on CentOS.
        tom pierce made changes -
        Attachment HADOOP-7021.patch [ 12466768 ]
        tom pierce made changes -
        Affects Version/s 0.21.0 [ 12313563 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12466768/HADOOP-7021.patch
        against trunk revision 1051659.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        -1 javac. The applied patch generated 1050 javac compiler warnings (more than the trunk's current 1048 warnings).

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/146//testReport/
        Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/146//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/146//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12466768/HADOOP-7021.patch against trunk revision 1051659. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 1050 javac compiler warnings (more than the trunk's current 1048 warnings). +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/146//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/146//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/146//console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12466768/HADOOP-7021.patch
        against trunk revision 1071364.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        -1 javac. The applied patch generated 1074 javac compiler warnings (more than the trunk's current 1072 warnings).

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/253//testReport/
        Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/253//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/253//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12466768/HADOOP-7021.patch against trunk revision 1071364. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 1074 javac compiler warnings (more than the trunk's current 1072 warnings). +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/253//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/253//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/253//console This message is automatically generated.
        John Lee made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        John Lee made changes -
        Attachment HADOOP-7021.patch [ 12488471 ]
        John Lee made changes -
        Attachment HADOOP-7021.patch [ 12488854 ]
        John Lee made changes -
        Attachment AbstractMapWritable.diff [ 12458909 ]
        John Lee made changes -
        Attachment AbstractMapWritable.java [ 12458910 ]
        John Lee made changes -
        Attachment TestMapWritableNPE.java [ 12458908 ]
        John Lee made changes -
        Attachment HADOOP-7021.patch [ 12488471 ]
        John Lee made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Uma Maheswara Rao G added a comment -

        Hi Jhon,

        Your patch contains hardcore tabs, can you remove them?
        Also configure your IDE with correct coding formatter rules. Hadoop indentation is two spaces.

        Show
        Uma Maheswara Rao G added a comment - Hi Jhon, Your patch contains hardcore tabs, can you remove them? Also configure your IDE with correct coding formatter rules. Hadoop indentation is two spaces.
        John Lee made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        John Lee made changes -
        Attachment HADOOP-7021.patch [ 12488854 ]
        John Lee made changes -
        Attachment HADOOP-7021.patch [ 12488875 ]
        John Lee made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        John Lee added a comment -

        Thanks Uma for checking the patch. Following you comments I have removed the tabs.

        Show
        John Lee added a comment - Thanks Uma for checking the patch. Following you comments I have removed the tabs.
        Hide
        John Lee added a comment -

        Updated patch using the the latest hadoop-common trunk. A code review is requested for this patch.

        Show
        John Lee added a comment - Updated patch using the the latest hadoop-common trunk. A code review is requested for this patch.
        John Lee made changes -
        Attachment HADOOP-7021.patch [ 12490322 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12490322/HADOOP-7021.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these core unit tests:

        +1 contrib tests. The patch passed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/37//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/37//artifact/trunk/target/newPatchFindbugsWarningshadoop-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/37//artifact/trunk/target/newPatchFindbugsWarningshadoop-annotations.html
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/37//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12490322/HADOOP-7021.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/37//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/37//artifact/trunk/target/newPatchFindbugsWarningshadoop-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/37//artifact/trunk/target/newPatchFindbugsWarningshadoop-annotations.html Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/37//console This message is automatically generated.
        Hide
        John Lee added a comment -

        Does anyone know how to find out which tests did this patch fail?

        -1 core tests. The patch failed these core unit tests:

        Thanks!

        Show
        John Lee added a comment - Does anyone know how to find out which tests did this patch fail? -1 core tests. The patch failed these core unit tests: Thanks!
        Hide
        John Lee added a comment -

        Ran dev-support/test-patch.sh using the patch: http://issues.apache.org/jira/secure/attachment/12490322/HADOOP-7021.patch on a fresh svn checkout of hadoop-trunk and this is my output:

        +1 overall.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version ) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 system test framework. The patch passed system test framework compile.

        ======================================================================
        ======================================================================
        Finished build.
        ======================================================================
        ======================================================================

        Show
        John Lee added a comment - Ran dev-support/test-patch.sh using the patch: http://issues.apache.org/jira/secure/attachment/12490322/HADOOP-7021.patch on a fresh svn checkout of hadoop-trunk and this is my output: +1 overall. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version ) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 system test framework. The patch passed system test framework compile. ====================================================================== ====================================================================== Finished build. ====================================================================== ======================================================================
        John Lee made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        John Lee made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        John Lee made changes -
        Status Patch Available [ 10002 ] In Progress [ 3 ]
        Hide
        John Lee added a comment -

        I accidentally changed the status of this issue to "In Progress" and I can not change it back to "Patch Available" could someone assist me with this?

        Also, I have noticed that this issue was submitted more than a year ago (with code to fix the issue), since then I have updated the patch and test code for formatting and to be compatible with the update build layout. Is there anything else that need to done to get this patch review and committed? Thanks.

        Show
        John Lee added a comment - I accidentally changed the status of this issue to "In Progress" and I can not change it back to "Patch Available" could someone assist me with this? Also, I have noticed that this issue was submitted more than a year ago (with code to fix the issue), since then I have updated the patch and test code for formatting and to be compatible with the update build layout. Is there anything else that need to done to get this patch review and committed? Thanks.
        Doug Cutting made changes -
        Assignee John Lee [ jlee@mindset-media.com ]
        Hide
        Doug Cutting added a comment -

        I assigned it to you. Can you now transition it back to Patch Available?

        The best way to get some attention is to send a message to the dev mailing list, politely asking for someone to please look at this. I will try to have a look at it in a few days if I have a chance.

        Show
        Doug Cutting added a comment - I assigned it to you. Can you now transition it back to Patch Available? The best way to get some attention is to send a message to the dev mailing list, politely asking for someone to please look at this. I will try to have a look at it in a few days if I have a chance.
        John Lee made changes -
        Status In Progress [ 3 ] Open [ 1 ]
        Hide
        John Lee added a comment -

        Updated patch to reflect mvn directory structure.

        Show
        John Lee added a comment - Updated patch to reflect mvn directory structure.
        John Lee made changes -
        Attachment HADOOP-7021.patch [ 12503833 ]
        John Lee made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        John Lee added a comment -

        Thanks for your help Doug.

        Show
        John Lee added a comment - Thanks for your help Doug.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12503833/HADOOP-7021.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        -1 javadoc. The javadoc tool appears to have generated 15 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 7 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/378//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/378//artifact/trunk/hadoop-common-project/patchprocess/newPatchFindbugsWarningshadoop-common.html
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/378//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12503833/HADOOP-7021.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 15 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 7 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/378//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/378//artifact/trunk/hadoop-common-project/patchprocess/newPatchFindbugsWarningshadoop-common.html Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/378//console This message is automatically generated.
        John Lee made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Hide
        John Lee added a comment -

        Received -1 overall but that was due to warnings in findbugs and javadocs which are unrelated to the code introduced by the patch. Resubmitting for review.

        Show
        John Lee added a comment - Received -1 overall but that was due to warnings in findbugs and javadocs which are unrelated to the code introduced by the patch. Resubmitting for review.
        John Lee made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        tom pierce added a comment -

        Any reason this hasn't made it in yet? Seems like every time John manages to get Hudson to look at this patch, there's some unrelated (or at least not locally reproducable) problem that causes a -1...

        Show
        tom pierce added a comment - Any reason this hasn't made it in yet? Seems like every time John manages to get Hudson to look at this patch, there's some unrelated (or at least not locally reproducable) problem that causes a -1...

          People

          • Assignee:
            John Lee
            Reporter:
            John Lee
          • Votes:
            1 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:

              Development