Details

    • Type: Bug Bug
    • Status: Patch Available
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: 0.20.1, 0.21.0
    • Fix Version/s: 0.20.1
    • Component/s: io
    • Labels:
      None
    • Environment:

      Hadoop 0.20.1, Centos

    • Release Note:
      Found a NullPointerException with MapWritable. The root cause is the counter newClasses in AbstractMapWritable is allowed to get out of sync with the id to class mapping tables when addToMap(Class) is called.

      Description

      We have encountered a NullPointerException when we use MapWritable with custom Writable objects.

      The root cause is the counter newClasses in AbstractMapWritable is allowed to get out of sync with the id to class mapping tables when addToMap(Class) is called. We have a patch to AbstractMapWritable.addToMap(Class) that handles the case when newClasses gets out of sync with the id to class mapping tables and adds a serialization optimization to minimize the number of class names to write out per MapWritable object.

      1. HADOOP-7021.patch
        3 kB
        tom pierce
      2. HADOOP-7021.patch
        3 kB
        John Lee
      3. HADOOP-7021.patch
        3 kB
        John Lee
      4. HADOOP-7021.patch
        3 kB
        John Lee

        Activity

        Hide
        John Lee added a comment -

        Attached are a patch to AbstractMapWritable and unit test that demonstrates the NullPointerException for current version of MapWritable.

        Show
        John Lee added a comment - Attached are a patch to AbstractMapWritable and unit test that demonstrates the NullPointerException for current version of MapWritable.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12458910/AbstractMapWritable.java
        against trunk revision 1031422.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 patch. The patch command could not apply the patch.

        Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/57//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12458910/AbstractMapWritable.java against trunk revision 1031422. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/57//console This message is automatically generated.
        Hide
        John Lee added a comment -

        Could anyone help me out with properly submitting a patch and use Hudson? I have attached 3 files:

        • AbstractMapWritable.diff which is the output of svn diff.
        • AbstractMapWritable.java which is the full source with the patch.
        • TestMapWritableNPE.java which is a unit test that demonstrates when the NullPointerException is thrown.

        Thanks!

        Show
        John Lee added a comment - Could anyone help me out with properly submitting a patch and use Hudson? I have attached 3 files: AbstractMapWritable.diff which is the output of svn diff. AbstractMapWritable.java which is the full source with the patch. TestMapWritableNPE.java which is a unit test that demonstrates when the NullPointerException is thrown. Thanks!
        Hide
        tom pierce added a comment -

        I tidied the new test a bit, putting it into the current TestMapWritable class and using UTF8 as the "foreign class" instead of a one-off example Writable.

        Hopefully Hudson will find this patch more digestible; I followed the submission guidelines except for building the tar, which proved cumbersome on CentOS.

        Show
        tom pierce added a comment - I tidied the new test a bit, putting it into the current TestMapWritable class and using UTF8 as the "foreign class" instead of a one-off example Writable. Hopefully Hudson will find this patch more digestible; I followed the submission guidelines except for building the tar, which proved cumbersome on CentOS.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12466768/HADOOP-7021.patch
        against trunk revision 1051659.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        -1 javac. The applied patch generated 1050 javac compiler warnings (more than the trunk's current 1048 warnings).

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/146//testReport/
        Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/146//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/146//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12466768/HADOOP-7021.patch against trunk revision 1051659. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 1050 javac compiler warnings (more than the trunk's current 1048 warnings). +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/146//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/146//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/146//console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12466768/HADOOP-7021.patch
        against trunk revision 1071364.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        -1 javac. The applied patch generated 1074 javac compiler warnings (more than the trunk's current 1072 warnings).

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/253//testReport/
        Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/253//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/253//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12466768/HADOOP-7021.patch against trunk revision 1071364. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 1074 javac compiler warnings (more than the trunk's current 1072 warnings). +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/253//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/253//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/253//console This message is automatically generated.
        Hide
        Uma Maheswara Rao G added a comment -

        Hi Jhon,

        Your patch contains hardcore tabs, can you remove them?
        Also configure your IDE with correct coding formatter rules. Hadoop indentation is two spaces.

        Show
        Uma Maheswara Rao G added a comment - Hi Jhon, Your patch contains hardcore tabs, can you remove them? Also configure your IDE with correct coding formatter rules. Hadoop indentation is two spaces.
        Hide
        John Lee added a comment -

        Thanks Uma for checking the patch. Following you comments I have removed the tabs.

        Show
        John Lee added a comment - Thanks Uma for checking the patch. Following you comments I have removed the tabs.
        Hide
        John Lee added a comment -

        Updated patch using the the latest hadoop-common trunk. A code review is requested for this patch.

        Show
        John Lee added a comment - Updated patch using the the latest hadoop-common trunk. A code review is requested for this patch.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12490322/HADOOP-7021.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these core unit tests:

        +1 contrib tests. The patch passed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/37//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/37//artifact/trunk/target/newPatchFindbugsWarningshadoop-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/37//artifact/trunk/target/newPatchFindbugsWarningshadoop-annotations.html
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/37//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12490322/HADOOP-7021.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/37//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/37//artifact/trunk/target/newPatchFindbugsWarningshadoop-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/37//artifact/trunk/target/newPatchFindbugsWarningshadoop-annotations.html Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/37//console This message is automatically generated.
        Hide
        John Lee added a comment -

        Does anyone know how to find out which tests did this patch fail?

        -1 core tests. The patch failed these core unit tests:

        Thanks!

        Show
        John Lee added a comment - Does anyone know how to find out which tests did this patch fail? -1 core tests. The patch failed these core unit tests: Thanks!
        Hide
        John Lee added a comment -

        Ran dev-support/test-patch.sh using the patch: http://issues.apache.org/jira/secure/attachment/12490322/HADOOP-7021.patch on a fresh svn checkout of hadoop-trunk and this is my output:

        +1 overall.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version ) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 system test framework. The patch passed system test framework compile.

        ======================================================================
        ======================================================================
        Finished build.
        ======================================================================
        ======================================================================

        Show
        John Lee added a comment - Ran dev-support/test-patch.sh using the patch: http://issues.apache.org/jira/secure/attachment/12490322/HADOOP-7021.patch on a fresh svn checkout of hadoop-trunk and this is my output: +1 overall. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version ) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 system test framework. The patch passed system test framework compile. ====================================================================== ====================================================================== Finished build. ====================================================================== ======================================================================
        Hide
        John Lee added a comment -

        I accidentally changed the status of this issue to "In Progress" and I can not change it back to "Patch Available" could someone assist me with this?

        Also, I have noticed that this issue was submitted more than a year ago (with code to fix the issue), since then I have updated the patch and test code for formatting and to be compatible with the update build layout. Is there anything else that need to done to get this patch review and committed? Thanks.

        Show
        John Lee added a comment - I accidentally changed the status of this issue to "In Progress" and I can not change it back to "Patch Available" could someone assist me with this? Also, I have noticed that this issue was submitted more than a year ago (with code to fix the issue), since then I have updated the patch and test code for formatting and to be compatible with the update build layout. Is there anything else that need to done to get this patch review and committed? Thanks.
        Hide
        Doug Cutting added a comment -

        I assigned it to you. Can you now transition it back to Patch Available?

        The best way to get some attention is to send a message to the dev mailing list, politely asking for someone to please look at this. I will try to have a look at it in a few days if I have a chance.

        Show
        Doug Cutting added a comment - I assigned it to you. Can you now transition it back to Patch Available? The best way to get some attention is to send a message to the dev mailing list, politely asking for someone to please look at this. I will try to have a look at it in a few days if I have a chance.
        Hide
        John Lee added a comment -

        Updated patch to reflect mvn directory structure.

        Show
        John Lee added a comment - Updated patch to reflect mvn directory structure.
        Hide
        John Lee added a comment -

        Thanks for your help Doug.

        Show
        John Lee added a comment - Thanks for your help Doug.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12503833/HADOOP-7021.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        -1 javadoc. The javadoc tool appears to have generated 15 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 7 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/378//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/378//artifact/trunk/hadoop-common-project/patchprocess/newPatchFindbugsWarningshadoop-common.html
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/378//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12503833/HADOOP-7021.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 15 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 7 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/378//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/378//artifact/trunk/hadoop-common-project/patchprocess/newPatchFindbugsWarningshadoop-common.html Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/378//console This message is automatically generated.
        Hide
        John Lee added a comment -

        Received -1 overall but that was due to warnings in findbugs and javadocs which are unrelated to the code introduced by the patch. Resubmitting for review.

        Show
        John Lee added a comment - Received -1 overall but that was due to warnings in findbugs and javadocs which are unrelated to the code introduced by the patch. Resubmitting for review.
        Hide
        tom pierce added a comment -

        Any reason this hasn't made it in yet? Seems like every time John manages to get Hudson to look at this patch, there's some unrelated (or at least not locally reproducable) problem that causes a -1...

        Show
        tom pierce added a comment - Any reason this hasn't made it in yet? Seems like every time John manages to get Hudson to look at this patch, there's some unrelated (or at least not locally reproducable) problem that causes a -1...

          People

          • Assignee:
            John Lee
            Reporter:
            John Lee
          • Votes:
            1 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:

              Development