Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6595

Fix findbugs warnings in OutputCommitter and FileOutputCommitter

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    1. MAPREDUCE-6595.testing.patch
      2 kB
      Akira Ajisaka
    2. MAPREDUCE-6595.01.patch
      0.8 kB
      Akira Ajisaka
    3. findbugsHtml.html
      8 kB
      Akira Ajisaka

      Issue Links

        Activity

        Hide
        ajisakaa Akira Ajisaka added a comment -

        Attaching a html file.

        Show
        ajisakaa Akira Ajisaka added a comment - Attaching a html file.
        Hide
        ajisakaa Akira Ajisaka added a comment - - edited

        I'm thinking it's okay to add sections in findbugs-exclude.xml to ignore the warnings because this seems to be intentional and the warnings are unavoidable for keeping compatibility.

        Show
        ajisakaa Akira Ajisaka added a comment - - edited I'm thinking it's okay to add sections in findbugs-exclude.xml to ignore the warnings because this seems to be intentional and the warnings are unavoidable for keeping compatibility.
        Hide
        ajisakaa Akira Ajisaka added a comment -

        Attaching a patch to ignore the warnings.

        Show
        ajisakaa Akira Ajisaka added a comment - Attaching a patch to ignore the warnings.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 findbugs 5m 24s branch/hadoop-mapreduce-project no findbugs output file (hadoop-mapreduce-project/target/findbugsXml.xml)
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        -1 findbugs 5m 15s patch/hadoop-mapreduce-project no findbugs output file (hadoop-mapreduce-project/target/findbugsXml.xml)
        +1 asflicense 0m 26s Patch does not generate ASF License warnings.
        11m 26s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12780278/MAPREDUCE-6595.01.patch
        JIRA Issue MAPREDUCE-6595
        Optional Tests asflicense findbugs xml
        uname Linux d6e304ef4d21 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 7dafee1
        modules C: hadoop-mapreduce-project U: hadoop-mapreduce-project
        Max memory used 31MB
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6249/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 findbugs 5m 24s branch/hadoop-mapreduce-project no findbugs output file (hadoop-mapreduce-project/target/findbugsXml.xml) +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. -1 findbugs 5m 15s patch/hadoop-mapreduce-project no findbugs output file (hadoop-mapreduce-project/target/findbugsXml.xml) +1 asflicense 0m 26s Patch does not generate ASF License warnings. 11m 26s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12780278/MAPREDUCE-6595.01.patch JIRA Issue MAPREDUCE-6595 Optional Tests asflicense findbugs xml uname Linux d6e304ef4d21 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 7dafee1 modules C: hadoop-mapreduce-project U: hadoop-mapreduce-project Max memory used 31MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6249/console This message was automatically generated.
        Hide
        ajisakaa Akira Ajisaka added a comment -

        -1 findbugs

        I'm thinking this is because of the mismatch of the modules. The changed file (findbugs-exclude.xml) is in hadoop-mapreduce-project module, however, the warnings are in hadoop-mapreduce-client-core module. I'll attach a sample patch shortly.

        Show
        ajisakaa Akira Ajisaka added a comment - -1 findbugs I'm thinking this is because of the mismatch of the modules. The changed file (findbugs-exclude.xml) is in hadoop-mapreduce-project module, however, the warnings are in hadoop-mapreduce-client-core module. I'll attach a sample patch shortly.
        Hide
        ajisakaa Akira Ajisaka added a comment -

        Attaching a sample patch that modifies hadoop-mapreduce-client-core module.

        Show
        ajisakaa Akira Ajisaka added a comment - Attaching a sample patch that modifies hadoop-mapreduce-client-core module.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 7m 43s trunk passed
        +1 compile 3m 3s trunk passed with JDK v1.8.0_66
        +1 compile 3m 9s trunk passed with JDK v1.7.0_91
        +1 checkstyle 0m 20s trunk passed
        +1 mvnsite 1m 53s trunk passed
        +1 mvneclipse 0m 19s trunk passed
        -1 findbugs 3m 35s branch/hadoop-mapreduce-project no findbugs output file (hadoop-mapreduce-project/target/findbugsXml.xml)
        +1 javadoc 0m 52s trunk passed with JDK v1.8.0_66
        +1 javadoc 1m 8s trunk passed with JDK v1.7.0_91
        +1 mvninstall 1m 24s the patch passed
        +1 compile 2m 55s the patch passed with JDK v1.8.0_66
        +1 javac 2m 55s the patch passed
        +1 compile 3m 8s the patch passed with JDK v1.7.0_91
        +1 javac 3m 8s the patch passed
        +1 checkstyle 0m 19s the patch passed
        +1 mvnsite 1m 51s the patch passed
        +1 mvneclipse 0m 17s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 xml 0m 0s The patch has no ill-formed XML file.
        -1 findbugs 3m 34s patch/hadoop-mapreduce-project no findbugs output file (hadoop-mapreduce-project/target/findbugsXml.xml)
        +1 javadoc 0m 53s the patch passed with JDK v1.8.0_66
        +1 javadoc 1m 7s the patch passed with JDK v1.7.0_91
        -1 unit 122m 31s hadoop-mapreduce-project in the patch failed with JDK v1.8.0_66.
        -1 unit 132m 0s hadoop-mapreduce-project in the patch failed with JDK v1.7.0_91.
        -1 asflicense 0m 35s Patch generated 33 ASF License warnings.
        293m 31s



        Reason Tests
        JDK v1.8.0_66 Failed junit tests hadoop.mapreduce.v2.TestMRJobsWithProfiler
          hadoop.mapred.TestNetworkedJob
        JDK v1.7.0_91 Failed junit tests hadoop.mapred.TestNetworkedJob



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12780297/MAPREDUCE-6595.testing.patch
        JIRA Issue MAPREDUCE-6595
        Optional Tests asflicense findbugs xml compile javac javadoc mvninstall mvnsite unit checkstyle
        uname Linux 40e7aef46795 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 7dafee1
        Default Java 1.7.0_91
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
        unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6250/artifact/patchprocess/patch-unit-hadoop-mapreduce-project-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6250/artifact/patchprocess/patch-unit-hadoop-mapreduce-project-jdk1.7.0_91.txt
        unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6250/artifact/patchprocess/patch-unit-hadoop-mapreduce-project-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6250/artifact/patchprocess/patch-unit-hadoop-mapreduce-project-jdk1.7.0_91.txt
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6250/testReport/
        asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6250/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-mapreduce-project hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project
        Max memory used 75MB
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6250/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 43s trunk passed +1 compile 3m 3s trunk passed with JDK v1.8.0_66 +1 compile 3m 9s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 20s trunk passed +1 mvnsite 1m 53s trunk passed +1 mvneclipse 0m 19s trunk passed -1 findbugs 3m 35s branch/hadoop-mapreduce-project no findbugs output file (hadoop-mapreduce-project/target/findbugsXml.xml) +1 javadoc 0m 52s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 8s trunk passed with JDK v1.7.0_91 +1 mvninstall 1m 24s the patch passed +1 compile 2m 55s the patch passed with JDK v1.8.0_66 +1 javac 2m 55s the patch passed +1 compile 3m 8s the patch passed with JDK v1.7.0_91 +1 javac 3m 8s the patch passed +1 checkstyle 0m 19s the patch passed +1 mvnsite 1m 51s the patch passed +1 mvneclipse 0m 17s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 0s The patch has no ill-formed XML file. -1 findbugs 3m 34s patch/hadoop-mapreduce-project no findbugs output file (hadoop-mapreduce-project/target/findbugsXml.xml) +1 javadoc 0m 53s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 7s the patch passed with JDK v1.7.0_91 -1 unit 122m 31s hadoop-mapreduce-project in the patch failed with JDK v1.8.0_66. -1 unit 132m 0s hadoop-mapreduce-project in the patch failed with JDK v1.7.0_91. -1 asflicense 0m 35s Patch generated 33 ASF License warnings. 293m 31s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.mapreduce.v2.TestMRJobsWithProfiler   hadoop.mapred.TestNetworkedJob JDK v1.7.0_91 Failed junit tests hadoop.mapred.TestNetworkedJob Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12780297/MAPREDUCE-6595.testing.patch JIRA Issue MAPREDUCE-6595 Optional Tests asflicense findbugs xml compile javac javadoc mvninstall mvnsite unit checkstyle uname Linux 40e7aef46795 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 7dafee1 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6250/artifact/patchprocess/patch-unit-hadoop-mapreduce-project-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6250/artifact/patchprocess/patch-unit-hadoop-mapreduce-project-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6250/artifact/patchprocess/patch-unit-hadoop-mapreduce-project-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6250/artifact/patchprocess/patch-unit-hadoop-mapreduce-project-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6250/testReport/ asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6250/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-mapreduce-project hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project Max memory used 75MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6250/console This message was automatically generated.
        Hide
        ajisakaa Akira Ajisaka added a comment -

        -1 findbugs

        Still seeing -1. I'm thinking this is because there are no source code in hadoop-mapreduce-project module and that's why mvn findbugs:findbugs doesn't output findbugsXml.xml. I'll file a jira in Yetus project.

        Show
        ajisakaa Akira Ajisaka added a comment - -1 findbugs Still seeing -1. I'm thinking this is because there are no source code in hadoop-mapreduce-project module and that's why mvn findbugs:findbugs doesn't output findbugsXml.xml. I'll file a jira in Yetus project.
        Hide
        ajisakaa Akira Ajisaka added a comment -

        Filed YETUS-271.

        Show
        ajisakaa Akira Ajisaka added a comment - Filed YETUS-271 .
        Hide
        ajisakaa Akira Ajisaka added a comment -

        Rethinking this, even if YETUS-271 is fixed, findbugs will not run on hadoop-mapreduce-client-core module. To fix the mismatch of module, I propose to split findbugs-exclude.xml into submodules. I'll file a separate jira.

        Show
        ajisakaa Akira Ajisaka added a comment - Rethinking this, even if YETUS-271 is fixed, findbugs will not run on hadoop-mapreduce-client-core module. To fix the mismatch of module, I propose to split findbugs-exclude.xml into submodules. I'll file a separate jira.
        Hide
        ajisakaa Akira Ajisaka added a comment -

        Ran test-patch.sh for MAPREDUCE-6595.testing.patch with YETUS-271.03.patch.

        | Vote |      Subsystem |  Runtime   | Comment
        ============================================================================
        |  +1  |       @author  |  0m 00s    | The patch does not contain any @author 
        |      |                |            | tags.
        |  -1  |    test4tests  |  0m 00s    | The patch doesn't appear to include any 
        |      |                |            | new or modified tests. Please justify why
        |      |                |            | no new tests are needed for this patch.
        |      |                |            | Also please list what manual steps were
        |      |                |            | performed to verify this patch.
        |   0  |        mvndep  |  1m 41s    | Maven dependency ordering for branch 
        |  +1  |    mvninstall  |  7m 38s    | trunk passed 
        |  -1  |       compile  |  2m 17s    | hadoop-mapreduce-project in trunk 
        |      |                |            | failed.
        |  +1  |    checkstyle  |  0m 22s    | trunk passed 
        |  +1  |       mvnsite  |  3m 31s    | trunk passed 
        |  +1  |    mvneclipse  |  1m 32s    | trunk passed 
        |   0  |      findbugs  |  0m 00s    | Skipped branch modules with no Java 
        |      |                |            | source: hadoop-mapreduce-project
        |   0  |      findbugs  |  1m 38s    | 
        |      |                |            | hadoop-mapreduce-project/hadoop-mapreduce-
        |      |                |            | client/hadoop-mapreduce-client-core in
        |      |                |            | trunk has 2 extant Findbugs warnings.
        |  +1  |       javadoc  |  2m 31s    | trunk passed 
        |   0  |        mvndep  |  0m 41s    | Maven dependency ordering for patch 
        |  +1  |    mvninstall  |  2m 25s    | the patch passed 
        |  -1  |       compile  |  1m 37s    | hadoop-mapreduce-project in the patch 
        |      |                |            | failed.
        |  -1  |         javac  |  1m 37s    | hadoop-mapreduce-project in the patch 
        |      |                |            | failed.
        |  +1  |    checkstyle  |  0m 25s    | the patch passed 
        |  +1  |       mvnsite  |  2m 54s    | the patch passed 
        |  +1  |    mvneclipse  |  1m 04s    | the patch passed 
        |  +1  |    whitespace  |  0m 00s    | Patch has no whitespace issues. 
        |  +1  |           xml  |  0m 01s    | The patch has no ill-formed XML file. 
        |   0  |      findbugs  |  0m 00s    | Skipped patch modules with no Java 
        |      |                |            | source: hadoop-mapreduce-project
        |  +1  |      findbugs  |  1m 32s    | the patch passed 
        |  +1  |       javadoc  |  2m 05s    | the patch passed 
        |  -1  |    asflicense  |  0m 38s    | Patch generated 9 ASF License warnings. 
        |      |                |  36m 13s   | 
        

        The output shows that the 2 findbugs warnings are fixed. (actually ignored)

        Show
        ajisakaa Akira Ajisaka added a comment - Ran test-patch.sh for MAPREDUCE-6595 .testing.patch with YETUS-271 .03.patch. | Vote | Subsystem | Runtime | Comment ============================================================================ | +1 | @author | 0m 00s | The patch does not contain any @author | | | | tags. | -1 | test4tests | 0m 00s | The patch doesn't appear to include any | | | | new or modified tests. Please justify why | | | | no new tests are needed for this patch. | | | | Also please list what manual steps were | | | | performed to verify this patch. | 0 | mvndep | 1m 41s | Maven dependency ordering for branch | +1 | mvninstall | 7m 38s | trunk passed | -1 | compile | 2m 17s | hadoop-mapreduce-project in trunk | | | | failed. | +1 | checkstyle | 0m 22s | trunk passed | +1 | mvnsite | 3m 31s | trunk passed | +1 | mvneclipse | 1m 32s | trunk passed | 0 | findbugs | 0m 00s | Skipped branch modules with no Java | | | | source: hadoop-mapreduce-project | 0 | findbugs | 1m 38s | | | | | hadoop-mapreduce-project/hadoop-mapreduce- | | | | client/hadoop-mapreduce-client-core in | | | | trunk has 2 extant Findbugs warnings. | +1 | javadoc | 2m 31s | trunk passed | 0 | mvndep | 0m 41s | Maven dependency ordering for patch | +1 | mvninstall | 2m 25s | the patch passed | -1 | compile | 1m 37s | hadoop-mapreduce-project in the patch | | | | failed. | -1 | javac | 1m 37s | hadoop-mapreduce-project in the patch | | | | failed. | +1 | checkstyle | 0m 25s | the patch passed | +1 | mvnsite | 2m 54s | the patch passed | +1 | mvneclipse | 1m 04s | the patch passed | +1 | whitespace | 0m 00s | Patch has no whitespace issues. | +1 | xml | 0m 01s | The patch has no ill-formed XML file. | 0 | findbugs | 0m 00s | Skipped patch modules with no Java | | | | source: hadoop-mapreduce-project | +1 | findbugs | 1m 32s | the patch passed | +1 | javadoc | 2m 05s | the patch passed | -1 | asflicense | 0m 38s | Patch generated 9 ASF License warnings. | | | 36m 13s | The output shows that the 2 findbugs warnings are fixed. (actually ignored)
        Hide
        ajisakaa Akira Ajisaka added a comment -
        Show
        ajisakaa Akira Ajisaka added a comment - Filed MAPREDUCE-6600 .
        Hide
        djp Junping Du added a comment -

        Thanks Akira Ajisaka for reporting the issue and fix it. +1. Will commit it shortly.

        Show
        djp Junping Du added a comment - Thanks Akira Ajisaka for reporting the issue and fix it. +1. Will commit it shortly.
        Hide
        djp Junping Du added a comment -

        I have commit the patch to trunk, branch-2 and branch-2.8. Thanks Akira Ajisaka for the contribution.

        Show
        djp Junping Du added a comment - I have commit the patch to trunk, branch-2 and branch-2.8. Thanks Akira Ajisaka for the contribution.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-trunk-Commit #9199 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9199/)
        MAPREDUCE-6595. Fix findbugs warnings in OutputCommitter and (junping_du: rev 86560a4978c498bd5e892c509a466807a812b8a7)

        • hadoop-mapreduce-project/dev-support/findbugs-exclude.xml
        • hadoop-mapreduce-project/CHANGES.txt
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #9199 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9199/ ) MAPREDUCE-6595 . Fix findbugs warnings in OutputCommitter and (junping_du: rev 86560a4978c498bd5e892c509a466807a812b8a7) hadoop-mapreduce-project/dev-support/findbugs-exclude.xml hadoop-mapreduce-project/CHANGES.txt
        Hide
        ajisakaa Akira Ajisaka added a comment -

        Thanks Junping Du!

        Show
        ajisakaa Akira Ajisaka added a comment - Thanks Junping Du !

          People

          • Assignee:
            ajisakaa Akira Ajisaka
            Reporter:
            ajisakaa Akira Ajisaka
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development