Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8961

Surpress findbug warnings of o.a.h.hdfs.shortcircuit.DfsClientShmManager.EndpointShmManager in hdfs-client

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      There are two clauses in hadoop-hdfs to filter out the findbugs warnings in org.apache.hadoop.hdfs.shortcircuit.DfsClientShmManager$EndpointShmManager:

          <Match>
            <Class name="org.apache.hadoop.hdfs.shortcircuit.DfsClientShmManager$EndpointShmManager" />
            <Method name="allocSlot" />
            <Bug pattern="UL_UNRELEASED_LOCK_EXCEPTION_PATH" />
          </Match>
          <Match>
            <Class name="org.apache.hadoop.hdfs.shortcircuit.DfsClientShmManager$EndpointShmManager" />
            <Method name="allocSlot" />
            <Bug pattern="UL_UNRELEASED_LOCK" />
          </Match>
      

      These two warnings show up in the Jenkins run as these classes are moved into the hadoop-hdfs-client module. We either need to fix the code or move these clauses to the hadoop-hdfs-client module.

        Activity

        Hide
        liuml07 Mingliang Liu added a comment -

        As to the code itself, I think the two findbugs warning are false positive. While it does not follow the JSR-166 strictly, its releasing/retaking lock action is safe in DfsClientShmManager$EndpointShmManager::allocSlot() method.

        So I prefer the other solution for this issue. I will file a patch to move the findbugs filter rules from hadoop-hdfs to hadoop-hdfs-client module.

        Show
        liuml07 Mingliang Liu added a comment - As to the code itself, I think the two findbugs warning are false positive. While it does not follow the JSR-166 strictly, its releasing/retaking lock action is safe in DfsClientShmManager$EndpointShmManager::allocSlot() method. So I prefer the other solution for this issue. I will file a patch to move the findbugs filter rules from hadoop-hdfs to hadoop-hdfs-client module.
        Hide
        liuml07 Mingliang Liu added a comment -

        The v0 patch moves the findbugs filter rule from hadoop-hdfs to hadoop-hdfs-client module.

        Show
        liuml07 Mingliang Liu added a comment - The v0 patch moves the findbugs filter rule from hadoop-hdfs to hadoop-hdfs-client module.
        Hide
        hadoopqa Hadoop QA added a comment -



        +1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 0m 0s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 release audit 0m 15s The applied patch does not increase the total number of release audit warnings.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
            0m 19s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12752630/HDFS-8961.000.patch
        Optional Tests  
        git revision trunk / 4cbbfa2
        Java 1.7.0_55
        uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12152/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 0m 0s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 release audit 0m 15s The applied patch does not increase the total number of release audit warnings. +1 whitespace 0m 0s The patch has no lines that end in whitespace.     0m 19s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12752630/HDFS-8961.000.patch Optional Tests   git revision trunk / 4cbbfa2 Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12152/console This message was automatically generated.
        Hide
        wheat9 Haohui Mai added a comment -

        +1. I'll commit it shortly.

        Show
        wheat9 Haohui Mai added a comment - +1. I'll commit it shortly.
        Hide
        wheat9 Haohui Mai added a comment -

        I've committed the patch to trunk and branch-2. Thanks Mingliang Liu for the contribution.

        Show
        wheat9 Haohui Mai added a comment - I've committed the patch to trunk and branch-2. Thanks Mingliang Liu for the contribution.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #8356 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8356/)
        HDFS-8961. Investigate lock issue in o.a.h.hdfs.shortcircuit.DfsClientShmManager.EndpointShmManager. Contributed by Mingliang Liu. (wheat9: rev 1e5f69e85c035f9507e8b788df0b3ce20290a770)

        • hadoop-hdfs-project/hadoop-hdfs/dev-support/findbugsExcludeFile.xml
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs-client/dev-support/findbugsExcludeFile.xml
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8356 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8356/ ) HDFS-8961 . Investigate lock issue in o.a.h.hdfs.shortcircuit.DfsClientShmManager.EndpointShmManager. Contributed by Mingliang Liu. (wheat9: rev 1e5f69e85c035f9507e8b788df0b3ce20290a770) hadoop-hdfs-project/hadoop-hdfs/dev-support/findbugsExcludeFile.xml hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs-client/dev-support/findbugsExcludeFile.xml
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #316 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/316/)
        HDFS-8961. Investigate lock issue in o.a.h.hdfs.shortcircuit.DfsClientShmManager.EndpointShmManager. Contributed by Mingliang Liu. (wheat9: rev 1e5f69e85c035f9507e8b788df0b3ce20290a770)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/dev-support/findbugsExcludeFile.xml
        • hadoop-hdfs-project/hadoop-hdfs-client/dev-support/findbugsExcludeFile.xml
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #316 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/316/ ) HDFS-8961 . Investigate lock issue in o.a.h.hdfs.shortcircuit.DfsClientShmManager.EndpointShmManager. Contributed by Mingliang Liu. (wheat9: rev 1e5f69e85c035f9507e8b788df0b3ce20290a770) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/dev-support/findbugsExcludeFile.xml hadoop-hdfs-project/hadoop-hdfs-client/dev-support/findbugsExcludeFile.xml
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #1044 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1044/)
        HDFS-8961. Investigate lock issue in o.a.h.hdfs.shortcircuit.DfsClientShmManager.EndpointShmManager. Contributed by Mingliang Liu. (wheat9: rev 1e5f69e85c035f9507e8b788df0b3ce20290a770)

        • hadoop-hdfs-project/hadoop-hdfs/dev-support/findbugsExcludeFile.xml
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs-client/dev-support/findbugsExcludeFile.xml
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1044 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1044/ ) HDFS-8961 . Investigate lock issue in o.a.h.hdfs.shortcircuit.DfsClientShmManager.EndpointShmManager. Contributed by Mingliang Liu. (wheat9: rev 1e5f69e85c035f9507e8b788df0b3ce20290a770) hadoop-hdfs-project/hadoop-hdfs/dev-support/findbugsExcludeFile.xml hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs-client/dev-support/findbugsExcludeFile.xml
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #311 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/311/)
        HDFS-8961. Investigate lock issue in o.a.h.hdfs.shortcircuit.DfsClientShmManager.EndpointShmManager. Contributed by Mingliang Liu. (wheat9: rev 1e5f69e85c035f9507e8b788df0b3ce20290a770)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs-client/dev-support/findbugsExcludeFile.xml
        • hadoop-hdfs-project/hadoop-hdfs/dev-support/findbugsExcludeFile.xml
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #311 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/311/ ) HDFS-8961 . Investigate lock issue in o.a.h.hdfs.shortcircuit.DfsClientShmManager.EndpointShmManager. Contributed by Mingliang Liu. (wheat9: rev 1e5f69e85c035f9507e8b788df0b3ce20290a770) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs-client/dev-support/findbugsExcludeFile.xml hadoop-hdfs-project/hadoop-hdfs/dev-support/findbugsExcludeFile.xml
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #2260 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2260/)
        HDFS-8961. Investigate lock issue in o.a.h.hdfs.shortcircuit.DfsClientShmManager.EndpointShmManager. Contributed by Mingliang Liu. (wheat9: rev 1e5f69e85c035f9507e8b788df0b3ce20290a770)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/dev-support/findbugsExcludeFile.xml
        • hadoop-hdfs-project/hadoop-hdfs-client/dev-support/findbugsExcludeFile.xml
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2260 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2260/ ) HDFS-8961 . Investigate lock issue in o.a.h.hdfs.shortcircuit.DfsClientShmManager.EndpointShmManager. Contributed by Mingliang Liu. (wheat9: rev 1e5f69e85c035f9507e8b788df0b3ce20290a770) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/dev-support/findbugsExcludeFile.xml hadoop-hdfs-project/hadoop-hdfs-client/dev-support/findbugsExcludeFile.xml
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #303 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/303/)
        HDFS-8961. Investigate lock issue in o.a.h.hdfs.shortcircuit.DfsClientShmManager.EndpointShmManager. Contributed by Mingliang Liu. (wheat9: rev 1e5f69e85c035f9507e8b788df0b3ce20290a770)

        • hadoop-hdfs-project/hadoop-hdfs-client/dev-support/findbugsExcludeFile.xml
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/dev-support/findbugsExcludeFile.xml
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #303 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/303/ ) HDFS-8961 . Investigate lock issue in o.a.h.hdfs.shortcircuit.DfsClientShmManager.EndpointShmManager. Contributed by Mingliang Liu. (wheat9: rev 1e5f69e85c035f9507e8b788df0b3ce20290a770) hadoop-hdfs-project/hadoop-hdfs-client/dev-support/findbugsExcludeFile.xml hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/dev-support/findbugsExcludeFile.xml
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2241 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2241/)
        HDFS-8961. Investigate lock issue in o.a.h.hdfs.shortcircuit.DfsClientShmManager.EndpointShmManager. Contributed by Mingliang Liu. (wheat9: rev 1e5f69e85c035f9507e8b788df0b3ce20290a770)

        • hadoop-hdfs-project/hadoop-hdfs/dev-support/findbugsExcludeFile.xml
        • hadoop-hdfs-project/hadoop-hdfs-client/dev-support/findbugsExcludeFile.xml
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2241 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2241/ ) HDFS-8961 . Investigate lock issue in o.a.h.hdfs.shortcircuit.DfsClientShmManager.EndpointShmManager. Contributed by Mingliang Liu. (wheat9: rev 1e5f69e85c035f9507e8b788df0b3ce20290a770) hadoop-hdfs-project/hadoop-hdfs/dev-support/findbugsExcludeFile.xml hadoop-hdfs-project/hadoop-hdfs-client/dev-support/findbugsExcludeFile.xml hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        cmccabe Colin P. McCabe added a comment -

        Thank you for fixing the findbugs warning. Next time, can you please include what was done in the commit message and the JIRA name? "investigating" doesn't really describe what the commit does. Thanks.

        Show
        cmccabe Colin P. McCabe added a comment - Thank you for fixing the findbugs warning. Next time, can you please include what was done in the commit message and the JIRA name? "investigating" doesn't really describe what the commit does. Thanks.
        Hide
        wheat9 Haohui Mai added a comment -

        Updated the jira title to reflect the final solution. Please feel free to add additional details.

        Show
        wheat9 Haohui Mai added a comment - Updated the jira title to reflect the final solution. Please feel free to add additional details.

          People

          • Assignee:
            liuml07 Mingliang Liu
            Reporter:
            wheat9 Haohui Mai
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development