Hadoop Common
  1. Hadoop Common
  2. HADOOP-8587

HarFileSystem access of harMetaCache isn't threadsafe

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 1.0.0, 2.0.0-alpha
    • Fix Version/s: 1.2.0, 0.23.3, 2.0.2-alpha
    • Component/s: fs
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      HarFileSystem's use of the static harMetaCache map is not threadsafe. Credit to Todd for pointing this out.

      1. hadoop-8587.txt
        2 kB
        Eli Collins
      2. hadoop-8587.txt
        1 kB
        Eli Collins
      3. hadoop-8587-b1.txt
        1 kB
        Eli Collins

        Activity

        Hide
        Eli Collins added a comment -

        Patch attached, use a ConcurrentHashMap instead.

        Show
        Eli Collins added a comment - Patch attached, use a ConcurrentHashMap instead.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12535996/hadoop-8587.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common:

        org.apache.hadoop.ha.TestZKFailoverController
        org.apache.hadoop.io.file.tfile.TestTFileByteArrays
        org.apache.hadoop.io.file.tfile.TestTFileJClassComparatorByteArrays

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1192//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/1192//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-common.html
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1192//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12535996/hadoop-8587.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common: org.apache.hadoop.ha.TestZKFailoverController org.apache.hadoop.io.file.tfile.TestTFileByteArrays org.apache.hadoop.io.file.tfile.TestTFileJClassComparatorByteArrays +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1192//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/1192//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-common.html Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1192//console This message is automatically generated.
        Hide
        Eli Collins added a comment -

        Same patch for branch-1. Ran TestHarFileSystem for sanity.

        Show
        Eli Collins added a comment - Same patch for branch-1. Ran TestHarFileSystem for sanity.
        Hide
        Daryn Sharp added a comment -

        +1 Looks simple enough. I think the findbugs and test failures are not related, and it wouldn't be easy to write a test for this.

        Show
        Daryn Sharp added a comment - +1 Looks simple enough. I think the findbugs and test failures are not related, and it wouldn't be easy to write a test for this.
        Hide
        Eli Collins added a comment -

        Thanks for the review Daryn. I've committed this to trunk and merged to branch-2 and branch-1.

        Show
        Eli Collins added a comment - Thanks for the review Daryn. I've committed this to trunk and merged to branch-2 and branch-1.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #2453 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2453/)
        HADOOP-8587. HarFileSystem access of harMetaCache isn't threadsafe. Contributed by Eli Collins (Revision 1360448)

        Result = SUCCESS
        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1360448
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/HarFileSystem.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2453 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2453/ ) HADOOP-8587 . HarFileSystem access of harMetaCache isn't threadsafe. Contributed by Eli Collins (Revision 1360448) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1360448 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/HarFileSystem.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #2519 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2519/)
        HADOOP-8587. HarFileSystem access of harMetaCache isn't threadsafe. Contributed by Eli Collins (Revision 1360448)

        Result = SUCCESS
        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1360448
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/HarFileSystem.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2519 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2519/ ) HADOOP-8587 . HarFileSystem access of harMetaCache isn't threadsafe. Contributed by Eli Collins (Revision 1360448) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1360448 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/HarFileSystem.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #2471 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2471/)
        HADOOP-8587. HarFileSystem access of harMetaCache isn't threadsafe. Contributed by Eli Collins (Revision 1360448)

        Result = FAILURE
        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1360448
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/HarFileSystem.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2471 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2471/ ) HADOOP-8587 . HarFileSystem access of harMetaCache isn't threadsafe. Contributed by Eli Collins (Revision 1360448) Result = FAILURE eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1360448 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/HarFileSystem.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1101 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1101/)
        HADOOP-8587. HarFileSystem access of harMetaCache isn't threadsafe. Contributed by Eli Collins (Revision 1360448)

        Result = FAILURE
        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1360448
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/HarFileSystem.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1101 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1101/ ) HADOOP-8587 . HarFileSystem access of harMetaCache isn't threadsafe. Contributed by Eli Collins (Revision 1360448) Result = FAILURE eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1360448 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/HarFileSystem.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1134 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1134/)
        HADOOP-8587. HarFileSystem access of harMetaCache isn't threadsafe. Contributed by Eli Collins (Revision 1360448)

        Result = SUCCESS
        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1360448
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/HarFileSystem.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1134 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1134/ ) HADOOP-8587 . HarFileSystem access of harMetaCache isn't threadsafe. Contributed by Eli Collins (Revision 1360448) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1360448 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/HarFileSystem.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #312 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/312/)
        svn merge -c 1360448 FIXES: HADOOP-8587. HarFileSystem access of harMetaCache isn't threadsafe. Contributed by Eli Collins (Revision 1360738)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1360738
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/HarFileSystem.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #312 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/312/ ) svn merge -c 1360448 FIXES: HADOOP-8587 . HarFileSystem access of harMetaCache isn't threadsafe. Contributed by Eli Collins (Revision 1360738) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1360738 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/HarFileSystem.java
        Hide
        Matt Foley added a comment -

        Added 1.2.0 to fixVersion, per CHANGES.txt and svn log.

        Show
        Matt Foley added a comment - Added 1.2.0 to fixVersion, per CHANGES.txt and svn log.

          People

          • Assignee:
            Eli Collins
            Reporter:
            Eli Collins
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development