Hadoop Common
  1. Hadoop Common
  2. HADOOP-6231

Allow caching of filesystem instances to be disabled on a per-instance basis

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.18.0, 0.18.1, 0.18.2, 0.18.3, 0.19.0, 0.19.1, 0.19.2, 0.20.0, 0.20.1
    • Fix Version/s: 0.20.2
    • Component/s: fs
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Allow a general mechanism to disable the cache on a per filesystem basis by using property fs.<schemename>.impl.disable.cache. eg. fs.har.impl.disable.cache in core-default.xml

      Description

      HAR filesystem instances should not be cached, so this JIRA seeks to provide a general mechanism for disabling the cache on a per-filesystem basis. (Carried over from HADOOP-6097.)

      1. HADOOP-6231.patch
        3 kB
        Tom White
      2. HADOOP-6231-branch-0.20.patch
        3 kB
        Ben Slusky
      3. HADOOP-6231-v2.patch
        3 kB
        Tom White

        Issue Links

          Activity

          Show
          Tom White added a comment - A patch that implements this in the way suggested at https://issues.apache.org/jira/browse/HADOOP-6097?focusedCommentId=12745027&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12745027 .
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12418346/HADOOP-6231.patch
          against trunk revision 810384.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/8/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/8/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/8/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/8/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12418346/HADOOP-6231.patch against trunk revision 810384. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/8/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/8/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/8/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/8/console This message is automatically generated.
          Hide
          Chris Douglas added a comment -

          The unit test should use JUnit4 conventions rather than JUnit3, but aside from that the fix looks good

          Show
          Chris Douglas added a comment - The unit test should use JUnit4 conventions rather than JUnit3, but aside from that the fix looks good
          Hide
          Tom White added a comment -

          Thanks for the suggestion, Chris. Here's a modified patch.

          Show
          Tom White added a comment - Thanks for the suggestion, Chris. Here's a modified patch.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12418457/HADOOP-6231-v2.patch
          against trunk revision 810714.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/4/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/4/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/4/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/4/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12418457/HADOOP-6231-v2.patch against trunk revision 810714. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/4/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/4/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/4/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/4/console This message is automatically generated.
          Hide
          Tom White added a comment -

          I've just committed this.

          Show
          Tom White added a comment - I've just committed this.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #13 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/13/)
          . Allow caching of filesystem instances to be disabled on a per-instance basis.

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #13 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/13/ ) . Allow caching of filesystem instances to be disabled on a per-instance basis.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk #86 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/86/)

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk #86 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/86/ )
          Hide
          Ben Slusky added a comment -

          Patch against the 0.20 branch.

          Show
          Ben Slusky added a comment - Patch against the 0.20 branch.
          Hide
          Ben Slusky added a comment -

          This fixes a bug affecting all releases from 0.18 (when Hadoop archives were introduced). Without this patch, only one Hadoop archive can be opened at a time:

          $ hadoop dfs -ls har:///user/knoguchi/test.har har:///user/knoguchi/test2.har
          Found 1 items
          drw-r--r--   - knoguchi users          0 2009-08-18 18:52 /user/knoguchi/test.har/user
          ls: Invalid file name: /user/knoguchi/test2.har in har:///user/knoguchi/test.har
          
          $ hadoop dfs -ls har:///user/knoguchi/test2.har har:///user/knoguchi/test.har
          Found 1 items
          drw-------   - knoguchi users          0 2009-08-17 19:15 /user/knoguchi/test2.har/user
          ls: Invalid file name: /user/knoguchi/test.har in har:///user/knoguchi/test2.har
          
          Show
          Ben Slusky added a comment - This fixes a bug affecting all releases from 0.18 (when Hadoop archives were introduced). Without this patch, only one Hadoop archive can be opened at a time: $ hadoop dfs -ls har:///user/knoguchi/test.har har:///user/knoguchi/test2.har Found 1 items drw-r--r-- - knoguchi users 0 2009-08-18 18:52 /user/knoguchi/test.har/user ls: Invalid file name: /user/knoguchi/test2.har in har:///user/knoguchi/test.har $ hadoop dfs -ls har:///user/knoguchi/test2.har har:///user/knoguchi/test.har Found 1 items drw------- - knoguchi users 0 2009-08-17 19:15 /user/knoguchi/test2.har/user ls: Invalid file name: /user/knoguchi/test.har in har:///user/knoguchi/test2.har
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12419639/HADOOP-6231-branch-0.20.patch
          against trunk revision 815317.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/35/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12419639/HADOOP-6231-branch-0.20.patch against trunk revision 815317. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/35/console This message is automatically generated.
          Hide
          Mahadev konar added a comment -

          [exec]
          [exec] +1 overall.
          [exec]
          [exec] +1 @author. The patch does not contain any @author tags.
          [exec]
          [exec] +1 tests included. The patch appears to include 3 new or modified tests.
          [exec]
          [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
          [exec]
          [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
          [exec]
          [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
          [exec]
          [exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
          [exec]

          output of ant test-patch.... am still running ant test and will post the results....

          Show
          Mahadev konar added a comment - [exec] [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 3 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity. [exec] output of ant test-patch.... am still running ant test and will post the results....
          Hide
          Mahadev konar added a comment -

          I just committed this to 0.20 branch. thanks tom and ben!

          Show
          Mahadev konar added a comment - I just committed this to 0.20 branch. thanks tom and ben!
          Hide
          Nigel Daley added a comment -

          Isn't this really an improvement (not a bug). Can you write a release note for this?

          Show
          Nigel Daley added a comment - Isn't this really an improvement (not a bug). Can you write a release note for this?
          Hide
          Mahadev konar added a comment -

          nigel,
          This is really a bug, given that multiple archives in a single wont work without this patch. Please look at HADOOP-6097 for more details.
          I will add a release note on this.

          thanks

          Show
          Mahadev konar added a comment - nigel, This is really a bug, given that multiple archives in a single wont work without this patch. Please look at HADOOP-6097 for more details. I will add a release note on this. thanks
          Hide
          Mahadev konar added a comment -

          sorry pressed enter too soon,
          I meant "multiple archives in a single jvm wont work"

          Show
          Mahadev konar added a comment - sorry pressed enter too soon, I meant "multiple archives in a single jvm wont work"

            People

            • Assignee:
              Ben Slusky
              Reporter:
              Tom White
            • Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development