Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.19.0
    • Fix Version/s: 0.19.1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      HDFS append() is disabled. It throws UnsupportedOperationException.

      Description

      As discussed on core-dev@ (http://www.nabble.com/Hadoop-0.19.1-td21739202.html) we will disable append API in Hadoop 0.19.1 by throwing UnsupportedOperationException from FileSystem.append(...) methods.

      Does append in libhdfs (HADOOP-4494) need attention too?

      1. HADOOP-5224-libhdfs.patch
        0.4 kB
        Raghu Angadi
      2. HADOOP-5224.patch
        6 kB
        Raghu Angadi
      3. disableAppend.patch
        8 kB
        dhruba borthakur

        Issue Links

          Activity

          Hide
          Raghu Angadi added a comment -

          I just committed libhdfs.patch to 0.19 (only).

          Show
          Raghu Angadi added a comment - I just committed libhdfs.patch to 0.19 (only).
          Hide
          Raghu Angadi added a comment -

          libhdfs.patch : disables append tests in libhdfs.

          The command ' ant -Dcompile.c++=true -Dlibhdfs=true clean compile test-libhdfs ' fails with current 0.19 and passes with this patch applied.

          I will commit this correction soon today.

          Show
          Raghu Angadi added a comment - libhdfs.patch : disables append tests in libhdfs. The command ' ant -Dcompile.c++=true -Dlibhdfs=true clean compile test-libhdfs ' fails with current 0.19 and passes with this patch applied. I will commit this correction soon today.
          Hide
          Raghu Angadi added a comment -

          > 'ant test-libhdfs' passes as well.

          my bad. I had reverted the patch by mistake before running 'ant test-libhdfs'. So the patch I attached is incomplete since it does not remove the libhdfs append test. Please disregard it.

          Show
          Raghu Angadi added a comment - > 'ant test-libhdfs' passes as well. my bad. I had reverted the patch by mistake before running 'ant test-libhdfs'. So the patch I attached is incomplete since it does not remove the libhdfs append test. Please disregard it.
          Hide
          dhruba borthakur added a comment -

          Nigel, thanks for the suggestion. I am closing this one.

          Show
          dhruba borthakur added a comment - Nigel, thanks for the suggestion. I am closing this one.
          Hide
          Nigel Daley added a comment -

          Hi folks, please pardon me for re-opening this issue. I am wondering if there is a better way to disable "appends" for 0.19.1 We coudld set the config parameter dfs.support.append to false in hadoop-defaults.xml. This means that a default installation of hadoop will not support "appends". However, for people that are testing their apps (not in production), it gives them an easy way to test their apps.

          Dhruba, folks who want to test with append can use 0.19.0.

          There could be big-and-large warning message associated with the dfs.support.append parameter warning people to the fact that there could be data corruption if they set that flag to true.

          The uploaded patch creates a dfs.support.append config entry but doesn't seem to use it in the append method to affect it's behavior. What am I missing?

          I am re-opening this issue because I am not sure whether this approach was considered. If it was considered and was shot down, please let me know why and u can close this issue.

          I think this should be a new Jira that could be fixed in 0.19.2. If you agree, please close this and open a new Jira.

          Show
          Nigel Daley added a comment - Hi folks, please pardon me for re-opening this issue. I am wondering if there is a better way to disable "appends" for 0.19.1 We coudld set the config parameter dfs.support.append to false in hadoop-defaults.xml. This means that a default installation of hadoop will not support "appends". However, for people that are testing their apps (not in production), it gives them an easy way to test their apps. Dhruba, folks who want to test with append can use 0.19.0. There could be big-and-large warning message associated with the dfs.support.append parameter warning people to the fact that there could be data corruption if they set that flag to true. The uploaded patch creates a dfs.support.append config entry but doesn't seem to use it in the append method to affect it's behavior. What am I missing? I am re-opening this issue because I am not sure whether this approach was considered. If it was considered and was shot down, please let me know why and u can close this issue. I think this should be a new Jira that could be fixed in 0.19.2. If you agree, please close this and open a new Jira.
          Hide
          dhruba borthakur added a comment -

          Hi Raghu, I uploaded a patch where the only change is to set "dfs.support.append" to false. Does this look ok to you?

          Show
          dhruba borthakur added a comment - Hi Raghu, I uploaded a patch where the only change is to set "dfs.support.append" to false. Does this look ok to you?
          Hide
          Raghu Angadi added a comment -

          I think I shouldn't have been assigned this jira in the first place.

          Show
          Raghu Angadi added a comment - I think I shouldn't have been assigned this jira in the first place.
          Hide
          dhruba borthakur added a comment -

          Making the config parameter dfs.support.append to "false" by default might allow Jim Kellerman to continue testing his Hbase code with 0.19.1 (till these get resolved in later releases).

          Show
          dhruba borthakur added a comment - Making the config parameter dfs.support.append to "false" by default might allow Jim Kellerman to continue testing his Hbase code with 0.19.1 (till these get resolved in later releases).
          Hide
          dhruba borthakur added a comment -

          Hi folks, please pardon me for re-opening this issue. I am wondering if there is a better way to disable "appends" for 0.19.1 We coudld set the config parameter dfs.support.append to false in hadoop-defaults.xml. This means that a default installation of hadoop will not support "appends". However, for people that are testing their apps (not in production), it gives them an easy way to test their apps. There could be big-and-large warning message associated with the dfs.support.append parameter warning people to the fact that there could be data corruption if they set that flag to true.

          I am re-opening this issue because I am not sure whether this approach was considered. If it was considered and was shot down, please let me know why and u can close this issue.

          Show
          dhruba borthakur added a comment - Hi folks, please pardon me for re-opening this issue. I am wondering if there is a better way to disable "appends" for 0.19.1 We coudld set the config parameter dfs.support.append to false in hadoop-defaults.xml. This means that a default installation of hadoop will not support "appends". However, for people that are testing their apps (not in production), it gives them an easy way to test their apps. There could be big-and-large warning message associated with the dfs.support.append parameter warning people to the fact that there could be data corruption if they set that flag to true. I am re-opening this issue because I am not sure whether this approach was considered. If it was considered and was shot down, please let me know why and u can close this issue.
          Hide
          Raghu Angadi added a comment -

          I just committed committed this to 0.19 (only).

          Show
          Raghu Angadi added a comment - I just committed committed this to 0.19 (only).
          Hide
          Konstantin Shvachko added a comment -

          True. +1

          Show
          Konstantin Shvachko added a comment - True. +1
          Hide
          Raghu Angadi added a comment -

          correct. But TestStickyBit is not in 0.19.

          Show
          Raghu Angadi added a comment - correct. But TestStickyBit is not in 0.19.
          Hide
          Konstantin Shvachko added a comment -

          Looks like TestStickyBit uses append(). Something tells me it should have failed.

          Show
          Konstantin Shvachko added a comment - Looks like TestStickyBit uses append(). Something tells me it should have failed.
          Hide
          Raghu Angadi added a comment -

          'ant test-libhdfs' passes as well.

          ant test-patch :

               [exec] +1 overall.
               [exec]
               [exec]     +1 @author.  The patch does not contain any @author tags.
               [exec]
               [exec]     +1 tests included.  The patch appears to include 9 new or modified tests.
               [exec]
               [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
               [exec]
               [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
               [exec]
               [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
          
          Show
          Raghu Angadi added a comment - 'ant test-libhdfs' passes as well. ant test-patch : [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 9 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
          Hide
          Raghu Angadi added a comment -

          'ant test-contrib' passes as well.

          Show
          Raghu Angadi added a comment - 'ant test-contrib' passes as well.
          Hide
          Raghu Angadi added a comment -

          patch for review. The only real change is that DistributedFileSystem.append() throws UnsupportedOperationException. Rest of the changes are for disabling tests that depend on HDFS append.

          Show
          Raghu Angadi added a comment - patch for review. The only real change is that DistributedFileSystem.append() throws UnsupportedOperationException . Rest of the changes are for disabling tests that depend on HDFS append.
          Hide
          Robert Chansler added a comment -

          This particular item turns off append() at the interface without affecting the implementation or normal behavior of sync().
          A related item, Hadoop-5225, disables some of the recovery code associated with sync in the case of data node failure.

          The use of either of these facilities in 0.19.0 is dangerous to the integrity of the file system. The goal is to provide 0.19.1 with the same file system integrity as 0.18.3. Further development of append and sync recovery can then continue.

          Show
          Robert Chansler added a comment - This particular item turns off append() at the interface without affecting the implementation or normal behavior of sync() . A related item, Hadoop-5225, disables some of the recovery code associated with sync in the case of data node failure. The use of either of these facilities in 0.19.0 is dangerous to the integrity of the file system. The goal is to provide 0.19.1 with the same file system integrity as 0.18.3. Further development of append and sync recovery can then continue.
          Hide
          Raghu Angadi added a comment -

          Jim,

          btw, this is assigned to me just to provide a patch. The decision is not mine. I will let others reply to the above valid concern.

          Show
          Raghu Angadi added a comment - Jim, btw, this is assigned to me just to provide a patch. The decision is not mine. I will let others reply to the above valid concern.
          Hide
          Jim Kellerman added a comment -

          Where does this leave HADOOP-4379?

          The testing I have done to date indicates that sync() does, in fact, work. See HBASE-1155

          Show
          Jim Kellerman added a comment - Where does this leave HADOOP-4379 ? The testing I have done to date indicates that sync() does, in fact, work. See HBASE-1155

            People

            • Assignee:
              Unassigned
              Reporter:
              Nigel Daley
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development