Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8852

HDFS architecture documentation of version 2.x is outdated about append write support

    Details

    • Hadoop Flags:
      Reviewed

      Description

      In the latest version of the documentation, and also documentation for all releases with version 2, it’s mentioned that “A file once created, written, and closed need not be changed. “ and “There is a plan to support appending-writes to files in the future.”

      However, as far as I know, HDFS has supported append write since 0.21, based on HDFS-265 and the old version of the documentation in 2012

      Various posts on the Internet also suggests that append write has been available in HDFS, and will always be available in Hadoop version 2 branch.

      Can we update the documentation to reflect the current status?

      (Please also review whether the documentation should also be updated for version 0.21 and above, and the version 1.x branch)

      1. HDFS-8852.patch
        1 kB
        Ajith S
      2. HDFS-8852.2.patch
        1 kB
        Ajith S

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #289 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/289/)
          HDFS-8852. HDFS architecture documentation of version 2.x is outdated about append write support. Contributed by Ajith S. (aajisaka: rev fc509f66d814e7a5ed81d5d73b23c400625d573b)

          • hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsDesign.md
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #289 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/289/ ) HDFS-8852 . HDFS architecture documentation of version 2.x is outdated about append write support. Contributed by Ajith S. (aajisaka: rev fc509f66d814e7a5ed81d5d73b23c400625d573b) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsDesign.md hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2238 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2238/)
          HDFS-8852. HDFS architecture documentation of version 2.x is outdated about append write support. Contributed by Ajith S. (aajisaka: rev fc509f66d814e7a5ed81d5d73b23c400625d573b)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsDesign.md
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2238 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2238/ ) HDFS-8852 . HDFS architecture documentation of version 2.x is outdated about append write support. Contributed by Ajith S. (aajisaka: rev fc509f66d814e7a5ed81d5d73b23c400625d573b) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsDesign.md
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #281 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/281/)
          HDFS-8852. HDFS architecture documentation of version 2.x is outdated about append write support. Contributed by Ajith S. (aajisaka: rev fc509f66d814e7a5ed81d5d73b23c400625d573b)

          • hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsDesign.md
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #281 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/281/ ) HDFS-8852 . HDFS architecture documentation of version 2.x is outdated about append write support. Contributed by Ajith S. (aajisaka: rev fc509f66d814e7a5ed81d5d73b23c400625d573b) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsDesign.md hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2219 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2219/)
          HDFS-8852. HDFS architecture documentation of version 2.x is outdated about append write support. Contributed by Ajith S. (aajisaka: rev fc509f66d814e7a5ed81d5d73b23c400625d573b)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsDesign.md
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2219 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2219/ ) HDFS-8852 . HDFS architecture documentation of version 2.x is outdated about append write support. Contributed by Ajith S. (aajisaka: rev fc509f66d814e7a5ed81d5d73b23c400625d573b) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsDesign.md
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #292 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/292/)
          HDFS-8852. HDFS architecture documentation of version 2.x is outdated about append write support. Contributed by Ajith S. (aajisaka: rev fc509f66d814e7a5ed81d5d73b23c400625d573b)

          • hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsDesign.md
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #292 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/292/ ) HDFS-8852 . HDFS architecture documentation of version 2.x is outdated about append write support. Contributed by Ajith S. (aajisaka: rev fc509f66d814e7a5ed81d5d73b23c400625d573b) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsDesign.md hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #1022 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1022/)
          HDFS-8852. HDFS architecture documentation of version 2.x is outdated about append write support. Contributed by Ajith S. (aajisaka: rev fc509f66d814e7a5ed81d5d73b23c400625d573b)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsDesign.md
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1022 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1022/ ) HDFS-8852 . HDFS architecture documentation of version 2.x is outdated about append write support. Contributed by Ajith S. (aajisaka: rev fc509f66d814e7a5ed81d5d73b23c400625d573b) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsDesign.md
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8316 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8316/)
          HDFS-8852. HDFS architecture documentation of version 2.x is outdated about append write support. Contributed by Ajith S. (aajisaka: rev fc509f66d814e7a5ed81d5d73b23c400625d573b)

          • hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsDesign.md
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8316 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8316/ ) HDFS-8852 . HDFS architecture documentation of version 2.x is outdated about append write support. Contributed by Ajith S. (aajisaka: rev fc509f66d814e7a5ed81d5d73b23c400625d573b) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsDesign.md hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Committed this to trunk, branch-2, and branch-2.7. Thanks Ajith S for the contribution.

          Show
          ajisakaa Akira Ajisaka added a comment - Committed this to trunk, branch-2, and branch-2.7. Thanks Ajith S for the contribution.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          +1, committing this.

          Show
          ajisakaa Akira Ajisaka added a comment - +1, committing this.
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 3m 28s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
          +1 site 3m 12s Site still builds.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
              7m 12s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12750969/HDFS-8852.2.patch
          Optional Tests site
          git revision trunk / 71566e2
          Java 1.7.0_55
          uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12021/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 3m 28s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. +1 site 3m 12s Site still builds. +1 whitespace 0m 0s The patch has no lines that end in whitespace.     7m 12s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12750969/HDFS-8852.2.patch Optional Tests site git revision trunk / 71566e2 Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12021/console This message was automatically generated.
          Hide
          ajithshetty Ajith S added a comment -

          Thanks for the input Akira Ajisaka

          Uploaded the new patch as per your comments

          Show
          ajithshetty Ajith S added a comment - Thanks for the input Akira Ajisaka Uploaded the new patch as per your comments
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thanks Ajith S for updating the patch.

          except for appends

          truncate, introduced in Hadoop 2.7.0, can change the file. Would you change "appends" to "appends and truncates"? I'm +1 if that is addressed.

          Show
          ajisakaa Akira Ajisaka added a comment - Thanks Ajith S for updating the patch. except for appends truncate, introduced in Hadoop 2.7.0, can change the file. Would you change "appends" to "appends and truncates"? I'm +1 if that is addressed.
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 3m 11s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings.
          +1 site 2m 56s Site still builds.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
              6m 32s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12749753/HDFS-8852.patch
          Optional Tests site
          git revision trunk / fa1d84a
          Java 1.7.0_55
          uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11958/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 3m 11s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings. +1 site 2m 56s Site still builds. +1 whitespace 0m 0s The patch has no lines that end in whitespace.     6m 32s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12749753/HDFS-8852.patch Optional Tests site git revision trunk / fa1d84a Java 1.7.0_55 uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11958/console This message was automatically generated.
          Hide
          ajithshetty Ajith S added a comment -

          Please review the patch

          Show
          ajithshetty Ajith S added a comment - Please review the patch
          Hide
          ajithshetty Ajith S added a comment -

          +1 will update accordingly. Thanks Akira Ajisaka

          Show
          ajithshetty Ajith S added a comment - +1 will update accordingly. Thanks Akira Ajisaka
          Hide
          ajisakaa Akira Ajisaka added a comment -

          +1 for updating the document. Thanks Hong Dai Thanh for the report and Ajith S for taking this issue. I'm thinking

          • "the end of the files is supported" would be better than "files is supported at the end".
          • "cannot by updated" should be "cannot be updated".

          and it is also not possible to have multiple writers. Files can only be written by a single writer."

          Already written in "Data Replication" section (http://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html#Data_Replication), so we don't need to write it again.

          Show
          ajisakaa Akira Ajisaka added a comment - +1 for updating the document. Thanks Hong Dai Thanh for the report and Ajith S for taking this issue. I'm thinking "the end of the files is supported" would be better than "files is supported at the end". "cannot by updated" should be "cannot be updated". and it is also not possible to have multiple writers. Files can only be written by a single writer." Already written in "Data Replication" section ( http://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html#Data_Replication ), so we don't need to write it again.
          Hide
          ajithshetty Ajith S added a comment -

          May be we can update it to "Appending the content to files is supported at the end but cannot by updated at arbitrary point and it is also not possible to have multiple writers. Files can only be written by a single writer."

          Show
          ajithshetty Ajith S added a comment - May be we can update it to "Appending the content to files is supported at the end but cannot by updated at arbitrary point and it is also not possible to have multiple writers. Files can only be written by a single writer."

            People

            • Assignee:
              ajithshetty Ajith S
              Reporter:
              nhahtdh Hong Dai Thanh
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development