Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha4
    • Fix Version/s: 2.9.0, 3.0.0-beta1
    • Component/s: None
    • Labels:
      None

      Description

      Update filesystem docs to describe the purpose and usage of StreamCapabilities.

      1. HDFS-12251.00.patch
        1 kB
        Lei (Eddy) Xu
      2. HDFS-12251.01.patch
        3 kB
        Lei (Eddy) Xu
      3. HDFS-12251.02.patch
        3 kB
        Lei (Eddy) Xu

        Issue Links

          Activity

          Hide
          andrew.wang Andrew Wang added a comment -

          Thanks for working on this Eddy. Could we add some content to the EC docs as well? I'm thinking of a section on "downstream concerns" that explains what functionality is different with EC and what can be done about it.

          Show
          andrew.wang Andrew Wang added a comment - Thanks for working on this Eddy. Could we add some content to the EC docs as well? I'm thinking of a section on "downstream concerns" that explains what functionality is different with EC and what can be done about it.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          Hi, Andrew Wang

          Thanks for the suggestions. I added a section in EC. Since OutputStream is not only used by downstream projects, but also user applications, so I used the title "Limitations" instead of "Downstream concerns". Would it be ok?

          Show
          eddyxu Lei (Eddy) Xu added a comment - Hi, Andrew Wang Thanks for the suggestions. I added a section in EC. Since OutputStream is not only used by downstream projects, but also user applications, so I used the title "Limitations" instead of "Downstream concerns". Would it be ok?
          Hide
          andrew.wang Andrew Wang added a comment -

          Thanks for the rev Eddy. Could you add a link from the EC doc to the FileSystem for StreamCapabilities?

          +1 pending that.

          Show
          andrew.wang Andrew Wang added a comment - Thanks for the rev Eddy. Could you add a link from the EC doc to the FileSystem for StreamCapabilities? +1 pending that.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 21s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
                trunk Compile Tests
          0 mvndep 1m 28s Maven dependency ordering for branch
          +1 mvninstall 15m 39s trunk passed
          +1 mvnsite 2m 36s trunk passed
                Patch Compile Tests
          0 mvndep 0m 17s Maven dependency ordering for patch
          +1 mvnsite 2m 26s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
                Other Tests
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          23m 20s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue HDFS-12251
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12880126/HDFS-12251.01.patch
          Optional Tests asflicense mvnsite
          uname Linux 30f6be59253d 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 79df1e7
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20536/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags.       trunk Compile Tests 0 mvndep 1m 28s Maven dependency ordering for branch +1 mvninstall 15m 39s trunk passed +1 mvnsite 2m 36s trunk passed       Patch Compile Tests 0 mvndep 0m 17s Maven dependency ordering for patch +1 mvnsite 2m 26s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues.       Other Tests +1 asflicense 0m 17s The patch does not generate ASF License warnings. 23m 20s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-12251 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12880126/HDFS-12251.01.patch Optional Tests asflicense mvnsite uname Linux 30f6be59253d 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 79df1e7 modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20536/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          Updated to add links between EC docs and StreamCapabilities doc

          Show
          eddyxu Lei (Eddy) Xu added a comment - Updated to add links between EC docs and StreamCapabilities doc
          Hide
          busbey Sean Busbey added a comment -
          1233	 * `StreamCapabilties.HFLUSH` ("*hflush*"): the capability to flush out the data
          1234	 in client's buffer.
          1235	 * `StreamCapabilities.HSYNC` ("*hsync*"): capability to flush out the data in
          1236	 client's buffer and the disk device.
          

          StreamCapabilities.StreamCapability isn't public, we shouldn't refer to it in downstream facing documentation. Just list the strings.

          Show
          busbey Sean Busbey added a comment - 1233 * `StreamCapabilties.HFLUSH` ( "*hflush*" ): the capability to flush out the data 1234 in client's buffer. 1235 * `StreamCapabilities.HSYNC` ( "*hsync*" ): capability to flush out the data in 1236 client's buffer and the disk device. StreamCapabilities.StreamCapability isn't public, we shouldn't refer to it in downstream facing documentation. Just list the strings.
          Hide
          busbey Sean Busbey added a comment -

          Am I correctly deducing from these docs changes that there isn't an ability query about append? Should there be?

          Show
          busbey Sean Busbey added a comment - Am I correctly deducing from these docs changes that there isn't an ability query about append ? Should there be?
          Hide
          busbey Sean Busbey added a comment -

          Questions I have after reading hte current draft:

          Do all outputstreams returned from FileSystem implement StreamCapabilities? (I think no)

          Presuming no, what should I assume about an outputstream that gets returned to me that doesn't implement StreamCapabilities? (I think up to the application, but dataloss sensitive applications need to presume no operations actually work)

          Show
          busbey Sean Busbey added a comment - Questions I have after reading hte current draft: Do all outputstreams returned from FileSystem implement StreamCapabilities? (I think no) Presuming no, what should I assume about an outputstream that gets returned to me that doesn't implement StreamCapabilities? (I think up to the application, but dataloss sensitive applications need to presume no operations actually work)
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
                trunk Compile Tests
          0 mvndep 1m 31s Maven dependency ordering for branch
          +1 mvninstall 15m 29s trunk passed
          +1 mvnsite 2m 41s trunk passed
                Patch Compile Tests
          0 mvndep 0m 15s Maven dependency ordering for patch
          +1 mvnsite 2m 27s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
                Other Tests
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          23m 10s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue HDFS-12251
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12880272/HDFS-12251.02.patch
          Optional Tests asflicense mvnsite
          uname Linux 9cd925ec4f5c 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / f4c6b00
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20548/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags.       trunk Compile Tests 0 mvndep 1m 31s Maven dependency ordering for branch +1 mvninstall 15m 29s trunk passed +1 mvnsite 2m 41s trunk passed       Patch Compile Tests 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvnsite 2m 27s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues.       Other Tests +1 asflicense 0m 17s The patch does not generate ASF License warnings. 23m 10s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-12251 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12880272/HDFS-12251.02.patch Optional Tests asflicense mvnsite uname Linux 9cd925ec4f5c 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f4c6b00 modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20548/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          Hey, Sean Busbey

          StreamCapabilities.StreamCapability isn't public

          I feel that it should be public. Will file a new JIRA for it.

          Do all outputstreams returned from FileSystem implement StreamCapabilities?

          I think they are going to support it, as WIP in HADOOP-14402.

          Am I correctly deducing from these docs changes that there isn't an ability query about append.

          You are right. append is not in StreamCapabilities, as it is a FileSystem call.

          Show
          eddyxu Lei (Eddy) Xu added a comment - Hey, Sean Busbey StreamCapabilities.StreamCapability isn't public I feel that it should be public. Will file a new JIRA for it. Do all outputstreams returned from FileSystem implement StreamCapabilities? I think they are going to support it, as WIP in HADOOP-14402 . Am I correctly deducing from these docs changes that there isn't an ability query about append. You are right. append is not in StreamCapabilities , as it is a FileSystem call.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          Thanks for the reviews, Andrew Wang. Committed to branch-2 and trunk.

          Show
          eddyxu Lei (Eddy) Xu added a comment - Thanks for the reviews, Andrew Wang . Committed to branch-2 and trunk.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12120 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12120/)
          HDFS-12251. Add document for StreamCapabilities. (Lei (Eddy) Xu) (lei: rev fe3341786a0d61f404127bf21d1afc85b2f21d38)

          • (edit) hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSErasureCoding.md
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12120 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12120/ ) HDFS-12251 . Add document for StreamCapabilities. (Lei (Eddy) Xu) (lei: rev fe3341786a0d61f404127bf21d1afc85b2f21d38) (edit) hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md (edit) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSErasureCoding.md
          Hide
          busbey Sean Busbey added a comment -

          StreamCapabilities.StreamCapability isn't public

          I feel that it should be public. Will file a new JIRA for it.

          In the future, please atleast wait until your proposed JIRA actually lands. I'm not sure I agree with this assertion that it should be public.

          Do all outputstreams returned from FileSystem implement StreamCapabilities?

          I think they are going to support it, as WIP in HADOOP-14402.

          Even if/when HADOOP-14402 completes, there's nothing that requires an outputstream to implement the interface right? so third party implementations might not do so. We should call this out in the docs.

          Show
          busbey Sean Busbey added a comment - StreamCapabilities.StreamCapability isn't public I feel that it should be public. Will file a new JIRA for it. In the future, please atleast wait until your proposed JIRA actually lands. I'm not sure I agree with this assertion that it should be public. Do all outputstreams returned from FileSystem implement StreamCapabilities? I think they are going to support it, as WIP in HADOOP-14402 . Even if/when HADOOP-14402 completes, there's nothing that requires an outputstream to implement the interface right? so third party implementations might not do so. We should call this out in the docs.
          Hide
          busbey Sean Busbey added a comment -

          Another thing missing from the docs as they are in v2:

          from Steve L on HDFS-11644:

          I don't want an enum though, I'd like a string. Why? Let's us and implementation classes add new methods without renegotiating the source tree. For example, on S3a, I could add the new attributes "s3a:delayed-commit", blobstore:put-on-close ( the latter being something which other blobstores could also do), etc. Some convention that for store-specific options, we use a prefix, so as to stop incompatibilities.

          The docs should call out that the options from store implementations should use a prefix. Preferably also state a suggested place implementors should doc the options they provide (based on whatever we do for documenting the stores in HADOOP-14402)

          Show
          busbey Sean Busbey added a comment - Another thing missing from the docs as they are in v2: from Steve L on HDFS-11644 : I don't want an enum though, I'd like a string. Why? Let's us and implementation classes add new methods without renegotiating the source tree. For example, on S3a, I could add the new attributes "s3a:delayed-commit", blobstore:put-on-close ( the latter being something which other blobstores could also do), etc. Some convention that for store-specific options, we use a prefix, so as to stop incompatibilities. The docs should call out that the options from store implementations should use a prefix. Preferably also state a suggested place implementors should doc the options they provide (based on whatever we do for documenting the stores in HADOOP-14402 )
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          Sean Busbey sorry for committing this early, i thought the scope of this JIRA was done. Nonetheless, I filed HDFS-12263 as a follow on.

          Lemme copy the discussions there.

          Show
          eddyxu Lei (Eddy) Xu added a comment - Sean Busbey sorry for committing this early, i thought the scope of this JIRA was done. Nonetheless, I filed HDFS-12263 as a follow on. Lemme copy the discussions there.

            People

            • Assignee:
              eddyxu Lei (Eddy) Xu
              Reporter:
              eddyxu Lei (Eddy) Xu
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development