Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-14402

roll out StreamCapabilities across output streams of all filesystems

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.0.0-alpha4
    • Fix Version/s: None
    • Component/s: fs, fs/adl, fs/azure, fs/oss, fs/s3, fs/swift
    • Labels:
      None
    • Target Version/s:

      Description

      HDFS-11644 added a way to probe streams for capabilities, with dfsoutput stream implementing some.

      We should roll this out to all our output streams, with capabilities to define

      • is metadata in sync with flushed data (HDFS: no)
      • does LocalFileSystem really do a sync/flush (as I don't think any subclass of ChecksumFS does, so FSOutputSummer should declare this in its capabilities
      • are intermediate writes visible at all?
      • Is close() potentially a long operation (for object stores, yes, on a case-by-case basis)

      We'll need useful names for these options, obviously, tests, etc etc

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                stevel@apache.org Steve Loughran
                Reporter:
                stevel@apache.org Steve Loughran
              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated: