Uploaded image for project: 'Apache Ozone'
  1. Apache Ozone
  2. HDDS-4454

Ozone Streaming Write Pipeline

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • None

    Description

      https://issues.apache.org/jira/browse/RATIS-979 implements streaming semantics in Ratis.

      Currently, ozone write pattern is bursty and involves multiple buffers copies as well multiple ratis log syncs in a block write. The idea of the Jira is to use a zero buffer copy based ratis streaming in ozone write path for better performance and resource utilisation. 

      Attachments

        1. Ozone Write Streaming.pdf
          683 kB
          Shashikant Banerjee
        2. Ozone Write Pipeline with Ratis Streaming20210609.pdf
          43 kB
          Tsz-wo Sze
        3. image-2020-11-17-09-15-58-880.png
          46 kB
          runzhiwang

        Issue Links

          1.
          Implement stream method to ContainerStateMachine Sub-task Resolved mingchao zhao
          2.
          Add link() method to ContainerStateMachine Sub-task Resolved Kaijie Chen
          3.
          Fix stream() and link() method in ContainerStateMachine Sub-task Resolved Kaijie Chen
          4.
          Client and server should support stream setup. Sub-task Resolved mingchao zhao
          5.
          Add a new BlockOutputStream/KeyOutputStream to support streaming api Sub-task Resolved Kaijie Chen
          6.
          BlockDataStreamOutput: drop BufferPool and ChunkBuffer to avoid buffer copying Sub-task Resolved mingchao zhao
          7.
          Change ByteBufStreamOutput to ByteBufferStreamOutput Sub-task Resolved Kaijie Chen
          8.
          Avoid unnecessary Bytebuffer conversions Sub-task Resolved mingchao zhao
          9.
          Write exceptions occur after checksum is enabled Sub-task Resolved mingchao zhao
          10.
          Add FileRegion API for Ozone Streaming Sub-task Closed Kaijie Chen
          11.
          Streaming supports writing in Pipline mode Sub-task Resolved mingchao zhao
          12.
          Handle client retries on exception Sub-task Resolved Sadanand Shenoy
          13.
          Make raft.server.data-stream.client.pool.size configurable Sub-task Resolved mingchao zhao
          14.
          Provide an Executor for each LocalStream in ContainerStateMachine Sub-task Resolved Tsz-wo Sze
          15.
          On the Datanode's ContainerStateMachine, the Stream and Link methods may get better performance by using chunkExecutors Sub-task Resolved GuoHao
          16.
          Too many open files appear when an OzoneClient uses Streaming Write for a long time Sub-task Resolved GuoHao
          17.
          Add XceiverClientRatis stream config Sub-task Resolved GuoHao
          18.
          update the usage space of Containers in the stream write Sub-task Resolved GuoHao
          19.
          OzoneBucket add the createMultipartStreamKey method Sub-task Resolved GuoHao
          20.
          Add option to write files via streaming api in ofs and o3fs. Sub-task Resolved Sadanand Shenoy
          21.
          Support merge chunkInfos in client Sub-task Resolved mingchao zhao
          22.
          Define a PutBlock/maxBuffer fixed boundary for streaming writes Sub-task Resolved Sadanand Shenoy
          23.
          Define a minimum packet size during streaming writes. Sub-task Resolved Sadanand Shenoy
          24.
          When releaseBuffers will get “Couldn 't find the required future” Sub-task Resolved Tsz-wo Sze
          25.
          Optimize parameters improve write performance Sub-task Resolved mingchao zhao
          26.
          The 'Cannot allocate Memory 'problem occurs in client while test write Sub-task Resolved mingchao zhao
          27.
          Fix incorrect computation of totalAckDataLength Sub-task Resolved Sadanand Shenoy
          28.
          Fix NPE in HDDS-6139 Sub-task Resolved Sadanand Shenoy
          29.
          Add trace for streaming Sub-task Resolved GuoHao
          30.
          Update ratis version to 2.3.0-94db58b-SNAPSHOT version Sub-task Resolved GuoHao
          31.
          Add XceiverServerRatis stream config Sub-task Resolved GuoHao
          32.
          Define a limit on the size of the retry bufferList . Sub-task Resolved Sadanand Shenoy
          33.
          Fix CheckStyle problem Sub-task Resolved GuoHao
          34.
          Data Channel abstraction on datanode Sub-task Resolved GuoHao
          35.
          Fix BlockDataStreamOutput#doFlushIfNeeded NPE Sub-task Resolved GuoHao
          36.
          Streaming write support both pipeline model and star model Sub-task Resolved mingchao zhao
          37.
          BlockDataStreamOutput support FlushDelay Sub-task Resolved mingchao zhao
          38.
          Update Ratis version to 2.3.0-da5d868-SNAPSHOT Sub-task Resolved Tsz-wo Sze
          39.
          Setup TLS between the datanodes while streaming Sub-task Resolved Tsz-wo Sze
          40.
          Refactor KeyDataStreamOutput. Sub-task Resolved GuoHao
          41.
          Buffer the PutBlockRequest at the end of the stream Sub-task Resolved Tsz-wo Sze
          42.
          Add option to createKey via streaming api via Freon Sub-task Resolved Sadanand Shenoy
          43.
          Fix ContainerStateMachine#applyTransaction assert error Sub-task Resolved GuoHao
          44.
          Add BlockToken verification for streaming writes Sub-task Open Tsz-wo Sze
          45.
          Use WriteSmallFile to write small file. Sub-task Patch Available GuoHao
          46.
          Added support for stream on S3Gateway write path Sub-task Patch Available GuoHao
          47.
          Ozone Streaming Performance Test(this is a long-term task) Sub-task In Progress mingchao zhao
          48.
          RpcClient#createMultipartStreamKey method, need to support file encrypt Sub-task Open GuoHao
          49.
          Stream packet should be aligned with bytesPerChecksum to optimize checksum computation Sub-task Open Sadanand Shenoy
          50.
          Ensure putBlock gets executed on last acknowledged streaming chunk write on exception Sub-task Open Sadanand Shenoy
          51.
          Move checksum computation on ozone client to putBlock execution Sub-task Open Sadanand Shenoy
          52.
          MR Acceptance test failure on Streaming Branch (HDDS-4454) Sub-task Open Sadanand Shenoy
          53.
          Ozone Retry Path Optimizations Sub-task Open Unassigned
          54.
          Optimize the stream on S3Gateway write path Sub-task Open GuoHao
          55.
          The following assertion error occurs when transferring large files Sub-task Open Unassigned

          Activity

            People

              sadanand_shenoy Sadanand Shenoy
              shashikant Shashikant Banerjee
              Votes:
              0 Vote for this issue
              Watchers:
              17 Start watching this issue

              Dates

                Created:
                Updated: