Uploaded image for project: 'Apache Ozone'
  1. Apache Ozone
  2. HDDS-4454

Ozone Streaming Write Pipeline

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • None

    Description

      https://issues.apache.org/jira/browse/RATIS-979 implements streaming semantics in Ratis.

      Currently, ozone write pattern is bursty and involves multiple buffers copies as well multiple ratis log syncs in a block write. The idea of the Jira is to use a zero buffer copy based ratis streaming in ozone write path for better performance and resource utilisation. 

      Attachments

        1. Ozone Write Pipeline with Ratis Streaming20210609.pdf
          43 kB
          Tsz-wo Sze
        2. image-2020-11-17-09-15-58-880.png
          46 kB
          runzhiwang
        3. Ozone Write Streaming.pdf
          683 kB
          Shashikant Banerjee

        Issue Links

          1.
          Implement stream method to ContainerStateMachine Sub-task Resolved mingchao zhao
          2.
          Add link() method to ContainerStateMachine Sub-task Resolved Kaijie Chen
          3.
          Fix stream() and link() method in ContainerStateMachine Sub-task Resolved Kaijie Chen
          4.
          Client and server should support stream setup. Sub-task Resolved mingchao zhao
          5.
          Add a new BlockOutputStream/KeyOutputStream to support streaming api Sub-task Resolved Kaijie Chen
          6.
          BlockDataStreamOutput: drop BufferPool and ChunkBuffer to avoid buffer copying Sub-task Resolved mingchao zhao
          7.
          Change ByteBufStreamOutput to ByteBufferStreamOutput Sub-task Resolved Kaijie Chen
          8.
          Avoid unnecessary Bytebuffer conversions Sub-task Resolved mingchao zhao
          9.
          Write exceptions occur after checksum is enabled Sub-task Resolved mingchao zhao
          10.
          [Won't Do] Add FileRegion API for Ozone Streaming Sub-task Resolved Kaijie Chen
          11.
          Streaming supports writing in Pipline mode Sub-task Resolved mingchao zhao
          12.
          Handle client retries on exception Sub-task Resolved Sadanand Shenoy
          13.
          Make raft.server.data-stream.client.pool.size configurable Sub-task Resolved mingchao zhao
          14.
          Provide an Executor for each LocalStream in ContainerStateMachine Sub-task Resolved Tsz-wo Sze
          15.
          On the Datanode's ContainerStateMachine, the Stream and Link methods may get better performance by using chunkExecutors Sub-task Resolved GuoHao
          16.
          Too many open files appear when an OzoneClient uses Streaming Write for a long time Sub-task Resolved GuoHao
          17.
          Add XceiverClientRatis stream config Sub-task Resolved GuoHao
          18.
          update the usage space of Containers in the stream write Sub-task Resolved GuoHao
          19.
          OzoneBucket add the createMultipartStreamKey method Sub-task Resolved GuoHao
          20.
          Add option to write files via streaming api in ofs and o3fs. Sub-task Resolved Sadanand Shenoy
          21.
          Support merge chunkInfos in client Sub-task Resolved mingchao zhao
          22.
          Define a PutBlock/maxBuffer fixed boundary for streaming writes Sub-task Resolved Sadanand Shenoy
          23.
          Define a minimum packet size during streaming writes. Sub-task Resolved Sadanand Shenoy
          24.
          When releaseBuffers will get “Couldn 't find the required future” Sub-task Resolved Tsz-wo Sze
          25.
          Optimize parameters improve write performance Sub-task Resolved mingchao zhao
          26.
          The 'Cannot allocate Memory 'problem occurs in client while test write Sub-task Resolved mingchao zhao
          27.
          Fix incorrect computation of totalAckDataLength Sub-task Resolved Sadanand Shenoy
          28.
          Fix NPE in HDDS-6139 Sub-task Resolved Sadanand Shenoy
          29.
          Use WriteSmallFile to write small file. Sub-task Patch Available GuoHao
          30.
          Added support for stream on S3Gateway write path Sub-task Patch Available GuoHao
          31.
          Ozone Streaming Performance Test(this is a long-term task) Sub-task In Progress mingchao zhao
          32.
          RpcClient#createMultipartStreamKey method, need to support file encrypt Sub-task In Progress GuoHao
          33.
          Define a limit on the size of the retry bufferList . Sub-task Patch Available Sadanand Shenoy
          34.
          Make local/closest node as the primary node Sub-task Open mingchao zhao
          35.
          Add BlockToken verification for streaming writes Sub-task Open Sadanand Shenoy
          36.
          Stream packet should be aligned with bytesPerChecksum to optimize checksum computation Sub-task Open Sadanand Shenoy
          37.
          Setup TLS between the datanodes while streaming Sub-task Open Sadanand Shenoy
          38.
          Ensure putBlock gets executed on last acknowledged streaming chunk write on exception Sub-task Open Sadanand Shenoy
          39.
          Move checksum computation on ozone client to putBlock execution Sub-task Open Sadanand Shenoy
          40.
          Ozone Retry Path Optimizations Sub-task Open Unassigned
          41.
          Add trace for streaming Sub-task Open GuoHao
          42.
          MR Acceptance test failure on Streaming Branch (HDDS-4454) Sub-task Open Sadanand Shenoy
          43.
          Optimize the stream on S3Gateway write path Sub-task Open GuoHao
          44.
          Optimized the putBlock method of BlockDataStreamOutput Sub-task Open GuoHao

          Activity

            People

              sadanand_shenoy Sadanand Shenoy
              shashikant Shashikant Banerjee
              Votes:
              0 Vote for this issue
              Watchers:
              17 Start watching this issue

              Dates

                Created:
                Updated: