Uploaded image for project: 'Apache Ozone'
  1. Apache Ozone
  2. HDDS-4454

Ozone Streaming Write Pipeline

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • None

    Description

      https://issues.apache.org/jira/browse/RATIS-979 implements streaming semantics in Ratis.

      Currently, ozone write pattern is bursty and involves multiple buffers copies as well multiple ratis log syncs in a block write. The idea of the Jira is to use a zero buffer copy based ratis streaming in ozone write path for better performance and resource utilisation. 

      Attachments

        1. Ozone Write Streaming.pdf
          683 kB
          Shashikant Banerjee
        2. image-2020-11-17-09-15-58-880.png
          46 kB
          runzhiwang
        3. Ozone Write Pipeline with Ratis Streaming20210609.pdf
          43 kB
          Tsz-wo Sze

        Issue Links

          1.
          Implement stream method to ContainerStateMachine Sub-task Resolved mingchao zhao
          2.
          Add link() method to ContainerStateMachine Sub-task Resolved Kaijie Chen
          3.
          Fix stream() and link() method in ContainerStateMachine Sub-task Resolved Kaijie Chen
          4.
          Client and server should support stream setup. Sub-task Resolved mingchao zhao
          5.
          Add a new BlockOutputStream/KeyOutputStream to support streaming api Sub-task Resolved Kaijie Chen
          6.
          BlockDataStreamOutput: drop BufferPool and ChunkBuffer to avoid buffer copying Sub-task Resolved mingchao zhao
          7.
          Change ByteBufStreamOutput to ByteBufferStreamOutput Sub-task Resolved Kaijie Chen
          8.
          Avoid unnecessary Bytebuffer conversions Sub-task Resolved mingchao zhao
          9.
          Write exceptions occur after checksum is enabled Sub-task Resolved mingchao zhao
          10.
          Add FileRegion API for Ozone Streaming Sub-task Closed Kaijie Chen
          11.
          Streaming supports writing in Pipline mode Sub-task Resolved mingchao zhao
          12.
          Handle client retries on exception Sub-task Resolved Sadanand Shenoy
          13.
          Make raft.server.data-stream.client.pool.size configurable Sub-task Resolved mingchao zhao
          14.
          Provide an Executor for each LocalStream in ContainerStateMachine Sub-task Resolved Tsz-wo Sze
          15.
          On the Datanode's ContainerStateMachine, the Stream and Link methods may get better performance by using chunkExecutors Sub-task Resolved GuoHao
          16.
          Too many open files appear when an OzoneClient uses Streaming Write for a long time Sub-task Resolved GuoHao
          17.
          Add XceiverClientRatis stream config Sub-task Resolved GuoHao
          18.
          update the usage space of Containers in the stream write Sub-task Resolved GuoHao
          19.
          OzoneBucket add the createMultipartStreamKey method Sub-task Resolved GuoHao
          20.
          Add option to write files via streaming api in ofs and o3fs. Sub-task Resolved Sadanand Shenoy
          21.
          Support merge chunkInfos in client Sub-task Resolved mingchao zhao
          22.
          Define a PutBlock/maxBuffer fixed boundary for streaming writes Sub-task Resolved Sadanand Shenoy
          23.
          Define a minimum packet size during streaming writes. Sub-task Resolved Sadanand Shenoy
          24.
          When releaseBuffers will get “Couldn 't find the required future” Sub-task Resolved Tsz-wo Sze
          25.
          Optimize parameters improve write performance Sub-task Resolved mingchao zhao
          26.
          The 'Cannot allocate Memory 'problem occurs in client while test write Sub-task Resolved mingchao zhao
          27.
          Fix incorrect computation of totalAckDataLength Sub-task Resolved Sadanand Shenoy
          28.
          Fix NPE in HDDS-6139 Sub-task Resolved Sadanand Shenoy
          29.
          Add trace for streaming Sub-task Resolved GuoHao
          30.
          Update ratis version to 2.3.0-94db58b-SNAPSHOT version Sub-task Resolved GuoHao
          31.
          Add XceiverServerRatis stream config Sub-task Resolved GuoHao
          32.
          Define a limit on the size of the retry bufferList . Sub-task Resolved Sadanand Shenoy
          33.
          Fix CheckStyle problem Sub-task Resolved GuoHao
          34.
          Data Channel abstraction on datanode Sub-task Resolved GuoHao
          35.
          Fix BlockDataStreamOutput#doFlushIfNeeded NPE Sub-task Resolved GuoHao
          36.
          Streaming write support both pipeline model and star model Sub-task Resolved mingchao zhao
          37.
          BlockDataStreamOutput support FlushDelay Sub-task Resolved mingchao zhao
          38.
          Update Ratis version to 2.3.0-da5d868-SNAPSHOT Sub-task Resolved Tsz-wo Sze
          39.
          Setup TLS between the datanodes while streaming Sub-task Resolved Tsz-wo Sze
          40.
          Refactor KeyDataStreamOutput. Sub-task Resolved GuoHao
          41.
          Buffer the PutBlockRequest at the end of the stream Sub-task Resolved Tsz-wo Sze
          42.
          Add option to createKey via streaming api via Freon Sub-task Resolved Sadanand Shenoy
          43.
          Fix ContainerStateMachine#applyTransaction assert error Sub-task Resolved GuoHao
          44.
          Add BlockToken verification for streaming writes Sub-task Open Tsz-wo Sze
          45.
          Use WriteSmallFile to write small file. Sub-task Patch Available GuoHao
          46.
          Added support for stream on S3Gateway write path Sub-task Patch Available GuoHao
          47.
          Ozone Streaming Performance Test(this is a long-term task) Sub-task In Progress mingchao zhao
          48.
          RpcClient#createMultipartStreamKey method, need to support file encrypt Sub-task Open GuoHao
          49.
          Stream packet should be aligned with bytesPerChecksum to optimize checksum computation Sub-task Open Sadanand Shenoy
          50.
          Ensure putBlock gets executed on last acknowledged streaming chunk write on exception Sub-task Open Sadanand Shenoy
          51.
          Move checksum computation on ozone client to putBlock execution Sub-task Open Sadanand Shenoy
          52.
          MR Acceptance test failure on Streaming Branch (HDDS-4454) Sub-task Open Sadanand Shenoy
          53.
          Ozone Retry Path Optimizations Sub-task Open Unassigned
          54.
          Optimize the stream on S3Gateway write path Sub-task Open GuoHao
          55.
          The following assertion error occurs when transferring large files Sub-task Open Unassigned

          Activity

            People

              sadanand_shenoy Sadanand Shenoy
              shashikant Shashikant Banerjee
              Votes:
              0 Vote for this issue
              Watchers:
              17 Start watching this issue

              Dates

                Created:
                Updated: