Uploaded image for project: 'Apache Ozone'
  1. Apache Ozone
  2. HDDS-4454 Ozone Streaming Write Pipeline
  3. HDDS-6137

Refactor KeyDataStreamOutput.

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • HDDS-4454
    • None

    Description

      Optimized the putBlock method of BlockDataStreamOutput

       

      1. PutBlock does not use RaftRpc, but through the stream close() method

      // putBlock metadata

ByteBuffer buf =
    ContainerCommandRequestMessage.toMessage(putBlockRequestProto, null)
        .getContent().asReadOnlyByteBuffer();

out.writeAsync(buf, StandardWriteOption.CLOSE);

      2.Mark the stream data boundary by sending an empty stream packet

      // Mark the stream data boundary

out.writeAsync(ByteBuffer.allocateDirect(0).asReadOnlyBuffer());

       

      3. Flush is also in the form of async RPC, because putBlock needs to fetch bcsId (Raft log index)

       

      Attachments

        1. 3195_buffer_copying.patch
          35 kB
          Tsz-wo Sze
        2. 3195_bugfix.patch
          28 kB
          Tsz-wo Sze
        3. 3195_bugfix2.patch
          28 kB
          Tsz-wo Sze
        4. 3195_review.patch
          17 kB
          Tsz-wo Sze
        5. 3195_synchronized.patch
          31 kB
          Tsz-wo Sze
        6. 3195_testBuffers.patch
          28 kB
          Tsz-wo Sze
        7. 3195_TestStreamDataChannel.patch
          20 kB
          Tsz-wo Sze
        8. ozone-root-datanode-9-29-173-57.log
          249 kB
          mingchao zhao
        9. ozone-root-datanode-9-29-173-57.log2.log
          288 kB
          mingchao zhao
        10. ozone-root-datanode-9-29-173-57.log3.log
          225 kB
          mingchao zhao
        11. ozone-root-datanode-9-29-173-57.log4
          5.84 MB
          mingchao zhao

        Issue Links

          depends upon

          Sub-task - The sub-task of the issue RATIS-1556 Support reference-counted buffer in StateMachine.DataChannel

          • Major - Major loss of function.
          • Resolved
          is depended upon by

          Sub-task - The sub-task of the issue HDDS-6500 Buffer the PutBlockRequest at the end of the stream

          • Major - Major loss of function.
          • Resolved
          links to

          Web Link GitHub Pull Request #3195

          Activity

            People

              Nicholas Niu GuoHao
              Nicholas Niu GuoHao
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: