Uploaded image for project: 'Ratis'
  1. Ratis
  2. RATIS-1931

Support Zero-Copy in ratis-grpc

Attach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • gRPC
    • None

    Description

      As GRPC has stabilized the API for zero-copy for protobuf (https://github.com/grpc/grpc-java/issues/7387), it's possible to have such memory efficiency feature in Ratis GRPC.

      Attachments

        Issue Links

        is duplicated by

        Improvement - An improvement or enhancement to an existing feature or task. RATIS-1405 Avoid buffer copy with Grpc zero-copy marshaller

        • Major - Major loss of function.
        • Resolved
        1.
        		 Create zero-copy Marshaller Sub-task Resolved Duong

        100%
        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 50m
        		Actions
        2.
        		 Add TestGrpcZeroCopy Sub-task Resolved Tsz-wo Sze

        100%
        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 1h 10m
        		Actions
        3.
        		 Restore ZeroCopyMessageMarhaller#popStream Sub-task Resolved Attila Doroszlai

        100%
        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 0.5h
        		Actions
        4.
        		 Avoid Ratis internal data from referring to external input protobuf data source Sub-task Resolved Duong

        100%
        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 1.5h
        		Actions
        5.
        		 Do not store CommitInfoProto in CommitInfoCache Sub-task Resolved Tsz-wo Sze

        100%
        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 1h 40m
        		Actions
        6.
        		 Refactor client request processing to support reference count Sub-task Resolved Tsz-wo Sze

        100%
        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 1h
        		Actions
        7.
        		 Support Zero-Copy in GrpcClientProtocolService Sub-task Resolved Duong

        100%
        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 4h 40m
        		Actions
        8.
        		 Refactor appendEntries processing to support reference count Sub-task Resolved Duong

        100%
        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 2h 10m
        		Actions
        9.
        		 Support Zero-Copy in GrpcServerProtocolService Sub-task Resolved Duong

        100%
        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 1h 10m
        		Actions
        10.
        		 ReferenceCount should work for all LogEntry types. Sub-task Resolved Duong

        100%
        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 1h 20m
        		Actions
        11.
        		 Zero-copy buffers are not released Sub-task Resolved Duong

        100%
        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 50m
        		Actions
        12.
        		 Zero-copy buffers are not released - 2nd chunk Sub-task Resolved Duong

        100%
        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 1h 10m
        		Actions
        13.
        		 Add metrics assertions to verify all zero-copy messages are released properly Sub-task Resolved Duong

        100%
        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 1.5h
        		Actions
        14.
        		 Refactor StateMachine interface to use ReferenceCountedObject Sub-task Resolved Duong

        100%
        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 0.5h
        		Actions
        15.
        		 Refactor TransactionContext to supply LogEntryProto via a ReferenceCountedObject Sub-task Resolved Duong

        100%
        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 1.5h
        		Actions
        16.
        		 Refactor RaftLog to supply log as ReferenceCountedObject Sub-task Resolved Duong

        100%
        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 2h
        		Actions
        17.
        		 LogAppender to consume log entries with reference count Sub-task Resolved Duong

        100%
        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 2h 10m
        		Actions
        18.
        		 Allow StateMachine.read to return a ReferentCountedObject Sub-task Resolved Duong

        100%
        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 1h 20m
        		Actions
        19.
        		 Missing reference count when putting log entries to cache on follower Sub-task Resolved Duong

        100%
        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 0.5h
        		Actions
        20.
        		 Timedout StateMachine retainRead is released twice. Sub-task Resolved Duong

        100%
        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 2h 10m
        		Actions
        21.
        		 Add metrics to expose number of zero-copy unclosed messages Sub-task Resolved Duong

        100%
        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 20m
        		Actions
        22.
        		 Decouple metadata and configuration entries from appendEntries buffer for stateMachineCache Sub-task Resolved Duong

        100%
        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 1h
        		Actions
        23.
        		 TransactionContext's stateMachineLogEntry and stateMachineContext may cause corruption Sub-task Resolved Duong

        100%
        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 50m
        		Actions
        24.
        		 Add a conf to enable/disable zero copy. Sub-task Resolved Duong

        100%
        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 40m
        		Actions
        25.
        		 Corruption due to SegmentedRaftLogWorker queue LogEntry without reference counter Sub-task Resolved Duong   Actions
        26.
        		 TestRetryCacheWithGrpc may fail with object already completely released. Sub-task Open Duong   Actions
        27.
        		 Test failure because SimpleStateMachine4Testing is not closed properly Sub-task Open Nandakumar   Actions
        28.
        		 Benchmark zero-copy in ozone Sub-task Open Unassigned   Actions
        29.
        		 Support Zero-Copy in loading log entries from local files Sub-task Open Unassigned   Actions
        30.
        		 TestRaftWithSimulatedRpc could "fail to retain". Sub-task Open Unassigned

        100%
        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 10m
        		Actions
        31.
        		 LeakDetector has a race condition Sub-task Open Tsz-wo Sze

        100%
        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 10m
        		Actions

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            duongnguyen Duong
            duongnguyen Duong
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

              Created:
              Updated:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 32h 50m
                32h 50m

                Slack

                  Issue deployment