Uploaded image for project: 'Mesos'
  1. Mesos
  2. MESOS-3870

Prevent out-of-order libprocess message delivery

Attach filesAttach ScreenshotAdd voteVotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Epic
    • Status: Accepted
    • Minor
    • Resolution: Unresolved
    • None
    • None
    • libprocess
    • Ordered delivery in libprocess

    Description

      I was under the impression that send() provided in-order, unreliable message delivery. So if P1 sends <M1,M2> to P2, P2 might see <>, <M1>, <M2>, or <M1,M2> — but not <M2,M1>.

      I suspect much of the code makes a similar assumption. However, it appears that this behavior is not guaranteed. slave.cpp:2217 has the following comment:

        // TODO(jieyu): Here we assume that CheckpointResourcesMessages are
        // ordered (i.e., slave receives them in the same order master sends
        // them). This should be true in most of the cases because TCP
        // enforces in order delivery per connection. However, the ordering
        // is technically not guaranteed because master creates multiple
        // connections to the slave in some cases (e.g., persistent socket
        // to slave breaks and master uses ephemeral socket). This could
        // potentially be solved by using a version number and rejecting
        // stale messages according to the version number.
      

      We can improve this situation by either: (1) fixing libprocess to guarantee ordered message delivery, e.g., by adding a sequence number, or (2) clarifying that ordered message delivery is not guaranteed, and ideally providing a tool to force messages to be delivered out-of-order.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            neilc Neil Conway
            neilc Neil Conway
            Benjamin Mahler Benjamin Mahler

            Dates

              Created:
              Updated:

              Slack

                Issue deployment