Uploaded image for project: 'Qpid'
  1. Qpid
  2. QPID-5799

Allow qpid.messaging.endpoints.session to have all associated receivers mointored via epoll mechanism

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.31
    • Component/s: Python Client
    • Labels:
      None

      Description

      Kombu [0] is a python producer/consumer software that I've written a qpid transport[1] for. This qpid transport for Kombu, uses qpid.messaging, and has all consuming receivers associated with a single session endpoint.

      Celery[2] uses Kombu, and has an asynchronous I/O event loop, and would like to monitor a single file descriptor for reading to know when there is any message available on any receiver associated with the session endpoint.

      Currently as a workaround, I create a pipe with one read and one write file descriptors. I register the read file descriptor with Celery to monitor. I then start a separate thread (Thread1) that calls next_receiver() on the session in an infinite loop. Each time next_receiver() returns instead of fetching the message, Thread1 writes a symbol to the file descriptor Celery is monitoring. Celery then calls the on_readable() callback from the MainThread, which removes a symbol from the pipe Celery is monitoring (clearing the epoll readable attribute), and then calls next_receiver() again only this time from MainThread, to acquire the receiver, and fetch() the message and process it.

      This architecture is not ideal in two ways.
      1. There is a redundant call to next_receiver() that happens with every read.

      2. A separate thread to "monitor" qpid messaging should not be required, when Celery is already capable of monitoring a single file descriptor, and can call the read method which would use next_receiver() to drain from all receivers associated with the session.

      [0]: https://github.com/celery/kombu
      [1]: https://github.com/pulp/kombu/blob/pulp-dep-3.0.15-with-qpid/kombu/transport/qpid.py
      [2]: https://github.com/celery/celery

        Attachments

          Activity

            People

            • Assignee:
              kgiusti Ken Giusti
              Reporter:
              bmbouter Brian Bouterse
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: